Hover a node · click to focus · ←/→ steps
Ingest → embed → index → retrieve → generate. Offline + online eval gates block regressions before promote.
When you need this
- RAG answers cite wrong documents
- Prompt changes ship without regression tests
- Enterprise asks for accuracy evidence
- Chunking or embedding model change has no quality gate
Prerequisites
- RAG pipeline in staging with fixed index snapshot
- 20–50 golden Q&A pairs from real user questions
- CI secrets for judge model (or local judge)
Tools
Self-host for regulated workloads; cloud for faster pilot.
Run in CI before prompt promotion - pairs with Langfuse datasets.
Wire into CI with a fixed golden set - do not rely on LLM-judge alone without human calibration.
Pairs well with Promptfoo; use when teams already live in pytest.
Steps
- 1
Create golden dataset in Langfuse
Import question, expected sources, and reference answers. Tag by product area and risk level. Version the dataset like code.
- 2
Define eval metrics with Ragas / DeepEval
Track faithfulness, context precision/recall, citation match, latency p95, and cost per query. Set minimum thresholds per environment (stricter in prod).
- 3
Run Promptfoo (or Ragas suite) in CI
On PR, run evals against the staging index. Fail if faithfulness drops >2% or citation match regresses. Publish a markdown report on the PR.
- 4
Trace failures in Langfuse
Link failing cases to full traces — retrieval chunks, reranker scores, and final prompt — so engineers fix root cause, not symptoms.
- 5
Add online sampling
Sample production traces weekly into the same metric suite. Alert when online faithfulness drifts from offline baseline.
Adoption pitfalls
- Golden set written by engineers, not users — misses real failure modes
- Judge LLM without calibration → noisy fails / false greens
- Eval against a stale index while prod moved
Adoption checklist
- Golden set covers top 20 production question types
- CI blocks merge on eval regression
- Weekly eval report shared with product owner
- Failure modes documented in runbook
- Index + embedding version pinned in eval jobs
SEER REAL assessment / sprint
Assessment scores whether RAG has offline gates and online sampling. A typical sprint builds the golden set + CI harness for one product surface and wires Langfuse traces.