Quick answer: you evaluate a RAG pipeline by scoring retrieval and generation separately. Retrieval gets context precision and context recall. Generation gets faithfulness and answer relevancy. Frameworks like RAGAS automate all four, and a small golden dataset of 50 to 100 question-answer pairs is enough to start.
Most teams skip this and eyeball a few answers instead. That works right up until the pipeline fails somewhere you didn't look, usually in retrieval, and usually silently.
Why "It Looks Right" Isn't Evaluation
A RAG answer can read beautifully and still be wrong in three different ways: the retriever pulled the wrong chunks, the retriever missed the right chunks, or the model ignored the chunks and made something up.
You can't tell which happened by reading the answer. A fluent hallucination and a well-grounded answer look identical to a human skimming output.
That's the core argument for component-level evaluation. When your end-to-end quality drops, you need to know whether to fix the retriever (chunking, embeddings, top-k) or the generator (prompt, model, context formatting). One aggregate "quality score" can't tell you that. Four targeted metrics can.
The Four Metrics That Matter
Each metric isolates one failure mode. Here's the map:
| Metric | Component | Question it answers |
|---|---|---|
| Context precision | Retrieval | Of the chunks retrieved, how many were actually relevant? |
| Context recall | Retrieval | Of the information needed, how much did we retrieve? |
| Faithfulness | Generation | Is every claim in the answer supported by the retrieved context? |
| Answer relevancy | Generation | Does the answer actually address the question asked? |
Context precision catches noisy retrieval. If your top-5 chunks include three blocks of navigation boilerplate, precision tanks even when the answer happens to be in chunk four. Low precision usually points at bad content cleaning or chunking, which I covered in more depth in the text chunking post.
Context recall catches missing information. The retriever returned five perfectly relevant chunks, but the one containing the actual answer never made it into the top-k. High precision plus low recall is the classic signature of an index that's too coarse or a top-k that's too small.
Faithfulness catches hallucination. It decomposes the answer into individual claims and checks each one against the retrieved context. This is the single most important metric for anyone shipping RAG to users, because it measures the thing RAG exists to prevent. See what LLM grounding means for why this matters beyond RAG.
Answer relevancy catches evasion. Models under strict grounding instructions sometimes produce faithful non-answers: technically supported by context, useless to the user. Relevancy scores how directly the response addresses the original question.
Notice what's absent: BLEU, ROUGE, exact match. Those compare output text to a reference string, which punishes correct answers phrased differently. RAG metrics compare claims to evidence instead, and that's the right frame.
Running RAGAS in Practice
RAGAS is the most common open-source implementation of these metrics. It uses an LLM as the judge, so scores cost tokens, but the setup is genuinely small:
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
)
from datasets import Dataset
data = Dataset.from_dict({
"question": ["What did Acme charge for the Pro plan in July 2026?"],
"answer": ["The Pro plan costs $49/month as of July 2026."],
"contexts": [["Pricing page snapshot: Pro $49/mo, Team $99/mo..."]],
"ground_truth": ["$49 per month."],
})
result = evaluate(data, metrics=[
faithfulness, answer_relevancy,
context_precision, context_recall,
])
print(result)
Two caveats from experience.
First, LLM judges are noisy. Run the same eval twice and individual scores wobble. Treat scores as trends across 50+ examples, not verdicts on single rows. A faithfulness drop from 0.92 to 0.78 across your whole set means something. A single 0.6 means almost nothing.
Second, context recall requires ground truth answers, which means somebody has to write them. There's no way around this. Budget an afternoon to build a golden set of 50 to 100 questions drawn from real user queries, with short reference answers and the source URL each answer came from. That dataset becomes the most valuable eval asset you own.
The Failure Mode Most Evals Miss: Stale Indexes
Here's the trap. You build your golden dataset in March. Your index was also built in March. Your eval scores look great all year, because you're testing March questions against March data.
Meanwhile the web moved. The pricing page changed in June. The API docs got rewritten. Your retriever still returns the old chunks, faithfulness scores stay high (the answer is faithful to the stale context), and every metric reports green while users get confidently wrong answers.
Faithfulness measures agreement between answer and context. It says nothing about whether the context is still true.
Two fixes, and you want both:
Add freshness checks to the eval itself. For each golden question, store the source URL. At eval time, re-fetch the live page and compare it to what your index holds. If they've diverged, flag the row. A fetch API that returns clean markdown makes the diff trivial; the link.sc fetch endpoint is what I use, and comparing two markdown snapshots is one line of difflib.
Retrieve live where it counts. For volatile sources (pricing, docs, news), skip the standing index and fetch at query time. I walked through that architecture in the live web data RAG post. Live retrieval turns the staleness problem into a latency problem, which is a much better problem to have.
If your eval never touches the live web, it's evaluating a snapshot, not a pipeline.
A Minimal Eval Loop You Can Actually Sustain
The eval setups that survive are boring and scheduled. Here's the loop I'd recommend:
- Weekly: run RAGAS over the golden set. Track the four metrics in a spreadsheet or dashboard. You're watching for deltas, not absolutes.
- Weekly, same job: re-fetch every golden source URL and diff against the indexed version. Stale rows either trigger a re-index or get their ground truth updated.
- On every retriever change: run the eval before merging. Chunk size changes and embedding model swaps are exactly where recall regressions hide.
- Monthly: sample 20 real production queries, judge them manually, and promote the interesting failures into the golden set. Your eval set should grow toward your actual traffic.
That's a few hours of setup and maybe an hour a week of attention. Compare that to the cost of discovering a retrieval regression from a customer complaint.
Start with the four metrics, keep the golden set small and real, and wire freshness into the loop from day one. The teams that measure retrieval separately from generation fix problems in hours. The teams that eyeball answers debug in the dark.
Need fresh, clean web content for your RAG evals and live retrieval? Get a free link.sc API key and start fetching.