Evaluating a RAG Pipeline End to End

The situation

An internal assistant answers staff questions from a corpus of policy documents, runbooks, and past support threads. Every answer carries citations back to the source passages; that was a hard requirement from the start, covered when the team first built the citations-required retrieval layer. Most days it works. Roughly one answer in twenty is wrong, and “wrong” arrives as a Slack complaint with a screenshot, not a metric.

The team wants to fix the wrong answers. The trouble is they cannot see where the wrongness enters. A retrieval-augmentedRAGA pattern where you retrieve relevant documents at query time and stuff them into the prompt so the model can ground its answer on them. answer passes through two stages, and either one can sink it. The retriever might fetch the wrong passage, or no relevant passage at all, in which case the model was answering blind. Or the retriever might fetch exactly the right passage and the model ignore it, contradict it, or invent a detail that was never there.

Those two failures look identical from the outside. Same wrong answer, same annoyed user. But the retrieval failure lives in chunking, embeddingsEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. , hybrid search, or the value of k, and the generation failure lives in the prompt and the model. Fixing the prompt when the real problem is a retrieval miss changes nothing except your confidence. The team needs an evaluation that scores each half on its own, and one that re-runs on demand so a change to chunking or the reranker can be checked before it ships.

What actually matters

The load-bearing decision is to measure the two stages separately, because they have separate causes and separate fixes. An end-to-end score that says “82% correct” tells you the pipeline is imperfect and nothing about which half to touch.

Retrieval quality is measured against a labelled set: a list of queries, each mapped to the passage or passages that actually answer it. With those labels you get context recall (of the passages that should have been fetched, how many were), context precision (of the passages that were fetched, how many are relevant, and are they ranked near the top), plus the ranking metrics, hit-rate, MRR, and NDCG@k. Recall is usually the one that matters most for a wrong answer: if the right chunk never entered the context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined. , the model never had a chance.

Generation quality is measured given the retrieved context, and the central metric is faithfulness, sometimes called groundedness: does every claim in the answer follow from the passages that were actually retrieved, with nothing invented? Alongside it sit answer relevance (does the response address the question that was asked) and citation correctness (do the cited sources genuinely support the sentences that cite them). The trap worth naming loudly is that faithfulness is not correctness. An answer can be perfectly faithful to a retrieved passage that happens to be the wrong passage. Grade faithfulness against the retrieved context and you learn whether the model behaved; grade correctness against the ground truth and you learn whether the pipeline as a whole got it right. You want both, and you want to know which stage is responsible when they diverge.

All of this rests on a golden dataset: queries paired with ground-truth answers, and, for the retrieval half, the relevant-chunk labels. The labels are the expensive part. Writing a ground-truth answer is quick; deciding exactly which of fifty thousand passages are the relevant ones for a query is slow human work, and it is what makes retrieval measurable.

There is also a split between component evaluation and end-to-end evaluation, and both earn their place. Evaluating the retriever alone is cheap, fully repeatable, and isolates any change to chunking, the embedding model, or a reranker; you change one knob and watch recall@k move with nothing else in the way. Evaluating the whole pipeline measures the answer the user actually sees. Run the component eval constantly and the end-to-end eval to confirm the user-visible result.

What we’ll filter on

1. Does it separate retrieval failures from generation failures, or collapse them into one number?
2. Does it measure faithfulness or groundedness of the answer against the retrieved context?
3. Does it need labelled relevant-chunks, and can it produce retrieval metrics from them?
4. Managed service or custom code to build and maintain?
5. Scale and cost, how many queries can it grade for what outlay?
6. Repeatable as a regression harness gated on every pipeline change?

The evaluation landscape

1. Amazon Bedrock Knowledge Bases RAG evaluation. The managed, AWS-native answer. Point an evaluation job at a Knowledge Base and a dataset of queries; Bedrock scores retrieval quality and response quality together, using an LLM-as-a-judge for the generation metrics (groundedness/faithfulness, relevance, correctness). It can also compare two Knowledge Base configurations head to head, which is exactly the shape of “did switching the chunking strategy help?”. This is the default place to start for the end-to-end split.

2. Bedrock model evaluation with LLM-as-a-judge on the generation step. A model-evaluation job where each record is the tuple of question, retrieved context, and generated answer, and the judge scores faithfulness and answer relevance against a rubric you write. This grades the generation stage in isolation given whatever context you fed it, and it scales to as many records as you can assemble.

3. RAGAS-style metrics through a custom pipeline. The open-source metric family, context precision, context recall, faithfulness, answer relevance, computed in your own code. Maximum flexibility over exactly what gets measured and how; more to write and maintain, and no managed reports or audit trail.

4. Retrieval-only metrics against a labelled relevance set. Recall@k, precision@k, MRR, and NDCG computed directly from the labels, driving the retriever through the Bedrock Knowledge Bases Retrieve API and comparing what comes back to the known-relevant chunk IDs. This is the cheap component eval: no generation, no judge, just the retriever measured against ground truth. It is the harness you re-run every time you touch chunking or embeddings.

5. Human review on a small stratified sample. The highest-fidelity signal, and too slow and costly to run on everything. Its real job is calibration: score a couple of hundred stratified examples by hand and correlate the human scores against the LLM judge, per metric, so you know which of the judge’s numbers to trust.

Side by side

No single row covers the whole pipeline and stays cheap enough to run often. The managed KB evaluation gives the end-to-end split; the retrieval-only harness gives the fast, isolated retriever signal; the human sample calibrates the judge. The working answer stacks them.

The two stages, and where each is graded

The pick in depth

Start with Bedrock Knowledge Bases RAG evaluation for the managed end-to-end split. One job scores both retrieval quality and response quality (groundedness/faithfulness, relevance, correctness) over the golden query set, and it compares two Knowledge Base configurations directly, so “we changed the chunk size, is it better?” becomes a run rather than an argument. That gives the user-visible answer graded on both halves in one place.

On its own, though, the managed job is heavier than you want to run after every small change, and its retrieval signal is only as good as the labels it can infer. So add a cheap retrieval-only recall@k harness against a labelled relevance set. Drive each golden query through the Retrieve API, compare the returned chunk IDs to the known-relevant IDs, and compute recall@k, precision@k, and MRR. No model call, no judge, a few seconds a query. This is the harness you gate on: change the chunking strategy, the embedding model, or add a reranker, and re-run it to see the retriever move in isolation with nothing downstream muddying the number.

Grade faithfulness with an LLM judge on the generation step, feeding it the question, the retrieved context, and the answer, and scoring against a rubric that asks whether every claim is supported by the context and whether the citations point at passages that back the sentences citing them. Then calibrate the judge: score a small stratified sample by hand, a couple of hundred queries spread across topics and across the easy and hard cases, and correlate the human scores against the judge per metric. A strong correlation earns the judge your trust at scale; a weak one on, say, citation correctness tells you to tighten the rubric or lean on human scores there.

Wire the whole thing as a regression harness gated on every pipeline change. The retrieval-only run is fast enough for a pull request; the full KB evaluation and the judge run on a schedule or before a config ships. The rule to hold onto: any change to chunking, the embedding model, or the reranker invalidates every prior retrieval number, so re-evaluate rather than assume.

A few gotchas earn their keep. Faithfulness is not correctness, so an answer that is faithful to a wrong retrieved passage will score well at gate B and still be wrong; that is precisely the case gate A catches. Retrieval recall needs labels, and labels are costly, so bootstrap them by having a strong modelModelA trained set of weights plus the architecture that makes them useful – the thing you load up and run inference against. propose the relevant chunks for each query and a human verify the proposals; verifying a shortlist is far faster than searching the corpus cold. Watch judge bias, both the tendency to favour longer answers and the tendency to favour outputs that resemble the judge. And the value of k sets the ceiling on recall: too small and relevant chunks fall off the list before generation ever sees them, too large and you dilute precision and pay for tokensTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word. the model does not need.

A worked example: the wrong answer was a retrieval miss

A recurring complaint: staff asking about the parental-leave top-up policy get an answer that quietly states the wrong eligibility window. The instinct is to blame the prompt, tighten the instruction to stick to the source, and ship. Run both gates first.

Policy-eligibility query set (40 queries), before any fix:

  Gate A · retrieval
    recall@5:      0.55      relevant chunk fetched in ~half of cases
    precision@5:   0.38
    MRR:           0.41

  Gate B · generation (graded on retrieved context)
    faithfulness:  0.93      answers stick closely to what was fetched
    answer relev.: 0.90

  End-to-end correctness (vs ground truth): 0.60

The shape is the whole story. Faithfulness is high, so the model is behaving: it is answering from the passages it was handed, inventing almost nothing. But recall@5 is 0.55, so nearly half the time the passage that actually contains the eligibility window never reached the model. The wrong answer is a retrieval miss, not a generation fault. A prompt change would move gate B, which is already fine, and leave the real problem untouched.

The fix belongs upstream. The eligibility details lived in a table that the fixed-size chunker had split mid-row, and pure vector search kept missing the exact policy term. Re-chunk on document structure so tables stay whole, add hybrid search so the keyword “top-up” pulls its weight, and re-run the retrieval-only harness.

Same 40 queries, after structure-aware chunking + hybrid search:

  Gate A · retrieval
    recall@5:      0.88      (+0.33)
    precision@5:   0.61      (+0.23)
    MRR:           0.74      (+0.33)

  Gate B · generation
    faithfulness:  0.93      unchanged, as expected
    answer relev.: 0.91

  End-to-end correctness: 0.86   (+0.26)

Correctness jumped because the retriever now delivers the right passage; generation never needed touching. Had the team read only the end-to-end score, they would have seen 0.60, guessed at the prompt, and watched the number refuse to move. The two-gate split named the stage, and the fix landed where the failure actually was.

What’s worth remembering

1. A RAG answer can fail in retrieval or in generation, and the two have different fixes, so measure them separately or you will fix the wrong half.
2. Retrieval is graded against labelled relevant chunks: recall@k (did we fetch it), precision@k (is what we fetched relevant), MRR and NDCG (is it ranked high).
3. Generation is graded against the retrieved context: faithfulness, answer relevance, and citation correctness.
4. Faithfulness is not correctness; an answer can be perfectly faithful to a passage that was the wrong passage to retrieve.
5. Bedrock Knowledge Bases RAG evaluation gives the managed end-to-end split and can compare two KB configurations directly.
6. A cheap retrieval-only recall@k harness against a labelled set isolates the retriever, so you can re-run it on every chunking, embedding, or reranker change.
7. Labels are the expensive input; bootstrap them by having a strong model propose relevant chunks and a human verify the shortlist.
8. Calibrate the LLM judge against a small stratified human-scored sample, per metric, before trusting its numbers at scale.
9. Any change to chunking, embeddings, or the reranker invalidates prior retrieval numbers, so re-evaluate rather than assume, and gate the harness on the change.
10. Read both gates before choosing a fix: low retrieval points at search, high retrieval with low faithfulness points at the prompt or model.