The situation
An internal policy assistant runs on Amazon Bedrock over a Knowledge Base of HR and finance documents. Employees ask it things like “how many days of carer’s leave am I entitled to” and “what is the mileage reimbursement rate”, and it retrieves a few passages and generates an answer with citations. Most of the time it is genuinely useful. Perhaps one answer in fifteen is confidently, specifically wrong: a leave figure that appears nowhere in the policy, a reimbursement rate from a document that was superseded two years ago, a crisp paragraph about a benefit the company does not offer.
The team already measures the pipeline end to end and knows the overall answer quality is not where they want it. What they cannot currently do is say why any single bad answer went wrong. Sometimes the retrieved passages plainly did not contain the answer and the model made something up anyway. Sometimes the right passage was sitting in the context and the model still drifted past it. Those are two different failures with two different fixes, and right now both land in the same “it hallucinated” bucket, so nobody can tell whether to spend the next week on ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. and retrieval or on the generation prompt and its guardrails.
Underneath the complaint is a measurement problem. Before you can reduce hallucination you have to define it precisely enough to count it, and attribute each instance to the stage that caused it.
What actually matters
The first distinction to get right is between faithfulness and correctness, because they are not the same property and a system can pass one while failing the other. Faithfulness, sometimes called groundedness, asks a narrow question: is every claim in the answer supported by the retrieved context? Correctness asks a broader one: is the answer actually right? An answer can be perfectly faithful to a passage that is out of date, and be wrong in the world. An answer can be right by luck, pulled from the model’s parametric memory rather than the retrieved text, and be unfaithful even though it happens to be correct. Faithfulness is reference-free; you can judge it with only the answer and the passages in front of you. Correctness needs a known-good reference answer to compare against. Conflating them is how teams end up chasing the wrong metric.
The second thing that matters is that a hallucination has two possible origins, and you cannot fix what you cannot locate. If retrieval returned passages that never contained the answer, the model was set up to fail; the honest move at that point is to abstain, and a fabricated answer is really a retrieval failure compounded by a refusal-to-abstain failure. If retrieval returned a passage that did contain the answer and the model still asserted something the passage did not say, that is a generation failure, and no amount of better retrieval will help. Attribution is the whole game. A grounding score of 0.4 tells you the answer is unsupported; it does not tell you whether the support was never retrieved or was retrieved and ignored.
Third, decide whether you are measuring at runtime or offline, because they serve different purposes. A runtime check scores each response as it is produced and can block or flag a low-scoring answer before it reaches the employee; it is a live safety gate with a latency and cost budget attached to every call. An offline evaluation runs a fixed set of questions on a schedule or before a release and gives you a trend line and a regression signal; it is where you catch a new embedding model or a reworded prompt quietly making things worse. You want both, and they use different tooling.
Fourth, the eval set has to include questions the system should not be able to answer. If every question in your set is answerable from the corpus, you never test the behaviour that matters most for hallucination: abstaining when the context is insufficient. A system that always produces a confident answer will score fine on an all-answerable set and fabricate freely in production the moment it meets a question outside the corpus. Known-unanswerable questions, where the correct output is “I do not have that information”, are the ones that expose a model that would rather invent than decline.
And finally, whatever automated judge you use is itself a model that can be wrong, so it needs validating against human labels before you trust its numbers. An LLM scoring groundedness is cheaper and faster than a human reviewer and drifts in its own ways; you calibrate it by having people label a sample and checking the judge agrees, then let it scale.
What we’ll filter on
- Faithfulness or correctness, is the method scoring support-by-the-context, or right-in-the-world? They need different inputs.
- Attribution, can it separate a retrieval-caused hallucination from a generation-caused one, or does it collapse both into one score?
- Runtime gate or offline signal, does it block a bad answer live, or track quality across a fixed eval set?
- Ground truth required, does it need reference answers and labelled unanswerable cases, or can it run reference-free?
- Cost and latency, what does each check add per answer, and can the budget carry it on every call?
The measurement landscape
Amazon Bedrock Guardrails contextual grounding check. A guardrail policy that scores a model response against two things: a grounding source (the retrieved passages) and the user query. It returns a grounding score, how well the response is supported by the source, and a relevance score, how well the response addresses the query, each between 0 and 1. You set a threshold on each; when a score falls below its threshold the guardrail intervenes and returns your configured blocked message instead of the ungrounded answer. It runs at runtime, applied inline during the model call or through the ApplyGuardrail API, so it is a live gate rather than an offline report. It measures faithfulness directly through the grounding score and relevance through the relevance score. What it does not do on its own is attribute the failure; a low grounding score tells you the answer is unsupported, not whether the support was retrievable.
LLM-as-a-judge groundedness scoring. A second model prompted to read the answer and the retrieved passages and score whether each claim in the answer is supported. This is reference-free like the grounding check, but you own the rubric, so you can ask it for a per-claim verdict, a short rationale, and a citation to the supporting sentence rather than a single number. That granularity is what lets you build attribution: pair it with a separate check on whether the passage even contained the answer, and you can place each miss. The cost is that you are now running and validating another model, and its scores need calibrating against human labels before they carry weight.
Amazon Bedrock Evaluations, RAG evaluation job. A managed offline evaluation over a Knowledge Base, run in retrieve-and-generate or retrieve-only mode. It computes retrieval metrics such as context relevance and context coverage, and generation metrics including faithfulness, which measures whether the response is grounded in the retrieved passages, alongside correctness and completeness, which compare against reference answers you supply in the dataset. Because it reports retrieval quality and generation faithfulness in the same job, it is the natural place to do attribution at the eval-set level: a question with poor context relevance and low faithfulness points at retrieval, while good context relevance and low faithfulness points at generation. It is a scheduled or pre-release job, not a live gate, and the faithfulness and correctness scoring is itself LLM-as-a-judge under the hood.
Bedrock Evaluations, human evaluation. The same evaluation framework run with human reviewers rather than a model judge, scoring answers on the criteria you define. Slower and more expensive per answer, and the reference standard you calibrate the automated judges against rather than a thing you run on every release.
A labelled eval set with answerable and unanswerable questions. Not a service but the input every other method leans on. A curated set where each question is tagged answerable or unanswerable, answerable ones carry a reference answer and the passage that supports it, and unanswerable ones expect an abstention. This is what turns any of the scoring methods above from a vibe into a number you can track, and it is the only way to measure whether the system abstains when it should.
Side by side
| Method | Measures faithfulness | Measures correctness | Attributes retrieval vs generation | Runtime or offline | Needs reference answers |
|---|---|---|---|---|---|
| Guardrails contextual grounding | ✓ | ✗ | ✗ | Runtime | ✗ |
| LLM-as-a-judge groundedness | ✓ | ✗ (unless given reference) | ✓ (paired with a retrieval check) | Either | ✗ |
| Bedrock RAG evaluation job | ✓ | ✓ | ✓ (retrieval + generation metrics together) | Offline | ✓ (for correctness) |
| Bedrock human evaluation | ✓ | ✓ | ✓ (with the right rubric) | Offline | ✓ |
| Answerable / unanswerable eval set | ✗ (it is the input) | ✗ (it is the input) | Enables it | Offline | ✓ |
Reading the table against the assistant: the runtime gate wants the Guardrails contextual grounding check on every answer; the offline regression signal wants a Bedrock RAG evaluation job on a fixed set; the attribution the team is actually missing comes from reading retrieval quality and faithfulness together, either in that evaluation job or through an LLM judge paired with a retrieval check. No single method does all of it, and the unanswerable questions have to be built by hand whichever way you go.
The picks in depth
For the live safety gate, the contextual grounding check is the right tool because it enforces faithfulness at the moment of answering with no reference data required. You feed it the retrieved passages as the grounding source and the employee’s question as the query, set a grounding threshold that reflects how much unsupported content you will tolerate, and let it block answers that fall below. The mechanics worth knowing: the grounding score and the relevance score are separate levers, so a factually supported answer that wanders off the question can still be caught by the relevance threshold, and an on-topic answer that invents detail is caught by the grounding threshold. Set the thresholds from data, not by guessing, by scoring a labelled sample and finding the cut-off that blocks the fabrications without blocking the good answers. The gotcha is that a strict grounding threshold turns some correct-but-loosely-worded answers into refusals, so tune it against real answers rather than reaching for 0.9 because it sounds safe.
For the offline signal, the Bedrock RAG evaluation job is the pick because it reports retrieval and generation quality in one place over a fixed dataset, which is exactly what a regression check needs. Run it before any change to the embedding model, chunking, RerankingA second pass that re-scores a wide set of retrieved candidates and keeps only the few most relevant, so the expensive model reads less. , or the generation prompt, and watch faithfulness and context relevance as separate lines. The reason this matters more than a single quality score: a change that improves retrieval can drop faithfulness if the model starts over-trusting longer context, and a single blended number would hide the trade. Supply reference answers so the job can also score correctness, because faithfulness alone will happily bless a well-grounded answer that cites a superseded document.
The attribution the team is missing is a cross, not a metric. For each question in the eval set, establish two facts independently. First, did retrieval surface a passage that actually contains the answer? That is a retrieval question, answered by context relevance or coverage against the passage you labelled as the source. Second, is the generated answer faithful to what was retrieved? That is the grounding or faithfulness score. Crossing the two places every failure:
The top half is the branch where retrieval missed. If the model abstained, the system did the right thing on a question it could not answer; if it produced a confident answer anyway, that is a retrieval-caused hallucination, and better retrieval or a firmer instruction to abstain is the fix, not a stricter faithfulness score. The bottom half is the branch where retrieval succeeded: a faithful answer is grounded and correct, and an unfaithful one is a generation-caused hallucination that better retrieval will never touch. Two failures that looked identical in the complaints now point at two different pieces of work. The broader end-to-end view of the pipeline, retrieval Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. , answer relevance, and cost together, is worth reading alongside this narrower hallucination cut; see evaluating a RAG pipeline end to end.
A worked example: two questions through the cross
Take two questions from the labelled set. The first, “what is the current mileage reimbursement rate”, is answerable, and the labelled source is the 2026 expenses policy. The second, “what is the sabbatical policy for contractors”, is unanswerable, because the company has no such policy and no document describes one; the expected output is an abstention.
Run the first. Context relevance is high, the retrieved passages include the 2026 expenses policy, so retrieval surfaced the answer and we are in the bottom branch. The generated answer quotes a rate from a 2024 document that also got retrieved. The grounding check scores it faithful, because the number really does appear in a retrieved passage, but the correctness check against the reference answer fails, because it is the superseded rate. This is the case faithfulness alone would wave through: grounded and wrong. It reads as a generation problem only if you stop at faithfulness; the correctness comparison and the retrieval detail together show the real fix is at retrieval and ranking, keeping the superseded document out of the top passages.
Run the second. Retrieval surfaces some loosely related benefits text but nothing about contractor sabbaticals, so context relevance against the expected source is low and we are in the top branch. The model, rather than abstaining, produces a fluent paragraph describing a three-month unpaid sabbatical available after two years. The contextual grounding check scores it low, because none of that is in the retrieved passages, and at runtime the guardrail would block it. In the offline attribution it lands squarely as retrieval-caused, compounded by a failure to abstain: the fix is to strengthen the instruction to decline when the passages do not support an answer, and to make sure the unanswerable question stays in the eval set so the abstention rate is tracked, not assumed.
Two questions, two branches, two different remedies, and neither remedy is the one you would have reached for from the single sentence “it hallucinated”.
What’s worth remembering
- Faithfulness and correctness are different properties: faithfulness asks whether the answer is supported by the retrieved context, correctness asks whether it is right in the world, and an answer can pass one while failing the other.
- A hallucination comes from one of two stages: retrieval, where the context never held the answer, or generation, where the model ran past what the context said; attribute every miss to its stage before you try to fix it.
- When the context is insufficient the correct behaviour is to abstain; a confident answer over missing context is a retrieval failure compounded by a refusal-to-abstain failure, not a generation quirk.
- Amazon Bedrock Guardrails contextual grounding checks score a response against the source and the query, return a grounding score and a relevance score between 0 and 1, and block answers below your thresholds at runtime.
- Set grounding thresholds from a labelled sample, not by guessing; too strict and you turn correct-but-loosely-worded answers into refusals.
- Bedrock RAG evaluation jobs report retrieval metrics and generation faithfulness together over a fixed set, which makes them the place to do attribution offline and to catch regressions before a release.
- Faithfulness scoring is reference-free, but correctness needs reference answers supplied in the eval dataset, so build both into the set.
- Include known-unanswerable questions in the eval set; they are the only way to measure whether the system abstains rather than fabricating when a question falls outside the corpus.
- An LLM-as-a-judge groundedness score is cheaper than human review but is itself a model that can drift, so calibrate it against human labels on a sample before trusting the numbers.
- Cross two independent facts per question, did retrieval surface the answer and is the answer faithful to what was retrieved, and every failure sorts cleanly into retrieval-caused or generation-caused.