The situation
A support-knowledge assistant on Amazon Bedrock is answering staff questions from a Knowledge Base built over product manuals, an internal wiki, and a table of parts. It works well enough in demos and then falls over in specific, repeatable ways. A question about error code E-4471 returns a passage about a different code entirely. A question that quotes a manual almost verbatim retrieves a vaguely related section instead of the exact one. An engineer asking about the HX-200 pump gets an answer about the HX-2000, a different product. And once, after the manuals were updated, the assistant kept citing a procedure that had been rewritten a week earlier.
Every one of these is a retrieval failure, not a generation failure. The model is doing exactly what it was asked: summarising the chunks it was handed. The chunks were wrong. When the right passage never reaches the model, no amount of prompt tuning fixes it, because the model cannot cite what it never saw.
The useful question is not “why is the model wrong” but “which stage of retrieval handed it the wrong passage, and what does each symptom point to.” Retrieval has a handful of distinct failure modes, each with a recognisable signature, and the diagnosis is mostly a matter of reading the signature.
What actually matters
The first thing worth naming is that “wrong chunk” is at least three different bugs, and they need different tests to tell apart. The right passage might not be in the index at all, so nothing could ever retrieve it. It might be in the index but score poorly against the query, so it never enters the candidate set. Or it might be retrieved into the candidate set and then ranked below the cut-off, so it is fetched and thrown away. These are Recall (retrieval)The share of genuinely relevant passages a search actually returns – what you lose when you retrieve fewer chunks. , similarity, and ranking failures respectively, and they live at different stages of the pipeline.
The cheapest diagnostic separates them fast. Take a failing query, widen the candidate count dramatically (ask for fifty or a hundred results instead of five), and look for the passage you expected. If it appears at rank forty, it was always retrievable and the problem is ranking or the cut-off, so RerankingA second pass that re-scores a wide set of retrieved candidates and keeps only the few most relevant, so the expensive model reads less. or a larger k is the lever. If it never appears even at a hundred, the problem is upstream: the embedding does not place the query near the document, or the passage is not in the index at all. That single test splits the failure space in half before you touch any configuration.
The second thing that matters is that the embedding is only as good as what went into it. A chunk that packs five unrelated topics produces one averaged vector that represents none of them sharply, so it matches broadly and precisely nothing; the specific answer inside it gets diluted. A chunk cut so small that it loses its surrounding context embeds a fragment that no longer means what the whole meant, so a pronoun-heavy sentence with the subject three lines up retrieves for the wrong thing. Chunk size is not a tidiness preference; it decides what each vector can represent.
The third is that semantic similarity and exact-token matching are different tools, and a lot of “wrong chunk” cases are a semantic system being asked to do a keyword job. Error codes, SKUs, part numbers, version strings, and API method names carry meaning in their exact characters, and an embedding model treats E-4471 and E-4470 as nearly identical because they look and read almost the same. Semantic search is built to ignore surface differences, which is exactly wrong when the surface is the signal. That is what hybrid search fixes: run a keyword match alongside the vector search and fuse the results, so the exact token gets a vote.
The fourth is that retrieval scope is a correctness property, not a nicety. If the index mixes tenants, product lines, or document versions and nothing filters them at query time, the nearest vector might be from the wrong tenant or a superseded manual. It can be semantically perfect and still the wrong answer, because relevance is not just about meaning; it is about which slice of the corpus the query is entitled to see. Metadata filtering is how you enforce that slice before similarity is even considered.
And the fifth, easy to forget: an index reflects the documents as of the last sync. If the source changed and the ingestion job did not re-run, retrieval is faithfully returning stale content. No similarity setting fixes an index that is describing last week’s manual.
What we’ll filter on
- Recall stage. Is the expected passage even in the index, and does it come back when the candidate count is widened far past the normal cut-off?
- Embedding fidelity. Does each chunk represent a single coherent idea, or is the vector averaging several topics or missing the context that gives a fragment its meaning?
- Metric agreement. Does the index’s distance metric match what the embedding model was trained and normalised for?
- Lexical exactness. Does the query hinge on exact tokens (codes, SKUs, versions) that semantic similarity will smear together?
- Scope correctness. Are results confined to the tenant, product, and document version the query is entitled to, or is the corpus unfiltered?
- Freshness. Does the index reflect the current source documents, or has the source moved on since the last ingestion?
The failure landscape
Distance-metric mismatch. An embedding model is trained for a particular notion of distance, and the vector index has to agree. Amazon Titan Text Embeddings V2 and Cohere Embed produce vectors intended to be compared by Cosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”.
, and Titan V2 returns normalised vectors by default. If the underlying index is configured for Euclidean (L2) or raw inner product when the model expects cosine, the ranking is subtly wrong: results are plausible but not the closest, and the symptom is a system that is “mostly right, oddly off” across every query rather than failing on one query type. There is a wrinkle worth knowing: for normalised vectors, cosine and inner product rank identically, because the magnitudes are all one, so a mismatch there is harmless; the danger is inner product or L2 applied to unnormalised vectors, or L2 where the model wanted angle. When Bedrock Knowledge Bases creates the OpenSearch Serverless index for you, it sets the space correctly; the mismatch creeps in when you bring your own index and hand-configure the space_type, or when you swap embedding models without rebuilding the index the old vectors were written for.
Chunks too large. A chunk covering half a manual page embeds as one vector that is the average of everything in it. The single sentence that answers the query is one signal among many, and averaging drowns it, so the chunk matches many queries weakly and the precise one poorly. The symptom is retrieval that returns the right general area but never the sharp answer, and answers that feel padded because the model is summarising a broad chunk. The fix is smaller, more focused chunks; the chunking-strategy choice is its own decision, covered in choosing a chunking strategy for Bedrock Knowledge Bases.
Chunks too small. Cut too fine and a chunk loses the context that gave it meaning. A step that reads “then set it to 40 psi” embeds without the “it” ever being resolved, so it retrieves for pressure questions in general and not for the pump it belongs to. The symptom is fragments that are individually retrievable but useless in isolation, and answers missing the qualifier that lived in the sentence before. The fix is larger or overlapping chunks, or a hierarchical strategy that keeps a parent’s context attached to each child.
Query phrased unlike the documents. Users ask “why won’t it turn on” while the manual says “unit fails to initialise on power-up.” Semantic search closes some of that gap but not all of it, and short or jargon-light queries land far from formally written source. The symptom is that short, colloquial questions miss while verbose, well-phrased ones hit. The fixes are query rewriting (expand or rephrase the query with the model before retrieval) and hybrid search so any shared keywords still contribute.
Exact-match tokens semantic search fumbles. This is the E-4471 and HX-200 case. Codes and identifiers carry meaning in their exact characters, and embeddings treat near-identical strings as near-identical meaning, so the wrong code or the wrong SKU ranks first. The symptom is precise: queries built around an identifier fail while prose queries succeed. The fix is hybrid search that runs a keyword match beside the vector search and fuses the scores, giving the exact token a path to the top; the mechanics are in hybrid search and reranking for Bedrock RAG.
Missing metadata filtering. The index holds several tenants or product lines and the query does not constrain to one, so the nearest neighbour is from the wrong slice. The symptom is answers that are on-topic but from the wrong product, tenant, or version, like the HX-2000 answer to an HX-200 question when both manuals are in one index. The fix is attaching metadata at ingestion (a .metadata.json sidecar per document in a Knowledge Base) and applying a metadata filter at retrieval so only the entitled slice is searched.
No reranking. The right chunk is retrieved into the candidate set but sits at rank twelve while k is five, so it is fetched and discarded before the model sees it. This is the failure the widen-k test exposes instantly. The symptom is that the answer exists in the corpus and shows up when you ask for more results, but not in the normal cut. The fix is a reranking step: retrieve a wider candidate set, then reorder with a Cross-encoderA model that reads a query and a passage together and scores the pair, more accurate than comparing two independently-made vectors. reranker such as Amazon Rerank or Cohere Rerank on Bedrock, and keep the top few after reranking. Reranking scores query and passage together rather than comparing two independent embeddings, so it recovers the right chunk from deep in the candidate list.
Stale index. The source changed and the ingestion job did not re-run, so retrieval returns the old content faithfully. The symptom is answers that were correct and are now citing superseded procedures, with no pattern by query type; it is purely temporal. The fix is re-syncing the data source (an incremental ingestion job picks up the changed and added documents) and, if this recurs, automating the sync so the index does not drift behind the source.
Side by side
Reading the symptom is most of the diagnosis. The table maps each cause to whether the right chunk is in the store, whether it comes back when you widen k far past the cut-off, whether the failure is specific to exact-token queries, and whether the fix requires re-ingesting or re-embedding the corpus rather than a query-time change.
| Cause | Right chunk in the store | Comes back at high k | Exact-token queries only | Fix needs re-ingest |
|---|---|---|---|---|
| Distance-metric mismatch | ✓ | ✗ (ranking skewed everywhere) | ✗ | ✓ (rebuild index) |
| Chunks too large | ✓ (diluted) | ✗ | ✗ | ✓ |
| Chunks too small | ✓ (fragment) | ✗ | ✗ | ✓ |
| Query unlike documents | ✓ | sometimes | ✗ | ✗ |
| Exact-match tokens | ✓ | ✗ | ✓ | ✗ |
| Missing metadata filter | ✓ (wrong slice ranks) | ✗ | ✗ | ✗ (add metadata, then filter) |
| No reranking | ✓ | ✓ | ✗ | ✗ |
| Stale index | ✗ (old content only) | ✗ | ✗ | ✓ (re-sync) |
The single most useful column is “comes back at high k.” A yes points almost uniquely at ranking, so reranking or a larger k is the fix. A no sends you upstream to embedding, metric, scope, or freshness, and the remaining columns split those apart.
The same logic drawn as a decision tree: start at the symptom, answer each test, and arrive at the fix.
The picks in depth
Start every investigation with the widen-k test, because it is free and it halves the search space. Pull a failing query, request a large candidate set, and scan for the passage you expected. If it is sitting at rank thirty, the corpus and the embeddings are fine and you are looking at a ranking problem: add a reranker over a wider retrieval, or raise k if the generation budget allows. If it is nowhere at a hundred, stop tuning k; the problem is that the query and the document do not embed near each other, or the passage is stale or filtered out, and no ranking change will conjure it.
For the metric mismatch, the tell is that everything is slightly off rather than one query class failing. Confirm what the embedding model expects (Titan Text Embeddings V2 and Cohere Embed want cosine, and Titan V2 normalises by default) and confirm the index’s configured space matches. If you let Bedrock Knowledge Bases quick-create the OpenSearch Serverless collection, this is set for you and is rarely the culprit; suspect it when someone hand-built the vector index, or when the embedding model was changed without re-embedding, because old vectors compared under a new model’s assumptions rank nonsensically. The fix for a genuine mismatch is rebuilding the index with the right space and, if the model changed, re-embedding the whole corpus, since vectors from two models are not comparable.
For exact tokens, the fix is hybrid search, and it is worth being precise about why pure semantic search cannot be tuned into doing this. An embedding compresses a string into a dense vector that captures meaning and deliberately discards surface form, so E-4471 and E-4470 collapse toward the same point; that is a feature for synonyms and a bug for identifiers. Hybrid search keeps a lexical index alongside the vector index and fuses the two rankings, so an exact keyword hit on the code can lift the correct passage regardless of what the embedding thinks. In Bedrock Knowledge Bases over OpenSearch Serverless, this is the HYBRID search type rather than SEMANTIC, selectable per retrieval.
For scope errors, metadata filtering is both the fix and a design lesson: relevance is scoped, and the scope must be data the retriever can filter on. Attach the tenant, product line, and version as metadata at ingestion, then filter on them at query time so similarity only ranks passages the query is entitled to. Without the metadata in place first, there is nothing to filter, so this is the one fix that can require re-ingesting to add the fields even though the filtering itself happens at query time.
For reranking, the value is separating retrieval recall from final precision. Let the vector search be generous and return a wide candidate set optimised for recall (get the right chunk in there somewhere), then let a cross-encoder reranker read query and passage together and reorder, so final precision comes from a model that actually compares the pair rather than from the distance between two vectors computed in isolation. Amazon Rerank and Cohere Rerank are available for this on Bedrock. It is the highest-leverage single addition when the widen-k test keeps saying “the chunk was there all along.”
A worked example: the wrong error code
An engineer asks, “what does error E-4471 mean on the HX-200,” and the assistant explains E-4470, a different fault on a different pump. Two symptoms in one query, so run the tree.
Widen k to a hundred and search the candidates for the E-4471 passage. It appears, at rank sixty-three, alongside a cluster of E-447x codes that all embed close together, and the top results are E-4470 and E-4472 from the HX-2000 manual. That tells us three things at once. The right chunk is in the store, so this is not a recall, chunking, or freshness problem. It comes back only at high k because the near-identical codes crowd it out, which is the exact-token signature. And the HX-2000 passages ranking at all means the HX-200 query is not scoped to its product, which is the missing-filter signature.
The fix is two query-time changes, no re-embedding. Turn on hybrid search so a keyword match on the literal string E-4471 fuses with the vector score and pulls the exact code up from rank sixty-three. Add a metadata filter on product = HX-200 so the HX-2000 manual is excluded before ranking even begins, which both removes the wrong-product answers and clears space for the right one. With the corpus and embeddings untouched, the correct E-4471 passage for the HX-200 now lands at the top, because the exact token finally has a vote and the wrong product line is no longer competing.
Had the widen-k test turned up nothing at a hundred, the story would be different: the passage would be missing (a stale index needing re-sync) or diluted into an oversized chunk needing a finer chunking strategy, and no amount of hybrid search or filtering would have helped, because you cannot rerank a chunk that never entered the candidate set.
What’s worth remembering
- “Wrong chunk” is at least three separate bugs: the passage is not in the index (recall), it scores poorly against the query (similarity), or it is retrieved and then ranked below the cut-off (ranking). Each needs a different fix.
- The widen-k test splits the space cheaply: request a large candidate set and look for the expected passage. If it appears deep in the list, the problem is ranking; if it never appears, the problem is upstream.
- A distance-metric mismatch skews ranking everywhere at once, not on one query type. Match the index space to what the embedding model expects; Titan Text Embeddings V2 and Cohere Embed want cosine, and Titan V2 normalises by default.
- For normalised vectors, cosine and inner product rank identically; the real hazard is L2 or raw inner product on unnormalised vectors, or an index reused after swapping embedding models.
- Chunks too large average several topics into one blurry vector; chunks too small lose the context that gave a fragment its meaning. Chunk size decides what each vector can represent.
- Exact tokens like error codes, SKUs, and version strings are a keyword job, not a semantic one; embeddings smear near-identical strings together. Hybrid search gives the exact token a vote.
- Retrieval scope is a correctness property. Attach tenant, product, and version as metadata at ingestion and filter at query time, or a semantically perfect neighbour from the wrong slice will win.
- When the right chunk is retrieved but ranked out, add a reranker over a wider candidate set; a cross-encoder scores query and passage together and recovers it, where two independent embeddings could not.
- An index reflects the last sync. Answers that were right and are now citing superseded content point at a stale index, not a similarity setting; re-sync the data source.
- Query-time fixes (hybrid search, metadata filters, reranking, query rewriting) are cheap and reversible; corpus fixes (rechunking, re-embedding, rebuilding the index for a new metric) are expensive, so read the symptom before you rebuild anything.