The situation
A team has a working retrieval-augmented-generation assistant over a knowledge base of a few thousand support articles and product docs. The documents are chunked, embedded, and stored in a vector index; at query time the retriever pulls the nearest ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. by embedding similarity, pastes them into the prompt as context, and a Claude model on Amazon Bedrock answers from them. The pipeline was stood up quickly, and the retriever returns whatever the starter template set: top-k of 3.
Two complaints have arrived from different directions. Support engineers say the assistant sometimes claims it cannot find an answer that is plainly written in an article they can point to, or worse, answers confidently with a detail that is not in any document. Separately, finance has noticed the Bedrock input-token bill climbing after someone bumped k to 20 to fix the first complaint, and the p95 latency roughly doubled, yet the “cannot find it” answers did not go away and a few new wrong answers appeared.
The knob in the middle of both complaints is the same one: how many chunks the retriever hands the model on each call. Nobody has measured what the right number is; it has been guessed twice and guessed wrong twice.
What actually matters
Top-k is a recall-versus-precision-and-cost trade-off, and both ends of the range fail in their own way. The first thing worth naming is what k actually buys. Retrieval by embedding similarity is imperfect: the chunk that literally contains the answer is not always the single Nearest-neighbour searchFinding the vectors closest to a query vector; at scale it’s approximated, trading a little accuracy for a lot of speed. , because wording differs, the question is phrased unlike the source, or several chunks look similar. Raising k widens the net so the answer-bearing passage is more likely to be somewhere in the set. That is recall, and if recall fails, nothing downstream can recover: the model cannot cite a passage it never received, so it either declines or fills the gap by inventing. A large share of RAG “hallucinations” are really retrieval misses, not generation faults.
So more chunks always helps recall. The reason you do not simply set k to 50 is that every extra chunk costs on three axes at once. It costs money, because each chunk is input tokens on every single call, and input tokens are most of a RAG bill. It costs latency, because a longer prompt takes longer to process. And, less obviously, it costs answer quality, because a bigger context is not a neutral bigger container. Padding the prompt with lower-relevance chunks dilutes the signal: the one good passage now sits among distractors, and the model can be pulled toward a plausible-looking but wrong chunk, or simply lose the relevant one. Long-context models also read the middle of a long context less reliably than the start and the end, so a relevant chunk buried at position 12 of 20 can be effectively skipped even though it was retrieved. This is the lost-in-the-middle effect, and it means precision matters to the generator, not just to the bill.
That gives the shape of the curve. As k rises from very low, answer quality climbs steeply, because you are rescuing answers that were being missed for lack of the right passage. It plateaus once the answer-bearing chunk is reliably in the set. Then, as k keeps rising, quality sags, because you are now adding distractors and diluting rather than adding coverage, while cost and latency keep climbing the whole way. The best k sits at the knee: high enough to clear the recall problem, low enough to stay out of the dilution zone. Where that knee falls is specific to the corpus and the chunking, so it has to be found by measurement, not inherited from a template.
Chunk size is the coupled variable that decides a lot of it. Small chunks are precise but each holds little, so an answer that spans a couple of paragraphs may need several chunks retrieved together to be complete, which pushes the right k higher. Large chunks carry more context each, so fewer of them cover an answer and k can be lower, but each one spends more tokens and drags in more off-topic text around the relevant sentence. You cannot tune k in isolation; a change to chunk size moves the whole curve, so the two are tuned together against the same eval set.
There is a way to get high recall without paying the full generation cost of a big k, and it is worth knowing because it reframes the whole trade-off. Retrieve a wide net cheaply, then re-rank and keep only the best few for the model. The vector search returns, say, the top 25 candidates; a reranker (a cross-encoder that scores each chunk against the query far more accurately than embedding distance does) reorders them by true relevance; you pass only the top 4 or 5 to the model. Recall comes from the wide first pass, precision comes from the reranker, and the model sees a short, high-signal context. Amazon Bedrock Knowledge Bases supports exactly this pattern with a reranking step on the retrieve call, so the wide-net-then-narrow approach is available without hand-building the second stage.
What we’ll filter on
- Recall at k, is the answer-bearing chunk actually in the retrieved set often enough on your own questions?
- Answer quality, do the model’s answers get better or worse as k changes, judged on an eval set rather than by feel?
- Cost per call, how many input tokens does this k spend on every query, and does the quality gain justify it?
- Latency, what does the added context do to p95 response time?
- Chunk-size coupling, is the right k being set for the chunk size actually in use, or inherited from a different one?
- Two-stage option, would a wide retrieve plus a reranker get the recall at a lower generation cost than a single large k?
The retrieval-depth landscape
Very low k (1 to 2). Cheapest and fastest, and fine when chunks are large and self-contained or the corpus is tiny and each answer lives in one obvious place. The failure mode is recall: any question whose answer is not the single nearest neighbour gets a miss, and misses become declines or inventions. On a general knowledge base this is usually too tight.
Moderate k (3 to 8). The working range for most RAG systems with sensibly sized chunks. Enough coverage that the answer passage is usually present, without so much padding that dilution and cost dominate. The exact figure inside this band is the thing worth measuring, because 4 and 8 can differ noticeably in both quality and bill.
High k (10 to 20 plus). Maximises recall and is defensible when chunks are small so an answer needs several to be complete, or when a downstream reranker will trim the set before it reaches the model. Passed raw to the generator, though, it invites the lost-in-the-middle effect and the largest token bill, and past the knee it can lower answer quality rather than raise it. High k is a means to recall, not a goal.
Wide retrieve, then rerank. Retrieve many candidates cheaply, score them with a reranker, keep the best few for the model. Decouples recall from generation cost: the first pass is wide and cheap, the model sees only a short high-signal context. Costs a reranking step in money and a little latency, and needs a reranker in the path, which Bedrock Knowledge Bases provides as a built-in option. The strongest general answer when a single fixed k cannot satisfy both recall and precision at once.
Dynamic / threshold-based k. Rather than a fixed count, keep every chunk above a similarity score, so easy queries with one strong match return few and broad queries return more. Adapts retrieval depth to the question, but a raw similarity threshold is hard to set well and varies by embedding model, so it usually wants a reranker’s calibrated scores to be dependable.
Side by side
| Approach | Recall | Answer precision to model | Token cost | Latency | Best when |
|---|---|---|---|---|---|
| Very low k (1-2) | ✗ | ✓ | Lowest | Lowest | Large self-contained chunks, tiny corpus |
| Moderate k (3-8) | ✓ | ✓ | Low-medium | Low | Most RAG with sensible chunk sizes |
| High k (10-20+) | ✓ | ✗ | High | High | Small chunks, or a reranker trims after |
| Wide retrieve + rerank | ✓ | ✓ | Medium | Medium | Recall and precision both needed at once |
| Threshold-based k | ✓ (varies) | ✓ (varies) | Varies | Varies | Query difficulty varies widely, scores calibrated |
Reading the table against the team’s problem: the starting k of 3 was risking recall on a general knowledge base, and the jump to 20 traded that miss for dilution, latency, and cost without fixing it, because the answer-bearing chunk was now present but buried. The row that resolves both is the wide-retrieve-then-rerank one, or a measured moderate k if a reranker is not on the table.
The picks in depth
Start by measuring recall, because it is the failure that masquerades as hallucination and it is the one you can quantify cleanly. Build a small eval set of real questions paired with the passage that answers each. Run retrieval at several values of k and record how often the answer passage appears anywhere in the returned set. That curve tells you the smallest k that clears the recall problem for your corpus. If recall is still poor even at high k, the fix is not more chunks; it is the embedding model, the chunking, or a reranker, because you are retrieving the wrong things, not too few of them.
With recall understood, tune for the knee rather than the ceiling. Above the k where recall plateaus, extra chunks stop adding coverage and start adding distractors, so answer quality flattens and then declines while cost and latency keep rising. Judge answer quality with an evaluation harness (an LLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. grade or exact-match against expected answers) across a sweep of k values, and pick the lowest k that sits on the quality plateau. That is the point where you have paid for recall and not yet paid for dilution. It is a per-corpus number; a value copied from another system’s blog post is a guess.
Tune chunk size and k together, because moving one moves the other’s best value. If you shrink chunks for precision, expect to raise k so a multi-paragraph answer is still covered; if you enlarge chunks, expect to lower k and watch the per-call token cost per chunk climb. Re-running the same recall-and-quality sweep after any chunking change is the discipline that keeps the two aligned, and skipping it is how a system ends up with a k that fit the old chunk size and nobody remembers why.
When a single fixed k cannot give both recall and precision, reach for the two-stage pattern. Retrieve a wide net, rerank, and pass only the top few to the model. This is usually the highest-quality option for a non-trivial knowledge base, because it puts a short, genuinely-relevant context in front of the generator while still casting a wide enough net to catch the answer. On Bedrock, the Knowledge Bases retrieve and retrieve-and-generate calls expose the number of results and an optional reranking configuration, so both the k and the wide-then-narrow shape are configuration rather than custom code. The trade is a reranking cost and a little latency for a markedly better signal-to-noise ratio in the prompt.
A worked example: finding the knee
The team builds an eval set of 120 real support questions, each tagged with the article passage that answers it, and runs a sweep. At k of 2, recall is 71%: nearly a third of questions never receive their answer chunk, which lines up exactly with the “it says it cannot find it” complaints. Recall climbs to 88% at k of 4, 95% at k of 8, and 97% at k of 15, flattening after that. So recall is essentially solved by k of 8, and k of 2 was the original sin.
Now the quality sweep, graded by an LLM judge against reference answers. Answer quality rises with recall up to k of 8, then, passed raw to the model, dips: at k of 15 several answers latch onto a plausible but wrong chunk, and a couple of correct-at-k-of-8 answers regress because the relevant passage is now sitting in the middle of a long context and getting skipped. Input tokens per call at k of 15 are nearly four times those at k of 4, and p95 latency is up by half. This is the k of 20 experiment, quantified: recall was fine, but dilution, latency, and cost all got worse together.
The decision writes itself from the two curves. Set k at the knee, around 8, if the context is passed straight to the model. Better, retrieve a wide net of 25, add the Knowledge Bases reranker, and pass the top 4 or 5: recall comes from the wide pass, the reranker floats the genuinely relevant chunks to the top so the model reads a short high-signal context, and the token bill lands near the k of 4 level rather than the k of 15 level. The “cannot find it” answers go away because recall is solved, and the confident-but-wrong answers drop because the model is no longer wading through distractors.
What’s worth remembering
- Top-k is a recall-versus-precision-and-cost trade-off; both a too-low and a too-high k fail, in opposite ways.
- Too few chunks misses the answer-bearing passage, and a model cannot answer from a passage it never received, so recall failures show up as declines or invented answers.
- Too many chunks costs input tokens and latency on every call and dilutes the signal, so the relevant passage gets lost among distractors and quality can drop.
- Long contexts are read less reliably in the middle, so a relevant chunk buried deep in a big retrieved set can be effectively skipped even though it was retrieved.
- Answer quality against k climbs to a plateau then sags; the best k is the knee, high enough for recall and low enough to avoid dilution.
- Chunk size and k are coupled: small chunks need a higher k to cover an answer, large chunks need fewer, so tune the two together.
- To get recall without paying the full generation cost, retrieve a wide net and rerank, then pass only the best few chunks to the model.
- Amazon Bedrock Knowledge Bases exposes the number of results and a reranking configuration on its retrieve calls, so both k and the wide-then-narrow pattern are configuration, not custom code.
- Measure the right k on your own eval set of questions paired with answer passages; a value copied from another system is a guess.
- If recall stays poor even at high k, the problem is the embeddings, chunking, or ranking, not the count; adding more chunks will not fix retrieving the wrong ones.