Choosing a Chunking Strategy for Bedrock Knowledge Bases

The situation

The knowledge base backs an internal assistant that answers policy and product questions with citations. Three kinds of source material feed it. There are long PDFs, product manuals and onboarding guides running to a hundred pages, written as flowing prose. There are structured policy documents with numbered sections, sub-clauses, and the occasional table (notice periods, fee schedules, eligibility grids). And there are short FAQ entries, a question and a two-sentence answer, hundreds of them exported from the help desk.

All of it was ingested with the default fixed-size chunking: 300 tokens a chunk, a small overlap, split blindly on token count. Retrieval is mediocre. Some answers come back half-formed because the relevant clause was sliced across a chunk boundary and only one half scored highly enough to be retrieved. Others come back buried, because a 300-token chunk swept up three unrelated ideas and the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. averaged them into something that matches nothing well.

Chunking is upstream of embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. and retrieval, so it caps the ceiling of everything downstream. A fact that never gets retrieved cannot be generated, and whether it gets retrieved is decided the moment the document is cut into pieces. Getting the cut right is the highest-leverage change available before touching the model or the promptPromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot. .

What actually matters

Chunk size is the trade nobody escapes. A small chunk embeds a single idea and matches a query precisely, but it may omit the surrounding context the model needs to actually answer, the clause without the section it sits in, the answer without the question it responds to. A large chunk carries that context, but it dilutes the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. : the vector becomes the average of several ideas and matches queries about none of them cleanly, and it burns more of the context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined. when it lands in the promptPromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot. . Precision pulls one way, context pulls the other, and the right point on that line depends on the document.

Boundaries matter as much as size. Splitting mid-sentence or mid-section destroys meaning: half a sentence embeds as noise, and a clause severed from its heading loses what it was about. Overlap is the cheap insurance against this, a modest run of shared tokens between adjacent chunks so a sentence that straddles a boundary survives whole in at least one of them.

Structure-aware chunking beats blind fixed-size on structured documents, because the document already tells you where the seams are: headings, sections, clause numbers, table rows. Ignoring that markup and counting tokens instead throws away the one signal that reliably marks a good cut point.

Two practical constraints sit underneath all of this. A chunk must fit inside the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s maximum input length; a chunk larger than the model accepts gets truncated, and the tail silently never embeds. And changing the chunking strategy is not a config tweak, it forces a full re-ingest and re-embed of the whole corpus, because every chunk boundary moves and every vector changes. That has a cost in dollars and time, and it means you commit to a strategy rather than flip between them casually.

Finally, chunking sets citation granularity. Cite a small child chunk and you point the reader at the exact clause; cite one giant chunk and the citation says “somewhere in this 100-page manual.” The grain of the chunk is the grain of the answer’s provenance.

What we’ll filter on

1. Does it respect document structure (headings, sections, tables), or split blind?
2. Where does it sit on the precision-versus-context trade?
3. Does it handle long and structured documents, including tables?
4. How precise is the resulting citation granularity?
5. What’s the ingestion cost and complexity?
6. Is it available natively in Bedrock Knowledge Bases, or does it need custom code?

The chunking landscape

1. No chunking. Treat each file as a single chunk. This is the right move for material that is already the right size: FAQ entries where the question-and-answer pair is the natural unit, or very short documents that would only be damaged by cutting. It fails hard on long documents, one 100-page PDF becomes one vector that matches everything vaguely and nothing sharply, and it will blow past the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s input limit and truncate.

2. Fixed-size chunking. Split by a target token count with an overlap percentage. Simple, uniform, and completely blind to structure: it cuts at token 300 whether that lands mid-sentence, mid-clause, or mid-table. It’s the default, and it’s a reasonable floor for uniform prose of consistent density. On structured or mixed corpora it produces exactly the failure the team is seeing.

3. Hierarchical chunking. Parent and child chunks. The document is split into large parent chunks (a section) and each parent into small child chunks (a paragraph or clause). Retrieval matches on the small child for precision, then returns the larger parent to the model for context. This is the move that resolves the size trade instead of picking a side of it: precise matching and full context at once. Native in Bedrock Knowledge Bases, and strong on long structured documents where a clause needs its section around it.

4. Semantic chunking. Split at semantic boundaries by measuring embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. similarity between adjacent sentences and cutting where the topic shifts, keeping one coherent idea per chunk. This is strongest on flowing prose that lacks reliable structural markers, the long manual written as continuous narrative. Native in Bedrock Knowledge Bases. It costs more at ingest (it embeds sentences to decide where to cut) and it leans on there being a detectable topic shift to find.

5. Custom chunking via a Lambda transformation. Full control through a transformation Lambda that Bedrock Knowledge Bases invokes during ingestion. Split on markdown headings, keep a table intact as one chunk, apply a domain rule (“never split a clause”), attach custom metadata. Maximum fidelity to the document, maximum code and maintenance. The escape hatch when the managed strategies can’t express the rule you need.

6. Advanced parsing. Before any of the above, use a foundation model to parse complex documents, tables, images, multi-column layout, into clean linearised text. Bedrock Knowledge Bases offers an FM parsing option for exactly this. It isn’t a chunking strategy on its own; it’s a pre-step you pair with one of the strategies above so that a table arrives as coherent text rather than as scrambled tokens the chunker then mangles further.

Side by side

No single row is the answer for the whole corpus. The three source types want different cuts, and the honest configuration matches the strategy to the document type rather than forcing one strategy across everything.

Three ways to cut one document

The pick in depth

Match the strategy to the document type. The mixed corpus doesn’t have one answer, it has three ingestion configurations, one per source.

Hierarchical for the long structured policy PDFs. This is the workhorse choice for the material causing the pain. Set a parent size that captures a whole section (say 1,500 tokens) and a child size that isolates a clause or paragraph (say 300 tokens). A query about a specific clause matches the child, which embeds that one idea cleanly, and Bedrock returns the parent so the model reads the clause with its section around it. Precision from the child, context from the parent, and a citation that points at the clause rather than the manual. This is the single change most likely to fix the team’s retrieval.

Semantic for the flowing-prose manuals. Where a document is continuous narrative without reliable headings, semantic chunking finds the topic shifts and keeps each idea whole, which fixed-size can’t do and hierarchical only approximates through size. It costs more at ingest, but for prose that doesn’t expose structure it retrieves noticeably better.

Fixed-size for the uniform short content. The FAQ entries and other short, consistent material don’t need anything cleverer. Fixed-size with a modest overlap, or no chunking at all when each entry is already one chunk, is correct here; reaching for hierarchical or semantic would be complexity with no payoff.

Custom Lambda when structure must be preserved exactly. If a policy document has tables that must stay intact as one chunk, or a rule like “never split a numbered clause,” the managed strategies can’t express it and a transformation Lambda can. Reach for it when a specific structural guarantee matters, not by default.

Advanced parsing first when documents carry tables or images. Run the FM parsing option before chunking on any source with tables or figures, so a fee schedule arrives as clean linearised text rather than scrambled tokens. Parsing and chunking are separate decisions; parse to clean up the input, then chunk the result with whichever strategy fits.

On overlap: a modest overlap (roughly 10 to 20 percent) keeps boundary sentences from being orphaned without bloating the vector count. More than that mostly duplicates content and inflates cost.

The gotchas are consistent. Changing chunking means re-ingesting and re-embedding the entire corpus, budget the cost and the time, and don’t treat it as a live toggle. Parent and child token sizes both need setting deliberately; defaults are a starting point, not an answer. No chunk may exceed the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s maximum input length, or its tail truncates and silently never embeds. Chunks that are too small starve the model of the context it needs to answer even when retrieval is perfect. And always re-evaluate retrieval after changing the chunking, a citations-required RAG pipeline lives or dies on whether the right chunk comes back, and the only way to know a chunking change helped is to measure retrieval before and after on a fixed set of questions.

A worked example: the notice-period clause

A subscriber-facing policy PDF has a section headed “Termination” with several numbered clauses. One reads: “4.3 Early termination. A party may terminate before the end of the term by giving no less than sixty (60) days’ written notice to the other party. Notice takes effect on receipt.” The section around it defines what “the term” means and how notice is served.

Under fixed-size chunking at 300 tokens, the cut landed after clause 4.2 and clause 4.3 straddled a chunk boundary: “A party may terminate before the end of the term by giving no less than” closed one chunk, and “sixty (60) days’ written notice to the other party” opened the next. A query for “what is the notice period for early termination?” embeds cleanly, but neither half-chunk contains both the trigger and the number, so the chunk that scores highest is the one with the phrase “early termination” and not the one with “sixty (60) days.” The model gets the topic and misses the answer.

Query: "what is the notice period for early termination?"

Fixed-size, top retrieved chunk:
  "...4.3 Early termination. A party may terminate before the end
   of the term by giving no less than"
  -> topic matches, the number is in the NEXT chunk. Answer: incomplete.

Re-ingested with hierarchical chunking, the parent chunk is the whole “Termination” section and clause 4.3 is one child chunk. The child embeds the complete clause, trigger and number together, and matches the query precisely. Bedrock returns the parent, so the model also sees how “the term” is defined and how notice is served.

Query: "what is the notice period for early termination?"

Hierarchical, matched child chunk:
  "4.3 Early termination. A party may terminate before the end of
   the term by giving no less than sixty (60) days' written notice
   to the other party. Notice takes effect on receipt."
  -> clause complete. Parent (the Termination section) returned for context.
  Answer: "Sixty days' written notice, effective on receipt."

Same document, same embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model, same query. The only thing that changed was where the document got cut, and that was the difference between a wrong answer and a cited, correct one.

What’s worth remembering

1. Chunking is upstream of everything; it caps the ceiling of retrieval and generation, because a fact that never gets retrieved cannot be generated.
2. Chunk size is a trade: small chunks match precisely but drop context, large chunks carry context but dilute the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. and waste the window.
3. Hierarchical chunking resolves the trade instead of picking a side, match the small child for precision, return the large parent for context; it’s native in Bedrock and the strongest default for long structured documents.
4. Semantic chunking is for flowing prose without reliable structure; it cuts on topic shift and keeps one idea per chunk.
5. Fixed-size (or no chunking) is fine for uniform short content like FAQ entries; don’t add machinery that buys nothing.
6. Reach for a custom Lambda transformation only when a specific structural rule (keep tables intact, never split a clause) has to be guaranteed.
7. Run advanced FM parsing before chunking on any document with tables or images, so the chunker sees clean text.
8. A chunk must fit the embeddingEmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. model’s max input, or its tail truncates and never embeds; a modest overlap keeps boundary sentences whole.
9. Changing chunking forces a full re-ingest and re-embed of the corpus; it’s a committed decision with real cost, not a live toggle.
10. Always re-evaluate retrieval after a chunking change against a fixed question set; measuring before and after is the only way to know it helped.

The mixed corpus doesn’t get one strategy, it gets three: hierarchical for the structured PDFs, semantic for the flowing manuals, fixed-size for the FAQ entries, with FM parsing in front of anything holding a table. Match the cut to the document, re-embed, and measure the retrieval you get back.