Cutting Ingestion Cost by Caching and Batching Embeddings

The situation

A knowledge team runs a retrieval-augmented assistant over an internal corpus: product docs, support runbooks, policy pages, and a wiki that a few hundred people edit. Roughly 400,000 chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. sit in the vector index. The corpus is large but calm; on a typical day a few dozen pages change, a release week touches a few hundred, and the rest is untouched for months.

The ingestion pipeline does not know any of that. Every nightly run re-chunks the entire corpus, sends all 400,000 chunks to an embedding model on Amazon Bedrock, and rewrites every vector into the store. The embedding bill is the same on a quiet Tuesday as on a release Friday, because the pipeline embeds everything regardless of what actually moved. Each run also takes hours, so a doc edited at 09:00 is not searchable until the next night, and a one-line fix pays to re-embed the 400,000 chunks around it.

Two more things are hiding in the corpus. A standard legal footer and a boilerplate “how to raise a ticket” block are pasted into hundreds of pages, so the pipeline embeds the same text hundreds of times and stores hundreds of near-identical vectors. And the model in use supports several output dimensionsEmbedding dimensionHow many numbers each embedding vector holds – fewer means a smaller, cheaper, faster index and slightly blurrier matching. , but the pipeline takes the largest one by default, so every vector is bigger than the retrieval quality needs, and the storage and query cost carry that weight on every search. The question underneath all of this is the same: what is worth embedding, and how often?

What actually matters

Embedding cost is a per-token charge levied on the text you send, once per call, every time you send it. That single fact reframes the whole pipeline. The vector for a chunk that has not changed is deterministic given the same model and input, so paying to recompute it is paying to arrive back where you started. The lever that matters most is not a cheaper model or a faster machine; it is sending less text to the model in the first place, and sending each distinct piece of text as few times as possible.

The dominant axis is how often the data changes relative to how often you re-embed. A corpus that turns over completely every day has little to save from change detection, because almost everything is genuinely new. A corpus that is 99% stable between runs, like this one, is spending almost its entire embedding bill re-deriving vectors it already holds. The wider that gap, the more an incremental approach pays, and the bookkeeping it costs is small against what it saves.

The second axis is duplication within the corpus. Embedding is a pure function of the input text, so two identical chunks produce the same vector, and embedding both is redundant by definition. Boilerplate, shared footers, and copy-pasted sections mean the same text is paid for many times over and stored many times over, inflating both the embedding bill and the index. De-duplication addresses both at once, but it needs care: “identical” is safe to merge, whereas “near-identical” is a judgement call, and merging two chunks that differ in the one clause that matters quietly loses a distinction retrieval depended on.

The third is request shape. Embedding models on Bedrock differ in how many inputs they accept per call. Where a model takes a batch of inputs in one request, packing many chunks per call cuts the per-request overhead and lifts throughput, so a backlog of new chunks embeds in fewer round-trips. Where a model takes one input per call, the win comes from concurrency rather than batch size, and the pipeline’s job is to keep the model busy without tripping throttling limits.

The fourth is what the vectors cost after they are made. Embedding is paid once at ingestion; storage and search are paid continuously. A larger embedding dimension means a bigger vector in the store and more work per similarity comparison on every query, for the life of the index. Some models let you request a smaller dimension, trading a little retrieval precision for a smaller, cheaper, faster index. That is a downstream saving that dwarfs the embedding charge over time, and it is set at ingestion, so it belongs in this conversation even though it is not an embedding cost.

The connective idea is that ingestion is a cache-and-diff problem before it is a machine-learning one. The embedding model is an expensive pure function, and the whole game is memoising it: detect what changed, skip what did not, collapse what repeats, pack what remains, and store the result no larger than retrieval needs.

What we’ll filter on

1. Change rate, what fraction of the corpus actually changes between runs?
2. Duplication, how much of the corpus is identical or near-identical text?
3. Change detection cost, is there a reliable, cheap way to tell a changed chunk from an unchanged one?
4. Batching support, does the embedding model take many inputs per request, or one?
5. Downstream cost, how much do storage and per-query search cost over the index’s life?
6. Correctness risk, does the technique ever drop or merge something that should have stayed distinct?

The ingestion-cost landscape

Re-embed everything. The baseline the pipeline is on. Every run embeds the full corpus, so cost scales with corpus size rather than with change. It is simple and stateless, and it is correct in the sense that every vector always reflects the current text. On a calm corpus it is almost pure waste, and the run time grows with the corpus, so freshness gets worse as the index grows.

Incremental processing with content hashing. Compute a stable hash of each chunk’s text, keep a record of the hash you last embedded, and on each run embed only the chunks whose hash is new or changed, plus delete vectors for chunks that vanished. Cost now scales with change, not corpus size, which is the whole prize on a stable corpus. The cost is bookkeeping: a store of hashes to maintain, and the discipline that the hash covers exactly the text sent to the model, so a formatting-only change does not needlessly count as a change while a real edit always does. Chunk-boundary shifts are the sharp edge, since re-chunking a page can change every chunk’s text even where the words did not move.

Managed incremental sync. Amazon Bedrock Knowledge Bases sync a data source into a managed vector index and, after the first full ingestion, re-embed only the documents that changed since the last sync rather than the whole source. A sync is a StartIngestionJob call naming the knowledge base and the data source, and the job reports back statistics that are the incremental picture in numbers: documents scanned against documents newly indexed, modified, deleted, and skipped. That is the same content-diff idea, run for you: the service tracks what it has ingested and skips unchanged documents on later syncs, so you get incremental behaviour without building the hash store yourself. The trade is less control over chunking and change granularity than a hand-rolled pipeline, in exchange for not owning the bookkeeping.

Caching embeddings on a content hash. Sit a cache in front of the embedding call, keyed on the hash of the chunk text. Before embedding, look the hash up; on a hit, reuse the stored vector and never call the model; on a miss, embed and write the vector back under that key. This is the mechanism that makes both incremental sync and de-duplication concrete, and it means an unchanged chunk, or a chunk identical to one already seen, is never embedded twice across runs. The cache is the memoisation table for the expensive pure function.

De-duplication. Collapse repeated text so it is embedded once. Exact de-duplication falls straight out of hash-keyed caching: identical chunks share a hash, so the second and every later copy is a cache hit. Near-duplicate detection goes further, treating chunks that differ only trivially as one, which saves more but introduces the risk of merging things that should stay separate. Exact dedup is close to free and close to safe; near-duplicate dedup is a tuning decision with a correctness cost attached.

Batching inputs per request. Where the embedding model accepts multiple inputs per call, send new chunks in batches rather than one request per chunk. Fewer requests means less per-call overhead and higher throughput, so the backlog of changed chunks clears faster and cheaper. This does not change the per-token embedding charge; it cuts the overhead around it and the wall-clock time. Which side of that line you are on is a property of the model, and it is worth checking rather than assuming: Amazon Titan Text Embeddings V2 takes a single inputText string per call, up to 8,192 tokens, so there is no batch to pack, whereas Cohere’s Embed models take a texts array and embed the whole array in one request. Where the model takes one input per request, the equivalent lever is bounded concurrency, and the limit to stay under is a requests-per-minute quota rather than a token-per-minute one, which is exactly the quota a one-chunk-per-call model burns fastest.

Smaller embedding dimension. Where the model supports configurable output dimensions, request a smaller vector. This shrinks the index, cuts storage, and speeds every similarity comparison at query time, for a modest and measurable loss of retrieval precision. It is the one lever here aimed squarely downstream: it barely touches the embedding bill and mostly pays back in storage and search over the life of the index. Because it is fixed at embedding time, changing it later means re-embedding, so it is worth settling early.

Side by side

Reading the table against this corpus: the calm-but-large shape makes incremental processing the biggest single win, content-hash caching is the mechanism that delivers it and exact dedup for free alongside, batching clears the changed-chunk backlog faster, and a smaller dimension trims the storage and query bill that every search pays. Near-duplicate dedup is the one to reach for last, and only with a measured threshold.

The picks in depth

Incremental processing is the change that resets the cost curve, so it comes first. Give every chunk a stable content hash over exactly the text that goes to the model, keep the hashes you have already embedded, and on each run compute the set difference: embed the new and changed hashes, delete vectors whose chunks are gone, and leave the rest alone. The bill now tracks the few hundred chunks a release touches instead of the 400,000 that did not move, and the nightly run that took hours takes minutes, which is what makes same-day freshness possible. Once the run is that cheap, the clock stops being the natural trigger. Point an S3 event notification at the bucket the documents land in and have it invoke a Lambda that hashes, embeds, and upserts just the object that changed, and the lag from edit to searchable drops from hours to seconds while the token bill stays the same, because the set of changed chunks is the set of changed chunks whenever you process it. The nightly sweep then survives as reconciliation, catching deletes and anything the event path dropped, rather than being the only way in. The failure to guard against is the hash covering the wrong thing. Hash the normalised chunk text the model sees, not the rendered HTML or a timestamped wrapper, or a cosmetic change re-embeds the world and a real edit slips through. Re-chunking is the other trap: if a boundary shift rewrites neighbouring chunks, they legitimately count as changed, so change chunking strategy deliberately and expect a full re-embed when you do.

If the corpus can live in a managed source, Bedrock Knowledge Bases give you the incremental behaviour without building the hash store. The first sync embeds everything; later syncs re-embed only the documents that changed since the last one and update the managed index in place, so the service does the diffing and skips unchanged documents for you. You trade fine control over chunking and change granularity for not owning that bookkeeping, and for a standard doc corpus that is usually the right trade. A hand-rolled hash-and-cache pipeline is the answer when you need control the managed sync does not give, like custom chunking tied to your own change signal.

The embedding cache is the piece that makes both concrete, and it is worth seeing as one idea doing two jobs. Keyed on the content hash, it turns “has this chunk changed since last run” and “have I already embedded this exact text anywhere” into the same lookup. A hit returns a stored vector and skips the model entirely; a miss embeds once and writes back. That single table gives you cross-run incrementality and exact de-duplication together, so the legal footer pasted into 300 pages is embedded once and served 299 times from cache. Exact dedup carries almost no correctness risk because identical text has an identical vector by definition. Near-duplicate dedup is a separate, more aggressive step: collapsing chunks that are merely similar saves more but can merge two policy paragraphs that differ in the one clause a user will search for, so treat it as a tuned decision with a similarity threshold you have measured against real queries, not a default.

Batching is throughput, not unit price, and the distinction matters. It does not lower the per-token embedding charge; it lowers the per-request overhead and the wall-clock time by packing many chunks into each call where the model supports batched inputs, so a release-week backlog of changed chunks embeds in far fewer round-trips. Where the model takes a single input per request, the same goal is met with bounded concurrency, keeping enough calls in flight to stay busy without breaching the model’s throughput limits and earning throttling. Either way the aim is the same: clear the set of genuinely-new chunks quickly and cheaply, now that incremental processing has made that set small.

The jobs where the set is not small are the ones worth treating differently: the first build of the index, and the full re-embed that a chunking change or a dimension change forces. Nothing is waiting on those, so the synchronous path is the wrong one. Write the chunks as JSONL to S3, submit a CreateModelInvocationJob naming the embedding model and the input and output locations, and collect the vectors from the output prefix when the job finishes. Batch inference is priced at half the on-demand per-token rate, and Titan Text Embeddings V2 is one of the models it covers, so the same 400,000 chunks cost half as much to embed as they would through the synchronous path. It is the wrong tool for tonight’s forty chunks, where waiting on an asynchronous job buys nothing, and the right one for the full re-embed.

The embedding dimension is the lever pointed at the forever-cost. Embedding is paid once; storage and per-query search are paid on every vector for as long as the index lives. Where the model offers configurable output dimensions, a smaller vector shrinks the index and speeds every similarity comparison, for a precision loss you can measure and decide is acceptable. Because the dimension is baked in at embedding time, changing it later is a full re-embed, so choose it early against a retrieval-quality check rather than defaulting to the largest size and paying for it on every search thereafter.

A worked example: the nightly run, before and after

Take the calm-Tuesday case: 400,000 chunks in the index, 40 chunks changed today, and a legal footer that appears on 300 pages as its own chunk.

Before. The pipeline re-chunks everything and embeds all 400,000 chunks. It pays to embed the 40 that changed, the 399,960 that did not, and the footer 300 times over. The run takes hours, the bill is flat regardless of how little moved, and today’s edit is not searchable until tomorrow night. The vectors are stored at the model’s largest dimension, so every one of the nightly queries afterwards compares against bigger vectors than retrieval needs.

After. Each chunk gets a content hash over its normalised text, checked against the cache:

for chunk in corpus:
    key = sha256(normalise(chunk.text))
    vector = cache.get(key)          # hit: unchanged or a known duplicate
    if vector is None:
        pending.append((key, chunk))  # miss: new or changed text

for batch in chunks_of(pending, BATCH_SIZE):
    vectors = embed([c.text for _, c in batch], dimension=512)
    for (key, _), v in zip(batch, vectors):
        cache.put(key, v)
        index.upsert(chunk_id, v)

index.delete(ids=missing_since_last_run)

The 399,960 unchanged chunks are cache hits and never reach the model. The footer hashes to one key, so the first copy embeds and the other 299 are hits, exact de-duplication for free. Only the 40 genuinely-new chunks land in pending, and they go to the model in a handful of batched requests instead of 40 separate calls. The vectors are written at a 512-dimension output chosen against a retrieval check rather than the maximum, so the index is smaller and every later query is cheaper. The embedding bill for the night tracks 40 chunks, not 400,000; the run finishes in minutes; and the morning’s edit is searchable the same day. Nothing here is a better model or a bigger machine, only the discipline of not paying twice for a vector you already have.

What’s worth remembering

1. Embedding is a per-token cost paid on every chunk you send, so re-embedding unchanged text is money spent to recompute a vector you already hold.
2. On a calm, large corpus the biggest win is incremental processing: hash each chunk’s text and embed only the new and changed hashes, so cost scales with change, not corpus size.
3. Hash exactly the normalised text the model sees, not rendered markup or timestamped wrappers, or cosmetic edits re-embed everything while real edits slip through.
4. Bedrock Knowledge Bases give managed incremental sync: after the first ingestion, later syncs re-embed only changed documents, trading chunking control for owning no bookkeeping.
5. A cache keyed on the content hash is one mechanism doing two jobs at once: cross-run incrementality and exact de-duplication, so identical text is embedded once and reused.
6. Exact de-duplication is close to free and close to safe; near-duplicate dedup saves more but can merge chunks that should stay distinct, so tune its threshold against real queries.
7. Batching many inputs per request cuts per-request overhead and wall-clock time, not the per-token charge; where the model takes one input per call, use bounded concurrency instead.
8. A smaller embedding dimension barely touches the embedding bill but shrinks the index and speeds every query for the life of the store, so it is a downstream saving worth settling early.
9. The embedding dimension is fixed at ingestion, so changing it later means a full re-embed; choose it up front against a retrieval-quality check rather than defaulting to the largest size.
10. Treat ingestion as a cache-and-diff problem: detect what changed, skip what did not, collapse what repeats, pack what remains, and store it no larger than retrieval needs.