Prompt Caching Versus Response Caching on Bedrock

The situation

A document-Q&A assistant runs on Bedrock. Every request carries a 1,900-token system block (persona, formatting rules, safety policy, a dozen few-shot examples) and then, for the current workload, a 4,000-token contract that the user is asking questions about. On top of that sits the user’s actual question, usually 20 to 60 tokens, and the running conversation. A single session might ask fifteen questions about the same contract before moving on.

Two cost patterns show up in the traffic. The first is that the 1,900-token system block and the 4,000-token contract are byte-for-byte identical across every turn of a session, and the system block is identical across every session in the product. The team is paying full input-token price to re-send and re-process the same prefix thousands of times an hour. The second is that across sessions, a good fraction of the questions are near-duplicates: “what is the notice period?” turns up in hundreds of different contract sessions, and half the time the answer is the same clause phrased the same way.

The bill is dominated by input tokens, not output. Someone has read that Bedrock supports prompt caching and someone else has read that you can cache responses, and the two ideas are being used interchangeably in the planning doc. They solve different problems and the design needs both named correctly before it can decide what to build.

What actually matters

The first thing to pin down is which repeated part of the request is costing money. Prompt caching and response caching attack different repetitions. Prompt caching targets a repeated input prefix: a long, stable run of tokens at the front of the prompt that many requests share. Response caching targets a repeated whole request: a question the system has effectively answered before, where the stored answer can go straight back without touching the model. One reuses the model’s processing of shared context; the other reuses a finished result.

The second is whether the model still runs. This is the sharpest line between them. Prompt caching always calls the model. It reads the cached prefix at a discounted rate, streams the first token sooner because the prefix is already processed, and then does real inference on the varying tail. Response caching, when it hits, does not call the model at all; it returns a stored answer in milliseconds. That difference sets both the ceiling on savings and the nature of the risk.

The third is the risk each one carries. Because prompt caching still runs inference on the actual question, a cache hit cannot produce a wrong answer; the worst case is a cache miss and full price. Response caching can serve a wrong answer, and that is its whole danger. An exact-match response cache is safe but rarely hits, since paraphrases and a different attached contract miss. A semantic response cache hits far more often and can false-hit: two questions whose embeddings sit within the similarity threshold but whose correct answers differ. The stale-answer and false-match failure modes, and how to defend against them, are worked through in the response-caching scenario; this post takes that as read and concentrates on how the two kinds of caching relate.

The fourth is staleness tolerance. A response cache holds an answer for as long as its TTL and invalidation rules allow, which could be hours, so it needs to know when the underlying knowledge changed. A Bedrock prompt cache is short-lived, on the order of a few minutes, and it caches input processing, not an answer, so it cannot go stale in the correctness sense at all. If the workload cannot tolerate any risk of an out-of-date answer, that pushes work toward prompt caching and toward tight invalidation on the response side.

The fifth is where the repeated content lives in the prompt. Prompt caching only helps when the shared content is a contiguous prefix, everything up to a cache checkpoint has to match exactly, so anything you want cached (system block, few-shot set, the shared document) belongs at the front, with the per-request variation (the user’s question, the turn history) after it. Put the varying part first and the cacheable prefix never repeats. That ordering constraint is a design decision, not a runtime flag.

What we’ll filter on

1. Repeated unit: is the identical part an input prefix, or the whole request-plus-answer?
2. Does the model still run: does a hit save a fraction of the call, or skip it entirely?
3. Wrong-answer risk: can a hit ever return an incorrect answer?
4. Staleness window: how long can a cached thing live, and does it need content-change invalidation?
5. Placement and identity: is the shared content a contiguous front-of-prompt prefix, or a whole request that recurs across users?

The caching landscape

1. Bedrock prompt caching (prefix reuse). Mark one or more cache checkpoints in the request; Bedrock caches the processed prefix up to each checkpoint server-side for a short window (minutes) and, on a subsequent request whose prefix matches exactly up to that checkpoint, charges cache-read tokens at a large discount to the normal input rate and skips re-processing them. The model still runs on the uncached tail. Best when many requests share a long identical prefix: multi-turn chat over the same context, a fixed instruction-plus-few-shot block, repeated Q&A against one large document. Savings are on input-token cost and time-to-first-token, never on output.

2. Exact-match response cache. Hash the fully-expanded request (normalised prompt plus any attached context) and map it to the stored answer, in DynamoDB or Redis, under a TTL. A hit returns the answer with no model call. Safe, since an exact match is exact, but the hit rate is low because paraphrases and a differing attached document miss.

3. Semantic response cache. Embed the query, find the nearest cached query by cosine similarityCosine similarityA measure of how closely two vectors point the same way, used as the default score for “how related is this text?”. , and if it clears a threshold return the stored answer without calling the model. Much higher hit rate; carries false-hit risk when near-neighbour questions have genuinely different answers, so the threshold needs tuning and the cache needs a cacheability gate so per-user questions never cross sessions.

4. Retrieval cache. Cache the retrieved chunksChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. for a canonical query rather than the answer. The model still generates. This trims the retrieval step, not the generation cost, and sits alongside either kind of caching above.

5. No caching (baseline). Pay full input and output price on every call. The honest option when prefixes are short and questions are genuinely unique, where neither lever has anything to bite on.

The first row and the middle rows are different categories of thing, which is the whole point. Prompt caching is a discount on the input side of a call that still happens. Response caching is the chance to not make the call. They are not alternatives to weigh against each other; they compose.

Side by side

Property	Bedrock prompt caching	Exact-match response cache	Semantic response cache
Repeated unit	Input prefix	Whole request	Query meaning
Model still runs	✓ (on the tail)	✗ (on a hit)	✗ (on a hit)
Saves output-token cost	✗	✓	✓
Can serve a wrong answer	✗	✗	✓ (false hit)
Hit rate	High for shared prefixes	Low	High
Staleness risk	None (input only)	TTL-bounded	TTL-bounded + false hits
Lifetime	Minutes (short)	Minutes to hours	Minutes to hours
Cross-user safety	Inherent (no answer stored)	Needs session-scoped keys	Needs a cacheability gate
Main lever	Input cost, time-to-first-token	Skip the whole call	Skip the whole call
Application effort	Low (mark checkpoints, order prefix)	Low	Moderate (embed, threshold, gate)

Read across the “model still runs” row and the two categories separate cleanly. Prompt caching keeps the call and makes its input cheaper; response caching tries to avoid the call and takes on answer-correctness risk to do it. That is why they stack rather than compete.

The two paths a request can take

The picks in depth

Bedrock prompt caching. The design work is placement and identity, not much else. Move everything stable to the front: the system block, the few-shot examples, and the shared document all belong ahead of the user’s turn, with a cache checkpoint marked after the last stable token. The prefix has to match exactly for a hit, so a single changed byte early in the prompt invalidates everything after it; that is why per-request content (the question, the growing conversation) goes last. Within a session asking fifteen questions about one contract, turns two through fifteen read the ~5,900-token prefix from cache at a fraction of the input rate and pay full price only on the short question and the accumulating history. Across sessions, the 1,900-token system block is a shared prefix for the entire product, so it stays warm as long as traffic keeps hitting it inside the cache window. The window is short, a few minutes, so prompt caching rewards bursty, clustered traffic and does nothing for a prefix seen once an hour. Support and minimum cacheable prefix length vary by model, so confirm both for the specific model in use rather than assuming. Because inference still runs on the real question, there is no correctness risk to manage: the only outcomes are a hit (cheaper, faster first token) or a miss (full input price), never a wrong answer.

Exact-match response cache. The safe skip. Hash the fully-expanded request, and note that “fully expanded” has to include the attached contract, because “what is the notice period?” against contract A and against contract B are different requests with different answers. Scope the key so that per-user or per-document context is part of the hash, and a hit is genuinely the same question in the same context. It will not hit often, because a different contract or a reworded question misses, but every hit it does score is correct by construction and skips both input and output cost.

Semantic response cache. The high-hit-rate skip, and the one that can be wrong. Embed the query, look up the nearest cached query, and return the stored answer above a cosine threshold. The false-hit trap here is specifically the attached-context problem: two contract questions can be near-identical in embedding space and have different correct answers because the underlying documents differ, so a naive semantic cache keyed on question text alone will confidently return contract A’s notice period for a contract B session. Gate cacheability (only cache context-free, cross-user-safe questions), fold the document identity into the key or the cacheability decision, and tune the threshold against evaluation data. The mechanics, the threshold tuning, the cacheability gate, and the invalidation strategy are the subject of the response-caching scenario; the point to carry here is that this is the only one of the three that trades correctness for hit rate.

How they stack. Put the response cache in front and the prompt cache underneath. A request first checks the response cache; on a hit it returns in milliseconds with no model call, saving input and output both. On a miss it falls through to Bedrock, where prompt caching discounts the shared prefix while the model runs on the tail, then the fresh answer is written back to the response cache for next time. The response cache decides whether to call the model; prompt caching makes the calls you do have to make cheaper. They are complementary because they cut different costs: the response cache removes whole calls, and the prompt cache shrinks the input bill on the calls that remain. Neither one makes the other redundant.

A worked example: one session, then the crowd

A session opens on a 4,000-token contract. The prompt is assembled prefix-first: 1,900-token system block, then the 4,000-token contract, a checkpoint, then the user’s question and the running history.

Turn one asks “what is the notice period?”. The response cache is checked first. If a cacheable, context-safe entry for this exact question against this exact contract exists, it returns in ~50 ms and the model is never called. Assume a miss. The request goes to Bedrock. This is the first time the ~5,900-token prefix has been seen in this cache window, so it is a prompt-cache write: full input price on the prefix, and the answer comes back in ~2 s. The answer is written back to the response cache.

Turns two through fifteen each ask a new question about the same contract. Each one re-checks the response cache first; the genuinely repeated ones (a user re-asking, or a stored context-safe answer) short-circuit. The rest fall through to Bedrock, where the ~5,900-token prefix is now warm: those turns read it at the discounted cache-read rate and pay full price only on the ~40-token question and the accumulating history. Fourteen turns reuse the prefix that was paid for once. Input cost for the session collapses toward the cost of the varying tails plus one full prefix, while output is billed in full every turn because prompt caching never touches output.

Now the crowd. Across the day, hundreds of sessions open different contracts but all carry the same 1,900-token system block. That block stays warm as a shared prefix whenever traffic keeps it inside the window, so most sessions read it at the discount even on their first turn. Independently, the recurring context-free questions (“how do I export my data?”, “what does the service tier include?”) accumulate in the response cache and start returning without any model call at all. The two effects are additive: the response cache thins out the number of Bedrock calls, and prompt caching makes the surviving calls cheaper on input. The bill falls on two axes at once, and because inference still runs on every question that reaches the model, the only correctness surface to watch is the semantic response cache’s false-hit rate, defended by the cacheability gate and the document-aware key.

What’s worth remembering

1. They are different levers. Prompt caching discounts a repeated input prefix on a call that still happens; response caching skips the call by returning a stored answer. Name them correctly before designing.
2. The dividing line is whether the model runs. Prompt caching always runs inference on the question; a response-cache hit runs none.
3. Only response caching can be wrong. A prompt-cache hit cannot produce a wrong answer because the real question is still processed; a semantic response cache can false-hit.
4. Prompt caching needs a contiguous front-of-prompt prefix. Put system block, few-shot, and shared document first, with per-request content after the checkpoint; one changed byte early invalidates the rest.
5. Prompt caching saves input and time-to-first-token only, never output. Output is billed in full on every call.
6. The prompt cache is short-lived. It rewards bursty, clustered traffic within a few-minute window; a prefix seen once an hour gains nothing.
7. Response caching saves input and output but needs freshness discipline. TTL plus content-change invalidation, and a cacheability gate so per-user context never crosses sessions.
8. For attached-document workloads, fold document identity into the response-cache key. The same question against a different contract is a different request; ignoring that is how semantic caches serve confident wrong answers.
9. They stack. Response cache in front to skip calls, prompt cache underneath to cheapen the misses; the two costs they cut are different, so neither makes the other redundant.
10. Confirm model-specific support and the minimum cacheable prefix length for the model in use before relying on prompt caching; both vary.