The situation
A lending team runs an LLM feature on Amazon Bedrock that summarises an applicant’s supporting documents and flags anything that needs a human to look closer. The summary and the flags feed a decision that a regulator can later ask about, so when an auditor picks a past application and asks the team to reproduce exactly what the model saw and said, “we can’t, it comes out slightly different each time” is not an acceptable answer.
The team has already turned temperature down to zero, and most of the time the output is stable. But not always. Two runs of the same document occasionally differ by a word or a reordered clause, and once, after a quiet Tuesday, every summary started coming out in a subtly different style with no code change on their side. Digging in, they found the model alias they call had rolled to a newer version underneath them. Nobody deployed anything; the behaviour just moved.
Alongside the audited lending feature, the same team runs a marketing-copy generator that writes three cheerful variations of a product blurb. That one is supposed to vary. Forcing it to be deterministic would defeat the point. So the question is not “how do we make every LLM call reproducible”, it is “which calls need it, how identical does identical have to be, and what does each level of guarantee cost”.
What actually matters
The first thing worth naming is that “reproducible” is not one property. There is a spectrum between output that is roughly stable and output that is byte-for-byte identical on every replay, and the further along that spectrum you go, the more machinery you take on. Deciding where a given feature needs to sit is the whole job; over-engineering determinism onto a creative feature is as much a mistake as under-engineering it onto an audited one.
Temperature is the lever most people reach for first, and it is worth understanding exactly what it does and does not buy. At temperature zero the model does greedy decoding: at each step it takes the single highest-probability token rather than sampling from the distribution. That removes the deliberate randomness, which is why output becomes far more stable. What it does not do is make the result mathematically guaranteed to be identical. Two subtle sources of drift remain. Floating-point arithmetic is not associative, so the same logits computed on different GPU hardware, or with a different batch size, or with a different kernel, can round differently and occasionally tip which token wins at a near-tie. And the provider can change the model or the serving stack underneath you. Greedy decoding narrows the variation dramatically; it does not close it to zero.
That second source, the model moving underneath you, is why version pinning matters as much as any sampling parameter. A model reference that points at a floating alias is a reference that can change behaviour without you touching a line of code, exactly the quiet-Tuesday failure. Pinning the specific, versioned model identifier means the weights answering your call today are the weights answering it next quarter, until you deliberately choose to move. This is the single highest-leverage step for audit and regression stability, and it costs nothing but the discipline of naming the version explicitly.
The only way to guarantee that a replay returns exactly what happened the first time is to not re-run the model at all. If you store the exact request and the exact response it produced, keyed on a hash of that request, then a later replay is a lookup, not a generation. A cache in front of the model gives you byte-identical repeats because you are handing back the stored bytes. This is also the only mechanism that survives the provider retiring or changing the model version entirely, which matters when the retention window for an audited decision is measured in years and the model version is not guaranteed to still be servable that far out.
The last thing to hold onto is that reproducibility is a means, not a virtue in itself. You engineer for it where a difference in output changes an outcome that someone can be held to: evaluation and regression testing, where you need to know a change came from your prompt and not from noise; audit and compliance, where you must reproduce what the system did; and any regulated or high-stakes decision. For open-ended creative generation, variation is the feature, and every mechanism above is cost with no benefit.
What we’ll filter on
- Required determinism level, roughly stable, run-to-run consistent, or byte-identical on replay?
- Provider and hardware drift, does the mechanism survive floating-point non-associativity and serving changes?
- Version stability, does it protect against the model silently changing underneath the call?
- Replay guarantee, can you return exactly what a past call produced, even years later?
- Cost and latency, what does the guarantee add per call and in storage?
- Fit to the use case, does the feature actually benefit from determinism, or does it need variety?
The reproducibility landscape
Temperature zero (greedy decoding). Set temperature to zero so the model always takes the top token instead of sampling. The biggest single reduction in variability for the least effort, and the right default for any feature that wants a stable answer. It leaves the sampling randomness gone but not the hardware-level and provider-level drift, so treat it as “much more stable”, never as “guaranteed identical”. Turning off the other sampling knobs (leaving top-p and top-k out of the picture) removes further sources of run-to-run wobble.
Fixed seed, where the model supports it. Some models expose a seed parameter so that sampling, when you do sample, follows a repeatable pseudo-random sequence. Seed is not part of the Converse API’s own inferenceConfig, which carries only maxTokens, stopSequences, temperature, and topP, so where a model does take one you pass it through additionalModelRequestFields and Converse hands it to the model untouched. Support is patchy enough to check before designing around it. Cohere’s Command R and Command R+ take a seed and the documentation is candid about what it buys, a best effort at deterministic sampling with determinism not totally guaranteed; the Anthropic Claude models publish no seed parameter at all. Even where it exists the floating-point and serving caveats still apply across different hardware. Useful when you want repeatable variety rather than greedy output; not a universal lever and not a hard guarantee on its own.
Pinned model version. Call the specific versioned model identifier rather than a floating alias or a “latest” pointer, so the weights behind your call do not move without a deliberate change on your side. This is what stops the silent-rollout failure and what makes a regression test meaningful, because a behaviour change now has to come from something you did. It says nothing about run-to-run drift on identical inputs; it fixes which model, not how deterministic that model is.
Prompt and parameter versioning. Keep the prompt template, the sampling parameters, and the model version together as one versioned, stored configuration rather than scattered across code. Reproducing a past decision means reproducing the whole request, and the prompt wording and parameters are as much a part of that as the model. This is the operational glue that makes the other levers auditable; on its own it changes nothing about the model’s behaviour.
Response cache (exact request-to-response store). Hash the exact request (prompt, parameters, model version, inputs) and store the response against it; on a repeat, return the stored response instead of calling the model. The only mechanism that gives a genuine byte-identical guarantee, and the only one that survives the model version being changed or retired later. The costs are storage, a cache-key discipline strict enough that “the same request” really means the same bytes, and the fact that it only helps on genuine repeats of an identical request, not on novel inputs.
Full request/response logging. Persist every request and its response as an immutable record. This does not make future calls reproducible, but it satisfies the audit question directly: you can show exactly what the model was asked and exactly what it answered on the day. Bedrock has this built in as model invocation logging, switched on per Region with PutModelInvocationLoggingConfiguration and off until you do, delivering the request body, the response body, the model ID, the request ID, the calling principal, and the token counts to an S3 bucket, a CloudWatch log group, or both. Bodies over 100 KB go to S3 as separate objects with the log entry pointing at them, which is the normal case for a long-document prompt. For many regulated use cases this recorded-evidence approach is what the auditor actually wants, and it pairs naturally with a cache that is keyed on the same request hash.
Side by side
| Mechanism | Determinism level | Survives HW/FP drift | Survives version change | Byte-identical replay | Added cost |
|---|---|---|---|---|---|
| Temperature zero | Much more stable | ✗ | ✗ | ✗ | None |
| Fixed seed (if supported) | Repeatable sampling | ✗ | ✗ | ✗ | None |
| Pinned model version | Stable across time | ✗ | ✓ (you choose when) | ✗ | None |
| Prompt/param versioning | Reproducible request | ✗ | ✗ | ✗ | Low |
| Response cache | Exact repeat | ✓ | ✓ | ✓ | Storage, key discipline |
| Request/response logging | Evidence, not replay | ✓ (as record) | ✓ (as record) | ✓ (as record) | Storage |
Reading the table against the two features: the marketing generator wants none of this and should keep sampling on. The lending summariser wants temperature zero and a pinned version as the baseline, prompt and parameter versioning so the whole request is reproducible, and a response cache plus immutable logging so a past decision can be replayed and evidenced exactly, whatever happens to the model version later.
The picks in depth
Start every stability-sensitive feature at temperature zero and a pinned model version, because together they cost nothing and remove the two loudest sources of surprise: sampling randomness and the model moving underneath you. On Bedrock this means calling the specific versioned model identifier rather than a floating alias, and being deliberate about Inference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. or aliases that resolve to “current”. The trap to avoid is assuming this pair gives you a guarantee. It gives you stability, and for a great many features stability is genuinely enough. But if an auditor needs bit-exact reproduction, temperature zero will let you down on the day two runs round differently at a near-tie, and you will not be able to explain the difference.
For the audit guarantee, the mechanism is a cache, not a parameter. Hash the full request (the pinned model version, the exact prompt text, every sampling parameter, and the input documents) into a key, and store the response bytes against it. A replay is then a lookup that returns the stored bytes, identical by construction, and it stays identical even after the model version you originally used has been retired and is no longer servable. On AWS this is ordinary infrastructure rather than anything model-specific: a durable store keyed on the request hash, sitting in front of the Bedrock call, with the request and response also written to immutable storage for the audit trail, which is the half you get by turning model invocation logging on rather than building it. Get the cache key wrong, though, and the guarantee evaporates: if the key omits the model version or normalises whitespace differently from the caller, you will either serve a stale response for a changed request or miss the cache for a request that was really the same. The key has to mean “the same bytes”, exactly.
The discipline that ties it together is versioning the whole request as one artefact. A reproducible decision is not just a reproducible model, it is a reproducible prompt, a reproducible set of parameters, and a reproducible model version, captured together. Storing those as a single versioned configuration is what lets you say, a year later, precisely what the system asked and answered, and it is what makes a regression test honest: when the output changes, you know the change came from a deliberate edit to that configuration and not from noise or a silent rollout. This is the same instinct as treating prompts as tested assets rather than inline strings, covered in the prompt-engineering rundown; reproducibility just raises the stakes on getting it stored and pinned.
And the mirror-image pick: do none of this to the marketing generator. It is meant to produce three different blurbs, so temperature stays up, no seed is pinned, no response is cached, and the only thing worth keeping is the log of what went out. Spending determinism engineering on a feature whose value is variety is the same category of error as leaving an audited feature on a floating alias. Match the guarantee to what the decision behind the output can be held to.
A worked example: the lending summariser
Trace one applicant through the audited path.
The request is assembled as a single versioned object: prompt template summariser@v7, parameters temperature 0, model the pinned versioned identifier, not the floating alias, and the applicant’s documents. Those bytes are hashed into a cache key.
On the first run, the key misses the cache, so the request goes to Bedrock. Temperature zero means greedy decoding, so the summary is as stable as the model can make it. The response comes back, and two things happen: it is written to the cache under the request hash, and the full request and response are written to immutable storage for the audit trail.
Six weeks later the model alias the rest of the business uses rolls to a newer version. The lending feature does not notice, because it never called the alias; it called the pinned version. Its output does not move.
A year after that, an auditor picks this application and asks what the system produced. The team replays the stored request. The cache key matches, so the stored response bytes come straight back, byte-for-byte identical, and the immutable log shows exactly what was asked and answered on the original day. It does not matter that the original model version has since been retired and can no longer be invoked, because nothing is re-generated; the guarantee lives in the stored bytes, not in the model. Temperature zero made the first run stable, the pinned version kept it from drifting, and the cache is what turned “stable” into “provably identical”.
What’s worth remembering
- “Reproducible” is a spectrum from roughly stable to byte-identical; decide how far along a feature needs to be before paying for it.
- Temperature zero switches the model to greedy decoding and removes sampling randomness, which makes output far more stable but is not a hard guarantee of identical results.
- Floating-point non-associativity across hardware, batch-size and kernel differences, and provider-side serving changes can still vary output even at temperature zero.
- A fixed seed, where the model supports it, gives repeatable sampling, but it is not universally available and carries the same hardware caveats.
- Pinning the exact model version rather than a floating alias stops the model changing behaviour underneath you with no deploy on your side; it is the highest-leverage step for audit stability and costs nothing.
- The only way to guarantee byte-identical output on replay is to cache the exact request-to-response mapping and return the stored bytes instead of re-running the model.
- A response cache also survives the model version being changed or retired, which matters when audit retention outlasts how long a version stays servable.
- Version the whole request together (prompt, parameters, and model version) so a past decision is fully reproducible and a regression change is traceable to a deliberate edit.
- Immutable request/response logging answers the audit question as recorded evidence even where you are not replaying the model.
- Engineer determinism for evaluation, testing, audits, and regulated decisions; leave creative generation free to vary, where reproducibility is cost with no benefit.