Tuning How a Model Samples: Temperature, Top-P, and Top-K

The situation

A team is running four LLM features on Amazon Bedrock behind one shared client wrapper: a ticket classifier that returns one of six labels, an extraction job that turns emails into structured records, a marketing-copy generator that writes three subject-line options, and a code assistant that drafts small functions. When the wrapper was first written, someone set a single default for the whole fleet, temperature 0.7, and every feature inherited it.

The results are exactly what you would predict once you know what the number does. The classifier disagrees with itself: the same ticket lands on “billing” one call and “account” the next, and the flakiness shows up in the evaluation harness as noise nobody can chase down. The extraction job occasionally invents a field value that was never in the email. The copy generator, on the other hand, is fine, and the code assistant is merely inconsistent. One default is right for one of the four features by accident.

Nobody wants to guess new numbers by superstition. The question underneath all four is the same: what does each sampling parameter actually change, and which setting does this task want?

What actually matters

A model does not emit an answer; at each step it produces a probability distribution over the whole vocabulary and then samples one token from it. Every sampling parameter is a way of reshaping or truncating that distribution before the draw. Understanding the parameters means understanding where in that distribution each one reaches.

The first thing worth naming is that determinism and creativity are the two ends of one axis, and most task pain comes from sitting at the wrong end. A classifier, an extractor, a factual lookup, a tool-calling agent choosing which function to invoke: these want the most probable token nearly every time, because there is a right answer and drifting off it is error, not variety. Brainstorming, subject lines, story openings, alternative phrasings: these want the model to reach past the single most likely continuation, because the whole value is in the options it would otherwise never offer. The mistake in the situation above is running a distribution-widening setting on tasks that wanted the opposite.

The second thing is that the parameters are not interchangeable levers pointed at the same place. Temperature rescales the entire distribution, making it flatter (every token more equally likely) or sharper (probability mass piling onto the front-runners). Top-p and top-k do something different: they truncate the distribution to a candidate set before sampling, cutting off the long tail of unlikely tokens entirely so they can never be drawn. Rescaling and truncating compose in ways that are hard to reason about together, which is why the standard advice is to move one of temperature or top-p and leave the other at its default rather than fighting both at once.

The third is that lower is not automatically safer. Pulling temperature to zero makes the model greedily take its top token every step, which is what you want for a label but tends toward flat, repetitive, sometimes degenerate text on anything generative, and it does not actually promise identical output across calls. Batching, floating-point non-associativity across hardware, and model-side details mean that even at temperature 0 you can see runs diverge. Treat it as strongly deterministic, not as a reproducibility guarantee; if you need bit-for-bit repeatability, that comes from caching or fixing a seed where the model exposes one, not from temperature alone.

The fourth is the cap that is not about randomness at all. Max tokens bounds how long the response can be, and it is a hard cut, not a hint; the model does not tidily wrap up when it approaches the limit, it stops mid-sentence when it hits it. Set it too low and structured output gets truncated into unparseable garbage; leave it unbounded on a chatty model and a runaway generation costs you tokens and latency. It belongs in the same conversation because it is set alongside the sampling parameters and it is the one people forget until a JSON payload arrives cut in half.

And the operational point that ties them together: these are inputs to the call, so they belong with the call. On Bedrock the Converse API carries the common ones in a single inferenceConfig block, and the model-family-specific ones ride alongside in a pass-through field. Setting them per feature rather than once for the whole fleet is the actual fix here.

What we’ll filter on

1. Determinism need, does the task have one right answer that must not drift, or does it want variety?
2. Distribution reach, should sampling stay on the front-runner tokens, or deliberately reach into the tail?
3. Interaction safety, does the setting change one dial cleanly, or fight two truncation-and-scaling knobs at once?
4. Portability, does the parameter exist on every model family on Bedrock, or only some?
5. Output-length control, is the response bounded so structured output cannot get truncated?
6. Reproducibility expectation, is “usually the same” enough, or is exact repeatability being assumed where it does not hold?

The sampling landscape

Temperature. A single scalar, typically in a range like 0 to 1 (some families allow up to 2), that scales the sharpness of the whole distribution before sampling. At low temperature the probability mass concentrates on the highest-scoring tokens, so the model almost always picks its top candidate and behaviour is focused and near-deterministic; this is what you want for classification, extraction, and factual answers. At high temperature the distribution flattens, the gap between likely and unlikely tokens narrows, and the model reaches for less obvious continuations, which is what powers brainstorming and creative copy. The failure modes sit at both ends: too low and generative text turns flat and repetitive, too high and it drifts into incoherence and invented detail. Temperature is the one parameter every model family on Bedrock exposes.

Top-p (nucleus sampling). Instead of touching the shape, top-p truncates. Sort tokens by probability, walk down the list accumulating their probabilities, and keep the smallest set whose cumulative probability first exceeds p; sample only from that nucleus and discard everything below it. At p = 0.9 the model samples from however many tokens it takes to cover 90% of the mass, which might be three tokens when the model is confident and forty when it is unsure, so the candidate set adapts to how peaked the distribution is at that step. Lower p tightens the pool toward the front-runners; p = 1 disables the cut. It is the usual companion to temperature and, like temperature, it is broadly supported.

Top-k. The blunter truncation: keep the k most likely tokens, full stop, and sample from those regardless of how much or how little probability they cover. k = 1 is greedy decoding, always the single top token; k = 40 samples from the top forty. The difference from top-p is that k is a fixed count rather than an adaptive mass, so it does not widen when the model is uncertain or narrow when it is confident. Top-k is not exposed by every model family on Bedrock; where it exists it is often a pass-through parameter rather than one of the common fields, so a prompt that depends on it is less portable across models.

Max tokens. Not a sampling parameter at all, but set in the same place and easy to get wrong. It caps the number of tokens the model may generate in the response, as a hard stop. It protects you from runaway cost and latency, and it is the thing to check first when a structured response comes back truncated: the shape was fine, the ceiling was too low. Size it to the longest legitimate output the feature produces, with headroom, rather than to the typical one.

Side by side

The rule the table encodes: temperature and top-p both aim at the same goal (how far past the front-runner the model may wander), which is exactly why you tune one and leave the other alone. Reading it against the four features, the classifier and extractor want low temperature and default top-p; the copy generator wants higher temperature; the code assistant wants something low but not zero; and every one of them wants a sensible max-tokens ceiling.

The picks in depth

The classifier and the extraction job are the determinism cases, and both were mis-set by the shared 0.7 default. Drop temperature to 0 (or very near it) and leave top-p at its default. At that setting the model takes its highest-probability token nearly every step, so the same ticket lands on the same label call after call and the evaluation noise disappears; for extraction, the model stops reaching into the tail for a plausible-sounding field value that was never in the source text, which is where the invented fields were coming from. One dial, moved on the two features that needed it. Do not also crank top-p or top-k down to “help”, because stacking three truncation-and-scaling changes makes the behaviour hard to reason about and buys nothing once temperature is already low.

The copy generator is the case where the default was accidentally right, and it is worth understanding why so nobody “fixes” it. Three distinct subject lines require the model to reach past its single most likely continuation, which is precisely what a temperature around 0.7 to 1.0 buys. If the options come back samey, raising temperature (or loosening top-p toward 1) widens the pool it samples from; if they drift into nonsense, pull back. Tune from the one that is already moving, temperature, and let top-p sit at its default rather than turning both.

The code assistant wants the middle, and it is the clearest illustration that “lower is safer” is not a rule. Code needs to be mostly deterministic, because there is usually a correct structure, but temperature 0 tends to make models loop or produce oddly rigid output, so a low-but-nonzero setting (in the rough vicinity of 0.2) gives stable drafts without the degeneracy. Whatever the feature, give it a max-tokens ceiling sized to its real output; the extraction job in particular must not have its JSON truncated by a ceiling set for one-line labels, because a half-emitted object fails the downstream parser exactly as a malformed one would.

On Bedrock, the mechanics are the same across all four: the Converse API takes temperature, topP, and maxTokens in its inferenceConfig, and anything family-specific such as top-k goes through additionalModelRequestFields, which passes model-native parameters the common config does not cover. Setting these per feature in the client wrapper, rather than inheriting one fleet-wide default, is the whole fix.

A worked example: the classifier, before and after

Before, every feature inherits the fleet default. The classifier call looks like this:

response = client.converse(
    modelId=MODEL_ID,
    messages=messages,
    inferenceConfig={"temperature": 0.7, "maxTokens": 1024},
)

At temperature 0.7 the distribution stays flat enough that the second- and third-choice labels retain real probability, so an ambiguous ticket that scores billing 0.55 and account 0.40 gets sampled as account a meaningful fraction of the time. Run the same ticket ten times and you get a spread, which is why the evaluation harness sees noise and the label boundaries look fuzzier than they are.

After, the sampling is set to the task. Classification wants the top token, and a maxTokens sized to a short label rather than a paragraph:

response = client.converse(
    modelId=MODEL_ID,
    messages=messages,
    inferenceConfig={"temperature": 0.0, "topP": 1.0, "maxTokens": 16},
)

Now the model takes its most probable label almost every call, the same ticket returns the same answer, and the harness measures the classifier instead of measuring sampling jitter. Top-p is left at 1.0 rather than also being tightened, because temperature 0 has already collapsed the choice to the front-runner and a second truncation dial would only muddy the reasoning. The one caveat to keep honest: this is strongly deterministic, not a contractual guarantee of identical bytes across calls, since hardware and batching effects can still nudge a borderline token; if the feature genuinely needs reproducible outputs for audit, that comes from caching the result, not from the temperature setting.

What’s worth remembering

1. Every sampling parameter reshapes or truncates the model’s next-token probability distribution before the draw; knowing where each one reaches is the whole game.
2. Temperature scales the whole distribution: low sharpens it toward the top tokens for focused, near-deterministic output, high flattens it for diversity.
3. Top-p keeps the smallest set of tokens whose cumulative probability exceeds p, so the candidate pool adapts, staying small when the model is confident and widening when it is unsure.
4. Top-k keeps a fixed count of top tokens regardless of their probability mass, and it is not exposed by every model family, so depending on it costs portability.
5. Tune temperature or top-p, not both hard at once; they aim at the same behaviour and stacking them makes the result hard to reason about.
6. Use low temperature for classification, extraction, factual answers, and tool or agent use; use higher temperature for brainstorming and creative copy.
7. Lower is not automatically safer: temperature 0 can make generative and code output flat, repetitive, or looping, so a low-but-nonzero value is often the better default there.
8. Temperature 0 is strongly deterministic but not a reproducibility guarantee; exact repeatability comes from caching or a fixed seed where available, not from the temperature dial.
9. Max tokens is a hard cap that cuts the response off mid-output, so size it to the longest legitimate response or watch structured payloads get truncated into garbage.
10. On Bedrock, set temperature, topP, and maxTokens in the Converse API inferenceConfig, and pass family-specific fields like top-k through additionalModelRequestFields, per feature rather than once for the whole fleet.