How to Match Inference Parameters to the Job at Hand

The situation

A product team at an insurance SaaS has three features running on the same foundation model through Amazon Bedrock, all using whatever inference parameters the console suggested on day one.

• An extraction pipeline: roughly 50,000 calls a day that turn inbound claims emails into a fixed JSON structure for the claims system. About one call in forty comes back as valid JSON wrapped in a paragraph of commentary (“Here is the extracted data you asked for…”), which breaks the parser. The bill for this feature is dominated by output tokens the parser throws away.
• A support-reply drafter: agents click a button, the model drafts a reply, the agent edits and sends. Agents complain that clicking “regenerate” produces a draft that is nearly word-for-word the previous one, so a bad first draft is a dead end.
• A tagline brainstormer for the marketing team: ask for ten campaign taglines, get ten taglines that are the same idea with the adjectives shuffled.

Three complaints, and none of them is about the model. The prompts are reasonable. What nobody has done is ask what each job needs the sampler to do.

What actually matters

A foundation model doesn’t emit text; it emits, one step at a time, a probability distribution over what the next token could be. Something then has to pick a token from that distribution, and the inference parameters are the rules for the picking. That’s the whole game: the model proposes, the sampler disposes. Two of the three complaints here are sampler complaints (not enough variety), and the third (prose around the JSON) is partly a sampler problem and partly a stopping problem.

The first property worth thinking about is how repeatable each job needs to be. The extractor wants the same email to produce the same JSON every time, because downstream systems and humans both need to trust it; surprise is a defect. The drafter wants the opposite on demand: the first draft should be sensible, but “regenerate” is a promise of a different draft, and a sampler tuned for repeatability breaks that promise. The brainstormer is further along still; variety is the entire product.

The second property is what stops generation, and when. Models stop when they decide they’re done or when they hit a ceiling, and both of those are configurable. For the extractor, an unbounded response is a cost leak and a latency problem: output tokens are the expensive ones on every pricing sheet, and response time grows with every token generated, so a model that adds a friendly paragraph after the JSON is charging us to slow us down. A hard length ceiling and an explicit stop string are circuit breakers, not niceties.

The third property is what these knobs can’t do. No temperature setting makes a model know things it doesn’t know; if the extractor mislabels a claim type, that’s a prompt or grounding problem, not a sampling problem. Parameters shape how the model says things: how adventurous the word choice is, how long it runs, where it stops. Reaching for temperature to fix a factual error is the most common way teams burn a week here.

What we’ll filter on

1. Repeatability: does the same input need to produce (as near as possible) the same output?
2. Diversity on demand: does asking again need to produce something genuinely different?
3. Length control: does the output need a hard ceiling, and is there a natural place to stop early?
4. Cost shape: output tokens cost more than input tokens, so anything that trims generated length trims the bill and the latency together.
5. Failure containment: what stops a runaway or malformed response before it reaches the parser, the agent, or the customer?

The parameter landscape

1. Temperature. Scales how sharply the model favours its top choices before a token is picked. Near zero, the most likely token wins almost every time and output becomes close to deterministic; raised towards and past one, lower-probability tokens get real chances and output becomes varied, sometimes inventively, sometimes incoherently. This is the primary repeatability-versus-diversity dial. One caveat worth knowing: temperature zero means “greedy and stable”, not “bit-identical forever”; providers generally don’t guarantee identical bytes across runs, so downstream validation still has to exist.

2. Top-p (nucleus sampling). Instead of reshaping the distribution, it truncates it: only the smallest set of tokens whose probabilities sum to p stay in the running. The clever part is that it’s adaptive. When the model is confident, the nucleus is one or two tokens; when it’s genuinely uncertain, the nucleus widens. Low top-p keeps output conservative even at moderate temperature; top-p near one lets the long tail of unusual tokens participate.

3. Top-k. The blunt cousin: keep only the k most likely tokens, whatever their probabilities. It’s a fixed-width cut where top-p is a confidence-aware one, useful mainly as a backstop that stops truly weird tokens sneaking in when temperature is high. Not every model on Bedrock exposes it; parameter menus vary by model provider, which is itself worth remembering.

4. Maximum tokens. A hard ceiling on response length. It doesn’t make the model concise; it makes the model stop, mid-sentence if necessary. Its real jobs are cost cap, latency cap, and runaway-generation circuit breaker. Set it just above the longest legitimate response, and a misbehaving call costs one response’s worth of tokens instead of a context window’s worth.

5. Stop sequences. Strings that end generation immediately when the model emits them. Where max tokens is a guillotine, a stop sequence is a full stop the model walks into naturally: end generation at a sentinel the prompt asked the model to emit after the JSON, at a delimiter between list items, at the start of a hallucinated next turn (“Human:”). The cheapest length control there is, because the model stops exactly where the job is done.

Temperature, top-p, and top-k overlap: all three shape which token gets picked. The standard advice is to lead with temperature, adjust top-p only when the long tail is the specific problem, and move one knob at a time so cause and effect stay visible.

Side by side

Parameter	Shapes which token is picked	Shapes when output ends	Repeatability lever	Cost & latency lever
Temperature	✓	✗	✓	✗
Top-p	✓	✗	✓	✗
Top-k	✓	✗	✓	✗
Maximum tokens	✗	✓	✗	✓
Stop sequences	✗	✓	✗	✓

The table splits cleanly in half: three knobs decide what gets said, two decide how long the saying goes on. Every job needs a position on both halves.

Matching jobs to settings

The picks in depth

The extractor gets pinned down. Temperature goes to zero (or as near as the model allows), so the highest-probability token wins at every step and the same email produces the same JSON run after run. Top-p drops low as a belt-and-braces backstop against tail tokens. Maximum tokens is set just above the longest legitimate JSON payload the schema permits, so a misfiring call is capped at one response’s cost instead of rambling to the context limit. The prose-wrapping problem gets a two-part fix: the prompt instructs the model to emit only JSON followed by a sentinel string, and that sentinel becomes a stop sequence, so generation ends the moment the payload does. The commentary paragraph never gets generated, which means it never gets billed and never reaches the parser. And because near-deterministic is not a contract, the pipeline keeps schema validation on every response; parameters reduce the failure rate, validation catches the remainder.

The drafter gets balance. Temperature around 0.7 keeps drafts coherent and on-register while making each generation a genuinely fresh sample, which is exactly what the regenerate button was promising all along. Top-p stays at the model default; the long tail isn’t the problem here, and moving two sampling knobs at once makes the next tuning session harder to reason about. Maximum tokens is sized to a complete reply with headroom, generous enough never to truncate mid-sentence, tight enough that a runaway draft can’t run far. The thing not to do is fix the drafter’s occasional stiffness with more temperature: register and tone are prompt problems, and pushing temperature towards one to fix tone buys incoherence without fixing the register.

The brainstormer gets opened up. Temperature in the 0.9 to 1.0 range and top-p near one let the unlikely-but-interesting tokens participate, which is where unexpected taglines live. The structural change matters as much as the knobs: ten suggestions from ten separate calls (or a few calls of two or three each) are far more diverse than one call asked for a numbered list of ten, because within a single response the model’s earlier lines anchor its later ones. Each call gets a small maximum-token ceiling, so the cost of the extra calls stays flat. The acceptance criterion changes too: at these settings some suggestions will be clunkers, and that’s the deal; the marketing team is the filter, and a 30% hit rate on genuinely varied ideas beats a 100% hit rate on the same idea ten times.

One cross-cutting habit ties the three together: change one parameter at a time and keep a small evaluation set of representative inputs for each job. “It feels better” doesn’t survive the next model upgrade; twenty saved emails, ten saved tickets, and a folder of rated taglines do.

What’s worth remembering

1. The model proposes a probability distribution; the parameters decide how a token gets picked from it, so most “the output feels wrong” complaints are sampling complaints, not model complaints.
2. Temperature is the primary repeatability-versus-diversity dial: near zero for machine-read output, moderate for human-edited drafts, high for deliberate variety.
3. Top-p truncates the candidate pool adaptively and top-k truncates it bluntly; tune temperature first and move one knob at a time.
4. Temperature zero means stable, not bit-identical; anything that parses model output still needs validation.
5. Maximum tokens is a circuit breaker for cost, latency, and runaway generation, because output tokens are the expensive ones and every generated token adds response time.
6. Stop sequences end generation exactly where the job is done, which makes them the cheapest length control available.
7. Diversity across several small calls beats diversity within one big call, because a response’s early lines anchor its later ones.
8. No parameter adds knowledge or fixes grounding; factual errors are prompt and retrieval problems, and reaching for temperature to fix them wastes the week.
9. Parameter menus differ by model provider, so a profile tuned for one model needs re-checking when the model changes.

One model, three jobs, three parameter profiles. The extractor wants the sampler silenced, the drafter wants it on a leash, and the brainstormer wants it off the leash entirely. None of that is a model decision; it’s five knobs, set per job, and an evaluation set to prove the settings are earning their keep.