Surviving a Model Deprecation on Bedrock

The situation

A subscription team runs three LLM features on Amazon Bedrock: a ticket classifier, a reply drafter, and a data-extraction job that turns free-text emails into records. All three call a single foundation model, referenced by an explicit version id baked into a config file about eighteen months ago. It has been quietly reliable ever since, which is exactly why nobody has touched it.

Then an AWS Health Dashboard notice lands: the model version they depend on is moving to legacy status, with an end-of-life date roughly six months out. After that date, calls to that version id will fail. There is a newer version of the same model family available now, and a couple of newer families besides, but none of them is a drop-in guarantee. The extraction job in particular is sensitive to output format, and one of the three features runs on a fine-tuned model with a Provisioned ThroughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. commitment attached.

The team has two bad instincts to resist. One is to do nothing until the deadline forces a panicked swap. The other is to flip everything to the newest model this afternoon and hope the quality holds. Neither is a plan. The real question is how to move off a retiring model version without breaking production, and how to build the app so the next deprecation is routine rather than a fire drill.

What actually matters

The first thing worth naming is that model versions are a stability contract, and deprecation is that contract expiring. Pinning an explicit version id is the right default, because it means the model behind your feature does not change under you between deployments. The price of that stability is that the version will eventually be retired, and the migration becomes your problem on AWS’s timeline rather than yours. The opposite posture, always pointing at the newest thing, spares you the scheduled migration but exposes you to silent behaviour change: a prompt that worked yesterday quietly starts formatting its answer differently, and nothing errors, so you find out from a downstream parser or a customer.

The second concern is how tightly the application is wired to one model’s request shape. If every call is hand-built against a specific model’s native payload, then switching model ids means rewriting request construction, and that friction is what turns a migration into a project. The Converse API is the lever here: it presents one consistent request and response shape across Bedrock text models, including system prompts and tool use, so changing which model answers is mostly a matter of changing the model id you pass. It does not erase every difference, since models still vary in supported inference parameters and in how they respond to a given prompt, but it removes the mechanical rewrite from the equation.

The third is whether you can prove the successor behaves before you trust it. A model swap is a behaviour change even within the same family, and the only honest way to catch a regression is to run the candidate against a saved evaluation set and compare. That eval set, a fixed collection of representative inputs with known-good expectations, is the asset that makes migration a measured decision instead of a leap. Bedrock’s model evaluation jobs can score a candidate against your own dataset, including an LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scoring path, but a home-grown harness that replays your golden inputs works too. What matters is that the comparison exists and runs before cutover, not the tooling brand.

The fourth is the cutover mechanism itself. Even a candidate that passes the eval set can surprise you on live traffic, so the change wants to be reversible in seconds, not redeployed over minutes. A feature flag or config value that selects the model id, ideally rolled out to a slice of traffic first, means a bad successor is a flag flip back rather than an incident. The rollback only exists while the old version is still invokable, which is another reason to start well before the end-of-life date rather than on it.

The fifth cuts across the others: prompts and few-shot examples are tuned to a specific model, not universal. A new model may need the instruction reworded, the examples trimmed or swapped, or the output contract restated, because it interprets the same prompt slightly differently. Budget retuning as part of the migration rather than assuming the prompt library travels unchanged.

And the expensive corner: a custom model, whether fine-tuned on Bedrock or brought in through Custom Model Import, carries migration cost the base models do not. A fine-tuned model is trained against a specific base version, so when that base is deprecated you may have to re-tune against the successor base, not just repoint an id. It also runs on Provisioned Throughput, a capacity commitment bought in model unitsModel unitThe billing block Provisioned Throughput is sold in – one unit delivers a fixed tokens-per-minute rate for a specific model. with an optional one- or six-month term for a lower rate, so a migration has to account for standing up new throughput for the successor and retiring the old commitment without paying for both any longer than necessary.

What we’ll filter on

1. Behaviour stability, does the feature need identical output over time, or can it absorb drift?
2. Migration lead time, how much notice before end-of-life, and how much work to actually move?
3. Request-shape coupling, how tightly is the app wired to one model’s native payload?
4. Regression detection, can we prove the successor behaves before we trust it with traffic?
5. Cutover and rollback safety, can we switch, canary, and revert without a redeploy?
6. Custom-model cost, is a fine-tuned or imported model plus a Provisioned Throughput commitment in the path?

The lifecycle landscape

Explicit version pinning. Bedrock model ids carry a version, and pinning one means your feature keeps calling exactly that model until you decide otherwise. This is the stable default and the reason production behaviour holds steady between deploys. Its cost is the scheduled migration: the version you pinned will eventually be marked legacy and then retired, and you own moving off it.

Floating to the newest version. The opposite posture, always reaching for the latest model in a family, spares you the forced migration but hands the model provider a lever over your behaviour. Quality often improves, but format, tone, and edge-case handling can shift with no error raised, so you inherit a silent-drift risk that a pinned version does not have. Suitable for tolerant, human-read features; dangerous for anything a machine parses.

Model lifecycle status. Bedrock foundation models move through statuses: active while fully supported, legacy once a successor is preferred and an end-of-life date is set, and end-of-life after which the version can no longer be invoked. AWS surfaces these transitions through the AWS Health Dashboard and account notifications, so the deprecation notice is the starting gun for a migration, not a surprise on the day. Watching for these notices is the difference between a six-month runway and a scramble.

Inference profiles. Many calls go through a cross-region inference profile id (for example a profile that routes a model across a set of regions) rather than a bare model id. Profiles still reference a specific model version, so they are subject to the same lifecycle; the profile is a routing and capacity convenience, not an exemption from deprecation.

The Converse API as an abstraction. Converse gives one request and response shape across text models, so the app talks to a stable interface and the model id becomes a swappable parameter. It absorbs the mechanical cost of switching models but not the behavioural cost, which is why an eval set still matters. The alternative, per-model native Invoke payloads, ties each feature to one model’s quirks and makes every migration a rewrite.

Model evaluation. A regression check against a saved dataset is what turns a swap into a decision. Bedrock model evaluation jobs can score a candidate against your own inputs with automatic metrics or an LLM-as-a-judge, and a custom replay harness does the same job for teams that already have golden data. Either way the eval set is a first-class asset that outlives any single model.

Custom and fine-tuned models. A fine-tuned model is bound to the base version it was trained on and runs on Provisioned Throughput. When its base is deprecated you may need to re-tune against the successor base and stand up fresh throughput, so its migration is heavier and its end-of-life planning is its own line item, not a footnote to the base-model swap.

Side by side

Option	Behaviour stability	Silent-drift risk	Migration effort	Cutover and rollback	Cost shape
Pin explicit version	✓ Stable	✗ None	Scheduled, on you	Flag-controlled	On-demand
Float to newest	✗ Drifts	✓ High	None until it breaks	Hard to reason about	On-demand
Converse abstraction	Neutral	✗ Removes mechanical cost	Low id swap	Clean	On-demand
Eval set before cutover	Proves it	✗ Catches drift	Upfront to build	Gates the switch	Job cost only
Flag cutover with rollback	Neutral	✗ Contains blast radius	Low	✓ Instant revert	Negligible
Fine-tuned + Provisioned Throughput	✓ Stable	✗ None	✓ Heavy, re-tune	Dual-run needed	Committed capacity

Reading the table against the three features: the classifier and drafter want a pinned version reached through Converse, an eval set, and a flag-controlled cutover; the fine-tuned extraction path wants all of that plus a re-tune against the successor base and a planned overlap of old and new Provisioned Throughput. Floating to the newest model is off the table for the extraction job the moment a parser depends on its output shape.

The picks in depth

The migration playbook is five steps, and the order matters. Pin an explicit version in config, never inline, so the model id is a value you can change in one place and roll out through the same mechanism as any other setting. Monitor deprecation notices through the AWS Health Dashboard and account notifications so a legacy status and its end-of-life date reach you as soon as they are set, giving you the full runway rather than the last week of it. Test the successor against your saved eval set and compare it to the incumbent on the same inputs, treating a format change or a quality dip as a blocker to investigate, not a rounding error to wave through. Cut over behind a flag, ideally to a canary slice of traffic first, watching your quality and error signals before widening the rollout. Keep a rollback available by leaving the old version invokable until the new one has proven itself on real traffic, which only works if you started before the end-of-life date closed that door.

Two details ride alongside the five steps. Prompts and few-shot examples are tuned to the incumbent, so expect to retune them for the successor; the eval set is what tells you whether the old prompt still holds or needs reworking, and it is far cheaper to discover that in a scored comparison than in production. And the Converse API is what keeps the id swap from becoming a rewrite: if the app already speaks Converse, changing the model behind a feature is a config change plus a validation pass, not a code change to request construction.

The fine-tuned model is the expensive pick and deserves its own timeline. Because a fine-tuned model is trained against a specific base version, a deprecation of that base can mean re-tuning against the successor base, which is a training job to schedule, a new model artefact to evaluate, and new Provisioned Throughput to provision. Plan for a window where the old commitment and the new one both exist so you can eval and canary the re-tuned model before cutover, then retire the old throughput promptly so you are not paying for idle committed capacity. Starting this the day the notice lands, not the month before end-of-life, is what keeps the dual-running cost small.

A worked example: the pinned classifier gets a deprecation notice

The classifier calls a model through a version id in a config value, over the Converse API, and has run untouched for eighteen months. A Health Dashboard notice marks that version legacy with an end-of-life date about six months out and names a successor in the same family.

Day one, not month five. The team opens the runway immediately. They already have a saved eval set for the classifier: a few hundred tickets with known-correct labels, collected from real traffic and frozen. They run the successor against it through Converse, changing only the model id, and compare label-for-label with the incumbent. The successor agrees on 97 per cent of the set but flips a cluster of billing-versus-account edge cases, because it reads an ambiguous phrase differently. That is a regression to fix, not to accept: they adjust the label definitions in the prompt and add two varied examples covering the edge cases, re-run the eval, and the disagreement clears.

With the candidate passing, cutover is a flag. They point five per cent of traffic at the new model id, watch the classifier’s confidence and the downstream correction rate for a few days, then widen to twenty-five, then to everything. The old version stays pinned and invokable throughout, so at any point a bad signal is a flag flip back to a known-good model, not an incident. Once the successor has held on full traffic, they retire the old version from config, well ahead of the end-of-life date. Total code change: a config value and two prompt examples. The Converse abstraction meant the id was swappable, the eval set meant the swap was a measured decision, and the flag meant the risk was contained. The next deprecation notice will run the same five steps.

What’s worth remembering

1. Bedrock models are versioned and pinning an explicit version is the stable default; the trade is that the version will be marked legacy and eventually retired, and the migration is yours to run.
2. Floating to the newest version avoids the scheduled migration but invites silent behaviour drift, which is fine for human-read output and dangerous for anything a machine parses.
3. Deprecation is not a surprise: watch the AWS Health Dashboard and account notifications for the legacy status and end-of-life date, and treat that notice as the start of a six-month runway, not a last-week scramble.
4. The Converse API gives one request and response shape across text models, so switching model ids is a config change rather than a rewrite of request construction.
5. Keep a saved evaluation set of representative inputs with known-good expectations; it is the asset that turns a model swap into a measured decision and outlives any single model.
6. Run the successor against that eval set and compare before cutover, treating a format or quality regression as a blocker to investigate, not a rounding error.
7. Expect to retune prompts and few-shot examples for the new model, since it interprets the same instruction slightly differently; the eval set tells you what needs reworking.
8. Cut over behind a flag to a canary slice first, and keep the old version invokable so a bad successor is an instant revert rather than a redeploy.
9. A fine-tuned or imported model is bound to its base version and runs on Provisioned Throughput, so its migration can mean re-tuning against the successor base and standing up new committed capacity, with a planned overlap and prompt retirement of the old commitment.
10. Build the app so migration is routine: pinned id in config, Converse for the request shape, an eval set for the decision, and a flag for the cutover, so the next deprecation runs the same five steps.