The situation
A product team runs a customer-facing assistant on Amazon Bedrock, calling one Claude model on-demand. It was comfortable in testing and through the first month of light traffic. Then a marketing push doubled sign-ups, an overnight batch job that summarises the day’s tickets started overlapping with daytime interactive load, and the logs filled with ThrottlingException. Some requests now fail outright; others succeed only after several seconds of silent retrying, and the p99 latency has crept from under two seconds to well over ten.
The team’s first instinct was a tight retry loop that hammers the model until it answers. That made the failures quieter but the latency worse, because every throttled request now spends its time re-queuing rather than erroring fast. Nobody has looked at whether the account is actually over its Bedrock quota, or whether the batch job and the interactive traffic even need to share the same capacity at the same moment.
Bedrock on-demand throughput is bounded by account-level service quotas: a requests-per-minute and a tokens-per-minute ceiling, set per model in each region. Cross the ceiling and the service returns ThrottlingException. The question underneath the noise is whether these throttles are transient bursts a good client can ride out, or a structural shortfall that no amount of retrying will fix.
What actually matters
Throttling is a symptom, and the same symptom has three different causes that want three different fixes. Reaching for retries first is right for one of them and useless for the other two, so the first thing to establish is which one you’re looking at.
A transient throttle is a short burst that briefly exceeds the per-minute ceiling while the average demand sits comfortably under it. This is what client-side resilience is for. Exponential backoff with jitter spreads the retries out so the burst drains and the retried requests land in a quieter window; the average was always servable, the arrivals were just clumpy. The AWS SDKs implement this for you: standard retry mode retries throttling and transient errors with backoff, and adaptive mode adds client-side rate limiting that slows the caller when it sees sustained throttling. Retries turn a jittery arrival pattern into a smooth one, and cost almost nothing when the underlying capacity is adequate.
A structural throttle is different: the sustained demand genuinely exceeds the quota, and no retry strategy adds a single token per minute of capacity. Retrying a structurally throttled workload just converts fast failures into slow ones and, if the whole fleet backs off and retries in step, into correlated stampedes. The fixes here change the capacity, not the client. You can request a service-quota increase for the model’s requests-per-minute or tokens-per-minute in that region. You can use a cross-region inference profile, which lets Bedrock route a request to one of several regions automatically, so the load draws on more than one region’s quota instead of piling onto one. Or you can buy Provisioned Throughput, which reserves a guaranteed floor of capacity for a model (billed hourly, with commitment terms) rather than sharing the on-demand pool. Each of these raises the ceiling; backoff never does.
Then there’s the shape of the demand itself, which you can change without touching the ceiling. A lot of throttling is self-inflicted synchronisation: a batch job that fires a thousand requests at once, or interactive and background work colliding at the same minute. Putting non-interactive work behind a queue, an SQS queue draining at a controlled concurrency, turns a spike into a steady stream that fits under the quota, and decouples the batch job’s timing from the interactive path so the two stop fighting over the same per-minute budget.
Two things decide which lever fits: latency tolerance and criticality. Interactive requests have seconds of budget at most, so their answer to overload is fast backoff then graceful degradation, shedding or deferring the request rather than making a user wait a minute. Background work has a generous latency budget, so it can absorb queueing and long backoffs invisibly. And when capacity is genuinely scarce, criticality decides what gives: shed or defer the low-priority work, and optionally fall back to a smaller or alternate model that has separate quota and lower cost, keeping the important path answered while the nice-to-have path waits.
What we’ll filter on
- Transient or structural? Is the average demand under the quota with clumpy arrivals, or genuinely over the ceiling?
- Latency tolerance, does this request have seconds to answer or minutes?
- Criticality, is this interactive work that must be served, or deferrable background work?
- Adds capacity or just reshapes arrivals? Does the lever raise the ceiling, or smooth the traffic under it?
- Time and cost to apply, an SDK setting today versus a quota request or a provisioned commitment.
The resilience landscape
Exponential backoff with jitter (SDK retries). The first line for transient throttles. On a ThrottlingException, wait a growing, randomised interval and retry, so retries from many callers don’t all fire at the same instant. The AWS SDKs give you this through retry mode: standard retries throttling and transient errors with backoff, adaptive adds client-side rate limiting that throttles the caller when it detects sustained pushback. Cheap, immediate, and the correct default. Its hard limit is that it adds no capacity: point it at a structurally over-quota workload and it just slows everything down.
Service-quota increase. Raise the account-level requests-per-minute or tokens-per-minute ceiling for a model in a region, through Service Quotas. The direct fix when demand has outgrown the default and you want more of the on-demand pool. It’s a request, not a switch, so it takes lead time and isn’t guaranteed, and it still leaves you on shared on-demand capacity with no reserved floor.
Cross-region inference profile. A profile that lets Bedrock automatically route a request to one of several regions, drawing on each region’s quota. Spreads load so a spike doesn’t concentrate on one region’s ceiling, and adds resilience if a region is busy. It needs the model available in the target regions and your data-residency rules to permit the routing, and it raises effective throughput rather than guaranteeing a floor.
Provisioned Throughput. Reserve dedicated model capacity for a guaranteed floor, billed hourly against a commitment term rather than per-token on-demand. The answer for steady, high-volume, latency-sensitive workloads that can’t tolerate on-demand throttling. It’s a cost commitment, so it fits predictable baseline load, not bursty or experimental traffic where you’d pay for idle reserved capacity.
Queue and controlled concurrency (SQS). Put non-interactive work behind a queue and drain it with a bounded number of workers, so a thousand-at-once batch becomes a steady stream that fits under the quota. Smooths demand and decouples background timing from the interactive path. It adds latency by design, so it suits deferrable work, not a user waiting on a reply.
Graceful degradation and model fallback. When capacity is genuinely scarce, shed or defer low-priority requests, and optionally fall back to a smaller or alternate model with separate quota and lower cost. Keeps the critical path answered under load instead of failing everything equally. The fallback model needs to be good enough for the degraded path, and you need a clear rule for what counts as low priority.
Side by side
| Lever | Fixes transient | Fixes structural | Adds capacity | Reshapes arrivals | Latency added | Time to apply |
|---|---|---|---|---|---|---|
| Backoff + jitter (SDK) | ✓ | ✗ | ✗ | ✓ | Seconds (on retry) | Immediate |
| Service-quota increase | ✓ | ✓ | ✓ | ✗ | None | Days (request) |
| Cross-region profile | ✓ | ✓ | ✓ | ✓ | Negligible | Hours to set up |
| Provisioned Throughput | ✓ | ✓ | ✓ | ✗ | None | Commitment term |
| Queue + concurrency (SQS) | ✓ | Partly | ✗ | ✓ | Seconds to minutes | Hours to build |
| Degradation / fallback | ✓ | ✓ | ✗ | ✗ | None (sheds instead) | Hours to build |
Reading the table against the situation: the interactive path wants backoff plus, if the average is genuinely over quota, a quota increase or cross-region profile, with degradation as the safety valve; the overnight batch job wants a queue so it stops colliding with daytime traffic; and if the interactive baseline is both high and steady, Provisioned Throughput buys it a floor. No single lever covers all of it, which is the point.
The picks in depth
Start by measuring, because the transient-versus-structural split decides everything and you can’t eyeguess it from the error count alone. Look at the model’s throttle metrics against the account quota over a representative window. If average requests-per-minute and tokens-per-minute sit under the ceiling and the throttles cluster in short spikes, it’s transient, and backoff is the whole answer. If the average is bumping the ceiling for sustained stretches, it’s structural, and no client change will help.
For the transient case, turn on the SDK’s retry behaviour rather than writing your own loop. Adaptive retry mode gives you exponential backoff, jitter, and client-side rate limiting that eases off when Bedrock is pushing back, which is exactly the behaviour a hand-rolled tight loop gets wrong. Cap the retry count and the total wait so an interactive request fails fast enough to degrade rather than hanging; unbounded retries are how a throttle becomes a latency incident.
For the structural case, pick the capacity lever by traffic shape. Bursty or still-growing traffic wants a quota increase and, if the model is available in more than one region and residency allows, a cross-region inference profile to spread the load; both raise the effective ceiling without a long-term commitment. Steady, high-volume, latency-sensitive traffic that can’t tolerate on-demand throttling at all is the case for Provisioned Throughput, where the hourly commitment buys a guaranteed floor. The trap is buying provisioned capacity for spiky or experimental load and paying for reserved throughput that sits idle between bursts.
The batch job is a demand-shape problem, not a capacity one. It fails because it fires everything at once and collides with interactive traffic, so the fix is a queue draining at controlled concurrency, which flattens the spike into a stream that fits under the quota and stops the two workloads competing for the same per-minute budget. It costs latency, which is free to spend on an overnight summarisation job and unaffordable on the interactive path, which is exactly why the two belong on different mechanisms.
Degradation is the safety valve underneath all of it. Even with the right capacity lever, a big enough spike can still exceed the ceiling, so decide in advance what sheds first: defer or drop the low-priority work, and if you have a smaller or alternate model with separate quota, route the degraded path to it rather than failing. That keeps the important requests answered when capacity is genuinely scarce, instead of spreading the pain evenly across everything.
A worked example: the collision at 2am
The overnight job summarises the day’s tickets. It reads a few thousand rows and, in a tight loop, fires a Bedrock request per ticket as fast as the code can iterate. Most nights it finishes before the interactive traffic wakes up. On the night the marketing emails go out at 2am local time, early-riser users start chatting to the assistant while the batch is still running, and both streams hit the same model’s per-minute token quota at once. Interactive requests throttle, the tight retry loop on the interactive path spins, and users watch a spinner for fifteen seconds.
The measurement shows the daytime interactive average sits comfortably under quota; the problem is purely that the batch spikes into the same minute. So the batch job goes behind an SQS queue drained by a small, fixed pool of workers, sized so its steady token rate leaves headroom under the ceiling for interactive traffic. The spike becomes a stream, the batch finishes an hour later than before (nobody notices; it’s a summary that’s read at 9am), and interactive throttling on collision nights disappears because the two workloads no longer arrive together.
On the interactive path itself, the hand-rolled retry loop is replaced with the SDK’s adaptive retry mode, capped so a request that can’t be served in a couple of seconds fails over to a degraded reply (“we’re busy, here’s a shorter answer”) backed by a smaller model with its own quota, rather than hanging. Two fixes for two different causes: the queue reshapes the demand that was colliding, the backoff-plus-degradation handles the residual bursts on the path that can’t wait. Neither of them is a bigger retry loop, and neither would have worked in the other’s place.
What’s worth remembering
- Throttling has three causes, transient bursts, structural over-quota demand, and colliding traffic shapes, and each wants a different fix; identify which before reaching for a lever.
- Exponential backoff with jitter is the right first response to a transient throttle, and it adds no capacity, so it does nothing for a workload that’s genuinely over quota.
- Use the SDK’s retry modes rather than a hand-rolled loop: standard adds backoff, adaptive adds client-side rate limiting that eases off under sustained throttling.
- Cap retries and total wait on interactive paths, so a throttle fails fast into degradation instead of turning into a latency incident.
- Bedrock on-demand throughput is bounded by account-level requests-per-minute and tokens-per-minute quotas, set per model per region; exceeding them returns ThrottlingException.
- Structural shortfalls are fixed by raising the ceiling: a service-quota increase, a cross-region inference profile to spread load across regions, or Provisioned Throughput for a guaranteed floor.
- Provisioned Throughput fits steady, high-volume, latency-sensitive load; it’s an hourly commitment, so it’s the wrong tool for spiky or experimental traffic that would leave reserved capacity idle.
- A queue with controlled concurrency reshapes demand rather than adding capacity, turning a batch spike into a steady stream and decoupling background work from the interactive path.
- Match the lever to latency tolerance and criticality: interactive work degrades or sheds under load; background work absorbs queueing and long backoffs invisibly.
- Decide in advance what sheds first, and keep a smaller or alternate model with separate quota as a fallback, so the critical path stays answered when capacity is genuinely scarce.