The situation
A team has fine-tuned a Bedrock foundation model on their own support transcripts and it tests well: noticeably better at their domain than the base model with a long prompt. They are ready to put it behind a live feature. The first surprise is that the on-demand invoke path they used all through prototyping, pay per token, no capacity to manage, refuses the custom model. Bedrock only serves a customised model through Provisioned Throughput, so before a single production request lands they have to decide how much capacity to reserve and for how long.
The traffic is not flat. Weekday business hours carry the bulk of it, with a sharp mid-morning peak when the support queue fills, near silence overnight, and a long quiet tail at weekends. Someone has pulled a number for the busiest minute: roughly the token volume the feature has to sustain when the queue is at its worst. Finance wants the cheapest per-unit rate, which means a six-month commitment. Engineering has been burned before by locking in capacity a fortnight before a traffic pattern changed, and wants to know what the commitment actually buys and what it forecloses.
Underneath the calendar question is a sizing question. Reserve too little and the mid-morning peak throttles real users; reserve too much and idle units bill around the clock for throughput nobody consumes.
What actually matters
The first thing to be clear-eyed about is that this is not a provisioned-versus-on-demand choice. For a base foundation model it would be: on-demand bills per token with no floor, Provisioned Throughput reserves guaranteed capacity, and most base-model workloads are happy on demand. A customised model removes the option. Fine-tune a model, or bring your own weights, and Provisioned Throughput is the only serving path, so the real decision narrows to how many units and on what term.
Capacity is reserved in model units, and a model unit is the thing worth understanding properly. One unit delivers a defined throughput for one specific model: a quota of input tokens per minute and output tokens per minute that the unit can sustain. It is not a share of a pool and it is not burstable goodwill; it is a fixed rate you have reserved. Two units give you twice the rate. The exact tokens-per-minute a unit provides depends on the model, so the sizing arithmetic starts from the per-unit figure for your specific customised model, not a generic number.
Then the cost shape, which is the trap. You pay for reserved units by the hour they exist, whether or not traffic fills them. A unit provisioned for a peak that lasts ninety minutes a day is still billing for the other twenty-two and a half hours. That is the over-provisioning failure: capacity sized to the worst minute, paid for around the clock, mostly idle. The opposite failure is under-provisioning, where the reserved rate is below the real peak and Bedrock throttles requests over the quota, so the mid-morning surge turns into errors and retries for actual users. Sizing lives between those two, and headroom is how you buy safety against the second without drowning in the first.
The commitment term is a separate lever from the unit count, and it trades price against flexibility. Provisioned Throughput can be bought three ways: no commitment, billed hourly at the highest per-unit rate, which you can adjust or release as demand moves; a one-month commitment at a lower per-unit rate; and a six-month commitment at the lowest per-unit rate, locked for the term. The cheaper rates are a reward for certainty. Take the six-month rate on a workload whose shape you are still learning and you have converted a variable cost into a fixed bet on a forecast.
The last thing that matters is that demand shape decides how well any of this fits. Steady, predictable load maps cleanly onto reserved units and rewards a commitment, because the units you pay for are the units you use. Spiky load with deep troughs is the awkward case: size to the peak and you pay for idle troughs, size to the average and the peaks throttle. Neither the unit count nor the term fixes a genuinely spiky profile on its own; it just moves where the pain sits.
What we’ll filter on
- Custom or base model, is Provisioned Throughput forced, or is on-demand still an option worth keeping?
- Peak throughput, what is the busiest-minute demand in input and output tokens per minute, and what does one model unit deliver for this model?
- Demand shape, steady and predictable, or spiky with long idle troughs?
- Commitment appetite, how confident is the traffic forecast over one month and over six?
- Cost of idle versus cost of throttling, which failure hurts this feature more, a bigger bill or dropped peak requests?
The throughput landscape
On-demand invocation. Pay per token processed, no capacity to reserve, no floor, no commitment. This is the default for base foundation models and it is where the prototype lived. For a customised model it is simply not available, so for this workload it drops out of contention regardless of how attractive its billing shape is.
Provisioned Throughput, no commitment. Reserve model units billed by the hour, at the highest per-unit rate, with the freedom to change the unit count or release the reservation as you learn the traffic. This is the term for a workload whose shape you do not yet trust, or one you expect to run only for a bounded window. You pay a premium per unit for the right to walk away.
Provisioned Throughput, one-month commitment. The same reserved units at a lower per-unit rate in exchange for holding them for a month. A reasonable middle when the near-term traffic is understood but the half-year is not, and a common way to run a steady production workload without betting six months on it.
Provisioned Throughput, six-month commitment. The lowest per-unit rate, locked for the term. This is the right call only when the demand is both steady and confidently forecast that far out, because the saving is real but the flexibility is gone; a traffic change inside the term does not release you from the units.
The unit count is orthogonal to the term. Each option above is bought as some number of model units, and the number comes from the same peak-tokens-per-minute arithmetic in every case. The term chooses the price and the lock; the unit count chooses the ceiling.
Side by side
| Option | Serves a custom model | Per-unit price | Flexibility | Idle-cost exposure | Best for |
|---|---|---|---|---|---|
| On-demand | ✗ | Per token, no floor | Highest | None (no reservation) | Base models, spiky or exploratory load |
| PT, no commitment | ✓ | Highest | High (adjust or release hourly) | Per reserved unit-hour | Unproven traffic, bounded-window runs |
| PT, one-month | ✓ | Lower | Medium (held one month) | Per reserved unit-hour | Understood near-term production load |
| PT, six-month | ✓ | Lowest | Low (locked six months) | Per reserved unit-hour | Steady, confidently forecast load |
Reading the table against the situation: on-demand is off the table because the model is customised, so the whole decision is which Provisioned Throughput term to take and how many units to reserve. Finance is reaching for the bottom row; engineering’s caution about the traffic forecast is an argument for one of the middle two until the shape is proven.
The picks in depth
Start with the unit count, because the term is worthless if the ceiling is wrong. Take the busiest-minute demand, in both input and output tokens per minute, and divide by what one model unit delivers for this specific customised model, taking whichever of input or output is the binding constraint. That gives the raw number of units to cover the peak. Then add headroom rather than rounding to the exact peak, because the peak you measured is an average over a minute and real traffic is burstier inside that minute, and because a fine-tuned model’s output length can drift as prompts evolve. Headroom is the cheap insurance against throttling; the exact margin is a judgement about how spiky the minute really is and how much a throttled request costs the feature. Size to the peak plus that margin, not to the daily average, or the mid-morning surge throttles every day.
Now the idle problem the peak sizing creates. A unit count set to the worst minute bills at that level for all twenty-four hours, including the overnight silence and the weekend tail. There is no autoscaling that quietly follows the curve down for you here; reserved units are reserved until you change them. If the trough is deep and long, the honest question is whether the throttling cost at a lower unit count is genuinely worse than the idle cost at the peak count. Sometimes accepting a little throttling at the very tip of the peak, and sizing to something below the absolute maximum, is cheaper overall than paying all night for headroom used ninety minutes a day. That is a per-feature call, and it turns on whether a throttled request degrades gracefully with a retry or hard-fails a user.
The term is the last decision and the reversible-versus-locked one. If the traffic forecast is honest only a few weeks out, the no-commitment or one-month term keeps the unit count adjustable while you watch the real curve, and the premium per unit is the price of not betting on a number you do not yet trust. Once a month or two of production data shows the peak is stable, converting the proven baseline to a longer commitment captures the lower rate on the capacity you now know you need. A defensible pattern is to commit the steady floor and top up the uncertain margin on a shorter term, so the lock only ever covers demand you are confident in. Jumping straight to six months on day one, before any production traffic has been seen, is the move most likely to end in either idle units you cannot release or a lock that no longer fits the curve.
One base-model note, because the two cases are easy to blur: a base foundation model can also be put on Provisioned Throughput, usually to guarantee a throughput floor for a latency-sensitive or high-volume workload that on-demand quotas would throttle. That is a legitimate but uncommon choice, since most base-model traffic is happier on demand. The asymmetry is the thing to hold onto: base models may use Provisioned Throughput, custom models must.
A worked example: sizing the fine-tuned support model
Call the busiest minute the moment the support queue peaks mid-morning. Suppose measurement puts that minute at a demand the team can express as tokens per minute in and out, and the per-unit throughput for their fine-tuned model covers a known fraction of it, so the peak divides out to, say, three units of raw coverage. Output tokens turn out to be the binding side because the drafted replies are long, so the arithmetic is done against the output rate, not the input rate.
Rounding to three units exactly would meet the average of the peak minute and throttle the bursts inside it, so they size to four: three for the measured peak, one for headroom against intra-minute spikes and output drift. Four units it is, as the ceiling.
Then the idle question. Those four units bill all night and all weekend, when demand is near zero. The team looks at the curve and decides the deep trough does not justify a second, smaller off-peak reservation, because the operational cost of resizing twice a day outweighs the saving, but they note it as a lever if the bill bites later.
On the term, they hold back from six months. The feature is new and the peak could move as adoption grows, so they take the four units on a one-month commitment: lower than the no-commitment rate, still adjustable at the next boundary. Two months of production data later, three of those four units are demonstrably the stable floor and the fourth is genuine swing capacity. They convert the three-unit floor to a six-month commitment for the lowest rate on the capacity they now trust, and keep the fourth unit on the shorter term where the uncertainty lives. The lock only ever covers demand they have actually seen.
What’s worth remembering
- A customised model, fine-tuned or imported, cannot be invoked on demand; Provisioned Throughput is its only serving path, so the decision is unit count and term, not provisioned-versus-on-demand.
- Capacity is reserved in model units, and one unit delivers a fixed input and output tokens-per-minute rate for one specific model; the exact rate depends on the model, so size from that model’s per-unit figure.
- You pay for reserved units by the hour they exist, filled or idle, which makes peak-sized capacity expensive across the overnight and weekend troughs.
- Size the unit count from busiest-minute tokens per minute, against whichever of input or output binds, then add headroom; sizing to the daily average throttles the peak.
- Over-provisioning burns money on idle units; under-provisioning throttles real requests at the peak; the sizing lives between the two and headroom buys safety against the second.
- The commitment term is a separate lever: no commitment bills hourly at the highest per-unit rate but stays adjustable, one-month is cheaper, six-month is cheapest and locked.
- The cheaper commitment rates are a reward for certainty; taking six months before any production traffic is seen bets a fixed cost on an unproven forecast.
- Steady, predictable demand rewards a commitment; spiky demand with deep troughs is the awkward case, and neither the unit count nor the term fixes it alone.
- A defensible pattern is to commit the proven steady floor on a longer term and keep the uncertain swing capacity on a shorter one, so the lock only covers demand you trust.
- Base foundation models may use Provisioned Throughput for a guaranteed floor but rarely need to; custom models have no other option.