Choosing an Inference Option for a GenAI Workload

The situation

A product team runs three GenAI workloads that all got built on whatever inference path was closest to hand, and the bill is now a mess of half-idle endpoints and throttling errors that nobody can explain.

The first is a support-reply assistant embedded in the agent console. It calls a Bedrock foundation model, sees steady traffic during business hours, roughly 15 to 40 requests per second, and needs a first token back fast because a human is waiting. The second is a nightly enrichment job: 4 million historical tickets get summarised and classified once, offline, with nothing waiting on the result before morning. The third is a fine-tuned open-weight model the data-science team trained on the company’s own taxonomy; it powers an internal triage tool that gets hammered for twenty minutes after each standup and then sees almost nothing for hours.

Three workloads, three completely different shapes. The question is which serving option each one wants, and why the answer isn’t the same for all of them.

What actually matters

Inference cost and inference pain both come from the same place: paying for capacity you aren’t using, or not having capacity when a request arrives. Every serving option on AWS is really a different answer to “who holds the capacity, and when do you pay for it.” Get the match right and the workload is cheap and calm; get it wrong and you’re either burning money on an idle endpoint or eating throttles at peak.

The first axis is latency sensitivity. If a human is waiting on the first token, cold starts and queue time are unacceptable and you pay for warm capacity to avoid them. If the result is consumed minutes or hours later, latency is nearly free to trade away, and that trade is where the big savings live.

The second is traffic shape: steady, spiky, or offline. Steady traffic wants persistent capacity sized to the load. Spiky, intermittent traffic wants something that scales to zero between bursts so you aren’t paying for idle. Offline, run-it-all-at-once traffic wants a batch mechanism that spins up, chews through the dataset, and shuts down, with no endpoint to babysit.

The third is throughput guarantees. On-demand serving shares a pool and is subject to account-level quotas; under real contention you can be throttled. When a workload must have a floor of guaranteed throughput, or must serve a model that on-demand simply won’t host, you reserve capacity and pay by the hour whether you use it or not.

The fourth is the cost model itself: per-token, per-hour, or per-job. Per-token has no floor and scales with use, which is ideal until volume is both high and predictable, at which point reserved per-hour capacity gets cheaper. Per-job (batch) is the cheapest per unit of work but only exists for latency-tolerant workloads.

The last axis decides which half of the menu you’re even ordering from: is the model a Bedrock-managed foundation model, or a self-hosted open-weight or custom model? Bedrock serves the managed FMs (and custom or imported models, with a caveat below). Anything you brought yourself, an open-weight checkpoint you fine-tuned, a bespoke architecture, lives on SageMaker hosting. That single fact splits the decision tree before any of the other axes come into play.

What we’ll filter on

1. Model provenance: a Bedrock-managed foundation model, or a self-hosted / custom model?
2. Latency sensitivity: is a human (or a synchronous caller) waiting on the response?
3. Traffic shape: steady, spiky and intermittent, or offline batch?
4. Throughput guarantee: best-effort shared quota, or a reserved floor?
5. Cost model that fits: per-token, per-hour reserved, or per-job?

The inference landscape

Bedrock on-demand. Pay per input and output token with no commitment and nothing to provision. You call InvokeModel or Converse and Bedrock serves it from a shared pool. This is the default for foundation-model workloads and the right starting point for almost anything interactive with variable or unpredictable volume. The constraint is that throughput is governed by account-level service quotas (requests and tokens per minute per model); a busy workload can hit ThrottlingException under contention, and cross-region inference profiles exist partly to spread that load.

Bedrock Provisioned Throughput. Reserve capacity in model units, each unit delivering a defined throughput for a specific model, billed per hour on a commitment (hourly with no term, or a discounted one- or six-month term). Two reasons to reach for it: you need a guaranteed throughput floor that on-demand quotas won’t promise, or you’re serving a customised model (a fine-tuned or imported model on Bedrock), which requires Provisioned Throughput to serve at all. The trade is that you pay for the reserved units whether or not traffic fills them, so it only makes sense when volume is high and steady enough to keep them busy.

Bedrock batch inference. Submit a large set of records as a single asynchronous job (CreateModelInvocationJob), pointing at input in S3 and getting output back in S3 when it finishes. It runs at roughly half the on-demand per-token price, and in exchange you give up interactivity: the job is queued and completes on its own schedule, so it fits offline workloads where nothing is waiting. This is the natural home for the nightly enrichment job, not an endpoint.

Cross-region inference profiles. A routing construct that spreads invocations across regions to raise effective throughput and smooth out throttling, rather than a distinct serving mode. Worth naming here so it’s on the map; it has its own coverage, and for this decision it’s a modifier on on-demand rather than a fourth option.

Then the workload isn’t a Bedrock FM at all, and you’re on SageMaker hosting, which offers four serving shapes:

SageMaker real-time endpoints. A persistent HTTPS endpoint backed by one or more always-on instances, autoscaling on load. Lowest and most consistent latency, and the right choice for steady, latency-sensitive traffic, but you pay for the instances around the clock, including idle time. This is where a self-hosted model serving steady interactive traffic belongs.

SageMaker Serverless Inference. An endpoint that provisions compute on demand and scales to zero when idle, billing only for the compute during a request plus the data processed. It tolerates cold starts (the first request after idle pays a startup penalty), which makes it a strong fit for spiky, intermittent traffic where paying for an always-warm instance would be mostly waste. The fine-tuned triage model that’s busy for twenty minutes and idle for hours is close to the textbook case.

SageMaker Asynchronous Inference. A queued endpoint for requests with large payloads or long processing times: you submit a request pointing at an S3 input, it’s placed on an internal queue, processed, and the result written to S3, with an optional SNS notification. It can also scale to zero when the queue is empty. Use it when a single inference is heavy or slow (large documents, long generations) and the caller can collect the result asynchronously rather than holding a synchronous connection open.

SageMaker Batch Transform. Offline scoring of an entire dataset with no persistent endpoint at all: point a transform job at data in S3, it spins up instances, processes every record, writes results back to S3, and tears the instances down. The SageMaker analogue of Bedrock batch inference, for self-hosted models. If the nightly job used a custom model instead of a Bedrock FM, this is where it would run.

One more worth a mention: SageMaker multi-model and multi-container endpoints let a single endpoint host many models behind one set of instances, loading them in and out of memory on demand. When you have dozens or hundreds of models each with light, sporadic traffic, this shares the compute across all of them instead of standing up an endpoint per model.

Side by side

Option	Model type	Latency fit	Traffic shape	Scales to zero	Cost model
Bedrock on-demand	Managed FM	✓ interactive	Variable / spiky	✓ (no floor)	Per-token
Bedrock Provisioned Throughput	Managed / custom FM	✓ interactive	Steady, high volume	✗	Per-hour, per model unit
Bedrock batch inference	Managed FM	✗ offline	Offline batch	✓ (per-job)	Per-job, ~half on-demand
SageMaker real-time	Self-hosted	✓ interactive	Steady	✗	Per-instance-hour
SageMaker Serverless	Self-hosted	✓ (cold starts)	Spiky / intermittent	✓	Per-request compute
SageMaker Async	Self-hosted	✗ synchronous	Long / heavy payloads	✓	Per-instance-hour (queued)
SageMaker Batch Transform	Self-hosted	✗ offline	Offline batch	✓ (per-job)	Per-job instance-hours

The two halves of the table never compete directly; model provenance picks the half, then latency and traffic shape pick the row.

The picks in depth

The support assistant lands on Bedrock on-demand. It’s a managed foundation model, a human is waiting, and traffic is variable within the business day. On-demand gives sub-second first-token latency with no idle cost and no capacity to plan. The one thing to watch is quotas: at 15 to 40 requests per second the workload can brush against per-minute request and token limits, so monitor ThrottlingException rates, request quota increases where the ceiling is real, and consider a cross-region inference profile to raise effective throughput before reaching for Provisioned Throughput. You only graduate to reserved model units if volume becomes high and steady enough that the per-hour maths beats per-token, or if a latency SLA demands a guaranteed floor.

The nightly enrichment job lands on Bedrock batch inference. Four million tickets, offline, nothing waiting: this is the definition of latency-tolerant, and running it through synchronous InvokeModel calls would both cost twice as much per token and fight the interactive workload for the same quota. Batch inference takes the records from S3, runs them as one managed asynchronous job at roughly half the on-demand price, and writes results back to S3. Size the input, kick the job off after hours, collect the output by morning. If this job used a self-hosted model instead, the equivalent move would be a SageMaker Batch Transform job.

The triage tool lands on SageMaker Serverless Inference. It’s a fine-tuned open-weight model, so it can’t live on Bedrock’s managed-FM paths at all; it belongs on SageMaker hosting. The traffic is the deciding factor: a twenty-minute burst after standup and near silence otherwise. A real-time endpoint would sit warm and billing all day for a fraction of use. Serverless Inference scales to zero between bursts and only bills for compute during requests, and the workload tolerates the cold-start penalty on the first call after idle (an internal tool, not a customer-facing SLA). If the bursts grew into steady all-day load, the calculus would flip toward a real-time endpoint; if a single request became large or slow, Asynchronous Inference would be the queued alternative.

There’s a subtlety worth stating plainly. A fine-tuned model can end up on either half of the tree depending on how it was made. Fine-tune or import a model into Bedrock and it serves on Bedrock, but only via Provisioned Throughput, not on-demand. Fine-tune an open-weight checkpoint yourself and it serves on SageMaker hosting. Same phrase, “we fine-tuned a model,” two entirely different serving decisions, so establish which one before picking anything.

A worked example: one team, three bills

Put numbers on the three workloads and the shapes separate cleanly.

The support assistant runs about 20 requests per second for eight business hours, call it 576,000 requests a day, each a few hundred tokens in and out. On-demand per-token pricing tracks that usage exactly and drops to near zero overnight; there is no idle floor to pay for, and the only operational task is quota headroom. Moving it to Provisioned Throughput would mean paying for reserved model units 24 hours a day to cover an 8-hour load, which only pays off if the day-time volume is high enough to keep those units saturated.

The enrichment job processes 4 million records once a night. As synchronous on-demand calls it would pay full per-token rate and contend with the assistant’s quota; as a batch job it pays roughly half and runs in its own lane. The saving is close to 50% on 4 million records of input and output tokens, every night, for the cost of accepting a result that lands by morning instead of instantly.

The triage tool sees maybe 400 requests in a twenty-minute window and a trickle afterwards. A single always-on real-time instance sized for the burst would bill 24 hours to serve well under an hour of real work. Serverless Inference bills only the compute the requests actually consume, so the idle 23 hours cost nothing, and the cold start on the first post-standup call is a couple of seconds the internal users won’t notice.

Same team, three workloads, three different serving options, and each choice falls out of the traffic shape and the model’s provenance rather than any property of the model itself.

What’s worth remembering

1. Model provenance splits the decision before anything else: Bedrock-managed foundation models go on Bedrock’s serving paths; self-hosted and open-weight models you trained go on SageMaker hosting.
2. Bedrock on-demand is the default for interactive FM workloads: per-token, no commitment, no idle cost, but throughput is bounded by account quotas, so watch for throttling and use cross-region inference profiles to spread load.
3. Bedrock Provisioned Throughput reserves model units by the hour for a guaranteed throughput floor, and is required to serve customised or imported models on Bedrock; it only pays off when volume is high and steady.
4. Bedrock batch inference runs large record sets as an async S3-to-S3 job at roughly half the on-demand price; reach for it whenever the workload is offline and nothing is waiting on the result.
5. SageMaker real-time endpoints give the lowest, most consistent latency for steady interactive traffic, at the cost of paying for always-on instances including idle time.
6. SageMaker Serverless Inference scales to zero and bills per request, which fits spiky, intermittent traffic that can absorb a cold start on the first call after idle.
7. SageMaker Asynchronous Inference queues requests with long or heavy payloads, writes results to S3, and can also scale to zero; use it when a single inference is slow and the caller can collect it later.
8. SageMaker Batch Transform scores an entire dataset with no persistent endpoint, the self-hosted analogue of Bedrock batch inference.
9. Multi-model and multi-container SageMaker endpoints share one set of instances across many lightly-used models, which beats standing up an endpoint per model when you host dozens of them.
10. The word “fine-tuned” doesn’t decide the serving path on its own: a model tuned into Bedrock serves via Provisioned Throughput, while an open-weight model you tuned yourself serves on SageMaker; confirm which before you pick.