The situation
A product team has two generative-AI features heading for production. The first is a customer-facing assistant that answers billing and account questions in natural language, spiky traffic that peaks during business hours and goes quiet overnight. The second is a batch job that runs every night over a large backlog of long case files, summarising each into a plain-language brief, steady and predictable, and it needs to be fine-tuned on the company’s own domain language and house style to produce summaries the compliance team will sign off on.
Right now both features call a proprietary foundation model through Amazon Bedrock on demand, billed per token. The assistant is fine that way. The summarisation job is not: the fine-tuning it needs is limited to what the managed model exposes, the per-token bill on millions of documents a night is climbing fast, and the compliance team has started asking whether the model weights and the training data ever leave AWS, and whether the company could keep serving the model if the vendor changed terms.
So the same question lands on both features from opposite directions. One team wants to keep the managed simplicity it already has. The other wants control it can’t get from a black-box endpoint, and is willing to run infrastructure to get it. Underneath both is one decision: for this workload, do you rent a model someone else operates, or do you take the weights and host them yourself?
What actually matters
The headline trade is control against managed simplicity, and it is worth being precise about what each side actually buys. A proprietary model on Bedrock is a fully managed, serverless endpoint: no capacity to provision, no scaling to tune, no GPU to keep warm, and you are calling a model that is often stronger out of the box than anything you would host. What you give up is visibility and portability. You cannot see the weights, you cannot export them, your customisation is bounded by what the managed service exposes, and the model runs on the vendor’s terms. An open-weight model inverts every one of those. You can download the weights, fine-tune them as deeply as you like, keep them, move them, and inspect what you are running, at the price of owning the serving infrastructure and everything that comes with it.
The second thing that actually decides the answer is the cost curve, because the two hosting styles bill in fundamentally different shapes. Managed on-demand inference is per-token: you pay for exactly what you call and nothing when idle, which is ideal for spiky or low-volume traffic. Self-hosting is per-instance-hour: you pay for the endpoint whether it is busy or idle, so the economics only work once utilisation is high enough that the hourly cost divided across your tokens beats the per-token rate. A quiet, bursty assistant wants per-token billing; a saturated, round-the-clock batch job is exactly where a reserved per-hour endpoint starts winning. Getting this backwards, self-hosting a low-traffic feature or pushing a huge steady load through per-token pricing, is the most common way the bill goes wrong.
Data and weight residency is the third axis, and it is where compliance requirements do the deciding. With a managed proprietary model you control where your prompts and outputs go under the service’s data terms, but the weights themselves are never yours and never portable. With an open-weight model you hold the weights and the fine-tuned artefact, so you can guarantee they stay inside an account, a region, or a VPC, and you are not exposed to a vendor changing access or pricing on a model you have built a product around. If the requirement is “we must be able to keep running this exact model regardless of any vendor”, only owning the weights satisfies it.
Then there is licensing, which people skip and later regret. Open-weight does not mean unrestricted. Some open models ship under a permissive licence like Apache 2.0; others carry community licences with conditions on commercial use above a user threshold, on using outputs to train other models, or on acceptable-use terms. The licence travels with the weights, so it has to be read before a model is baked into a product, not after.
The last two are latency and throughput needs, and the team’s own operational maturity. A self-hosted endpoint lets you pin instance type, region, and autoscaling policy to hit a specific latency and throughput target, which a shared managed endpoint cannot promise as tightly; but reaching for that control assumes a team that can actually run inference infrastructure, size GPUs, configure autoscaling, patch, monitor, and optimise for cost. Handing a model’s operations to a team without the MLOps maturity to carry it is how a self-hosted endpoint becomes a stalled project and a surprise bill. Managed serving exists precisely so a team can skip all of that.
What we’ll filter on
- Control and customisation, does the workload need weight access and deep fine-tuning, or is a managed model’s tuning enough?
- Cost shape against load, is traffic spiky and low-volume (favours per-token) or steady and high-volume (favours per-hour)?
- Data and weight residency, must the weights and fine-tuned artefact stay portable and under your control?
- Licensing, does the open model’s licence permit the intended commercial use?
- Latency and throughput, does the feature need a pinned, dedicated serving target?
- Operational maturity, can the team actually run and optimise inference infrastructure?
The hosting landscape
Proprietary model on Bedrock, on-demand. A fully managed, serverless call to a foundation model whose weights you never see, billed per input and output token. Nothing to provision, scales automatically, and typically the strongest model for the least operational effort. Customisation is bounded by what the service exposes, and you cannot export or self-serve the model. This is the default for spiky, low-to-moderate traffic where managed simplicity is worth more than control.
Proprietary model on Bedrock, Provisioned Throughput. The same managed model, but with dedicated capacity reserved and billed per Model unitThe billing block Provisioned Throughput is sold in – one unit delivers a fixed tokens-per-minute rate for a specific model. rather than per token, optionally at a discount for a term commitment. It buys guaranteed throughput and steadier latency for high, predictable volume, while keeping the model itself a black box. You still cannot see or move the weights; you have only changed the billing shape from per-token to per-hour.
Open-weight model on Bedrock (managed). Bedrock also serves a selection of open-weight models, so you can call one through the same managed, per-token API you use for proprietary models. You get the operational simplicity of Bedrock over an open architecture, with managed fine-tuning where offered, but you are still consuming it as a service rather than holding the weights yourself. The Bedrock Marketplace widens this catalogue, with some models deployed to managed endpoints you provision.
Bedrock Custom Model Import. Bring your own open-weight or customised model, weights you have fine-tuned elsewhere, into Bedrock’s managed serverless serving path, provided the architecture is one Bedrock supports. This is the middle ground: you own and customise the weights, but Bedrock handles the serving and you skip running the infrastructure. Billing is by the Custom Model Units your imported model occupies, charged in five-minute windows and only while the model is serving; after five idle minutes Bedrock scales the copies to zero and the charge stops, at the price of a cold start of tens of seconds on the next call. So it fits teams that want their own model without the operational load of an endpoint, and it pays nothing between bursts.
Open-weight model self-hosted on SageMaker. Deploy an open-weight model to a SageMaker real-time endpoint (JumpStart makes many of them one-click) on instances you choose, with autoscaling policies you set, billed per instance-hour. Full control over fine-tuning, instance type, and serving configuration, and the weights stay in your account. You own capacity planning, scaling, patching, and cost optimisation, and you pay for the endpoint whether it is busy or idle. SageMaker serverless and asynchronous inference options soften the idle-cost problem for intermittent loads.
Open-weight model self-hosted on EC2. The maximum-control end: run the model on GPU instances you manage directly, with your own serving stack. Total flexibility over every layer, and total responsibility for it, from driver versions to load balancing to keeping the GPUs utilised. Rarely the right first choice unless a requirement genuinely rules out the managed layers above it.
Side by side
| Hosting option | Weight access and portability | Deep fine-tuning | Billing shape | Ops burden | Scaling control | Best for |
|---|---|---|---|---|---|---|
| Proprietary on Bedrock, on-demand | ✗ | ✗ | Per token | Lowest | Automatic | Spiky, low-volume, managed simplicity |
| Proprietary on Bedrock, Provisioned Throughput | ✗ | ✗ | Per model-unit-hour | Low | Reserved capacity | High, steady volume on a black-box model |
| Open-weight on Bedrock (managed) | ✗ | Partial (managed) | Per token | Low | Automatic | Open architecture, managed serving |
| Bedrock Custom Model Import | ✓ | ✓ | Per serving capacity | Low-medium | Managed | Your own weights, no endpoint to run |
| Open-weight on SageMaker | ✓ | ✓ | Per instance-hour | High | You configure | Steady load, deep control, weights in-account |
| Open-weight on EC2 | ✓ | ✓ | Per instance-hour | Highest | You build it | Requirements the managed layers can’t meet |
The pattern in the table is that portability and deep customisation move together and only start at Custom Model Import, while operational burden climbs in step with control. Everything above the middle line trades weight ownership for a managed, mostly per-token life; everything below trades operational effort for weights you hold and a per-hour cost curve.
The picks in depth
The spiky assistant should stay a managed proprietary model on Bedrock, on demand. Its traffic is bursty and idle overnight, which is exactly the shape per-token billing is built for: it pays nothing while quiet and scales automatically through the peak with no capacity to plan. The feature does not need weight portability or deep fine-tuning, so the control an open-weight model would buy is control it has no use for, bought at the cost of an endpoint someone now has to run. If the assistant later grows into a high, flat daytime load, the first move is not to self-host but to switch that same model to Provisioned Throughput, keeping the managed model and only changing the billing shape from per-token to per-hour to steady the latency and cap the cost.
The nightly summarisation job is the case that justifies leaving the managed on-demand path. It needs fine-tuning deeper than the managed model exposes, its load is steady and high-volume so a per-hour endpoint beats per-token at that utilisation, and compliance wants the weights and the fine-tuned artefact to stay in the account and stay portable regardless of any vendor. That is three of the six filters pointing the same way: control, cost shape, and residency all favour owning the weights. The open question is only how much infrastructure the team wants to run. If they have the MLOps maturity, a self-hosted SageMaker endpoint on right-sized GPU instances, fine-tuned on their own case files and autoscaled to the batch window, gives them everything. If they want the owned, fine-tuned weights without operating an endpoint, Custom Model Import is the lighter path, and for a job that only wakes up at night it can be the cheaper one too: it bills in five-minute windows while the batch is running and scales the copies to zero once the run goes quiet, so the idle hours between nightly runs cost nothing, where a real-time endpoint left standing bills by the hour whether or not the batch is working. They fine-tune the open model, import the weights into Bedrock’s managed serving, and keep the customisation and portability while Bedrock carries the serving. Both keep the weights theirs; they differ in how much of the stack the team holds and in what the idle hours cost.
The one filter that can override all of this is licensing, and it has to be checked before either team commits. An open-weight model is only an option if its licence permits the intended commercial use; a community licence with a user-count threshold or an output-reuse restriction can quietly rule a model out of a product that would otherwise fit it perfectly. Read the licence attached to the specific weights, because it travels with them into whatever you build.
A worked example: the batch job’s cost crossover
Put rough numbers on why the batch job flips and the assistant does not. Say the summarisation job processes two million documents a night, several thousand tokens each once the source text and the generated brief are counted, comfortably into the billions of tokens a night, every night. On per-token managed pricing that is a large, flat bill that scales linearly with the backlog and never sleeps, because the load is constant. A self-hosted endpoint sized for that throughput costs a fixed number of instance-hours a night whether it processes 1.8 or 2.2 million documents, so above the utilisation where the hourly cost spread across the tokens drops below the per-token rate, the endpoint is simply cheaper, and it keeps getting cheaper per document as the backlog grows.
The assistant is the mirror image. Its traffic is a few thousand conversations clustered in business hours and almost nothing overnight, so a reserved endpoint would sit idle for most of the day while still billing by the hour. Per-token pricing charges it only for the conversations it actually has and nothing for the quiet hours, so managed on-demand stays the cheaper shape no matter how the team tunes an endpoint. Same company, same choice of hosting styles, opposite answers, and the deciding variable is not the model’s quality but the load curve meeting the billing curve. Fine-tuning depth and weight residency then decide which owned-weight path the batch job takes; utilisation alone decides that it takes one at all.
What’s worth remembering
- The core trade is control against managed simplicity: a proprietary model on Bedrock hides the weights and just works, while an open-weight model gives you the weights and hands you the operations.
- Managed on-demand inference bills per token and costs nothing idle, so it fits spiky, low-to-moderate traffic; self-hosting bills per instance-hour and only wins once utilisation is high and steady.
- Getting the cost shape backwards, self-hosting a bursty feature or pushing a huge steady load through per-token pricing, is the most common way the bill goes wrong.
- Only owning the weights guarantees residency and portability; a managed proprietary model never lets you export or independently keep serving the weights.
- Provisioned Throughput changes a managed model’s billing from per-token to per-model-unit-hour for high, predictable volume without giving you the weights.
- Bedrock Custom Model Import is the middle ground: bring your own fine-tuned open weights into managed serverless serving and skip running an endpoint, for supported architectures.
- Self-hosting on SageMaker or EC2 buys deep fine-tuning, pinned latency, and weights that stay in your account, paid for with real operational burden.
- Open-weight is not licence-free: check the specific model’s licence for commercial-use, user-threshold, and output-reuse conditions before building on it.
- Don’t reach for self-hosting without the MLOps maturity to size, scale, patch, and cost-optimise inference, or the endpoint becomes a stalled project and a surprise bill.
- Let the workload decide per feature: the same team can rightly keep a managed proprietary model for one feature and self-host an open-weight model for another.