Delivering Responses: Sync, Async, or Streaming

The situation

A product team is putting three generative features into one application on Amazon Bedrock. The first is a chat assistant embedded in a web app: a person types a question and expects a reply to start appearing quickly. The second is a document summariser triggered from a form: a user uploads a long contract and wants a one-page summary, which the model takes twenty to forty seconds to produce. The third is an overnight enrichment job that runs a classification prompt over roughly forty thousand support tickets and writes the results to a warehouse.

All three currently call the model the same way: a synchronous request behind Amazon API Gateway and AWS Lambda, the caller blocking until the full completion returns. The chat feature feels sluggish because nothing appears on screen until the whole answer is finished. The summariser intermittently returns a 504 to the browser, because the completion sometimes runs past the gateway’s integration timeout. And the nightly job takes hours and occasionally trips Lambda’s fifteen-minute ceiling on the larger tickets, so it has been split into fragile retrying chunks.

One delivery pattern is being asked to serve three very different interaction shapes, and it is a poor fit for at least two of them.

What actually matters

The property that decides the most is whether a human is waiting on the other end and watching. An interactive chat lives and dies on perceived latency: the felt speed is how soon something starts appearing, not how soon it finishes. A background job has no one watching, so total throughput and cost matter far more than the first-token moment. Getting these two backwards is where most of the pain in this scenario comes from, so it is worth naming the interactivity of each feature before reaching for a transport.

Output length is the second force, and it interacts badly with request timeouts. A synchronous call holds a connection open for the entire generation, so the longer the completion runs, the closer it creeps to whatever the transport in front of it will tolerate. API Gateway’s integration timeout sits around twenty-nine to thirty seconds, and Lambda caps a single invocation at fifteen minutes. A long summary or a large batch can blow through the first limit and, at the extreme, the second. Short answers rarely brush these ceilings; long or unbounded ones need a delivery mode that does not hold one connection open for the whole job.

The third is transport complexity, and it is a real cost, not a footnote. A plain synchronous request is the simplest thing to build and operate: request in, response out. Streaming needs a transport that can push bytes to the client as they arrive, which rules out anything that buffers the whole response first. Asynchronous delivery needs somewhere to keep the work, somewhere to put the result, and a way to tell the caller it is ready. Each step up in interactivity buys a better experience and costs you moving parts to run and debug.

The fourth is whether the caller can wait at all, and in what form the answer is collected. A user staring at a form can wait twenty seconds if something reassures them, but not five minutes. A pipeline enriching a warehouse can wait hours and simply wants the cheapest correct throughput. Once the caller can genuinely disconnect and come back, what remains is how they learn the result is done: a push notification, an event, or a poll. That decoupling is what lets a job outlive any single request.

Cost shape follows from all of this. Real-time inference is billed per call at full rate and you pay for the latency you hold. Bedrock batch inference trades immediacy for a lower per-token price on bulk work, which is exactly the trade the nightly job wants and exactly the wrong trade for chat.

What we’ll filter on

1. Interactivity: is a human watching and waiting, or is the work happening in the background?
2. Time to first token versus time to completion: does perceived speed depend on the answer starting, or only on it finishing?
3. Output length against timeout limits: will a single completion brush the gateway or Lambda ceilings?
4. Transport complexity: how much delivery machinery are we willing to build and operate?
5. Can the caller disconnect: is total wait bounded by a held connection, or can the result be collected later?
6. Cost shape: full-rate real-time throughput, or discounted bulk with relaxed latency?

The delivery landscape

Synchronous request-response. Call InvokeModel or the Converse API, block until the full completion comes back, return it to the caller. This is the simplest pattern to build, test, and reason about, and for short outputs behind a reasonable timeout it is the right default. The failure mode is exactly the summariser’s: the caller holds a connection open for the whole generation, so a long completion risks the API Gateway integration timeout at roughly thirty seconds, and an unusually long one can approach Lambda’s fifteen-minute cap. It also feels slow in chat even when it is not, because the user sees nothing until the last token is written.

Streaming. Call InvokeModelWithResponseStream or ConverseStream and the model returns tokens as they are generated rather than in one final block. The total generation time is unchanged, but time to first token drops sharply, so a chat reply starts scrolling almost immediately and the feature feels responsive. The catch is the transport: something between the model and the browser has to forward each chunk as it arrives instead of buffering the whole response. Standard API Gateway REST and HTTP integrations buffer, so they defeat the point. The streaming-capable options are Lambda response streaming through a function URL, an API Gateway WebSocket API pushing chunks over the socket, AWS AppSync GraphQL subscriptions, or an application that speaks server-sent events directly from a container or ALB target. More capability, more transport to run.

Asynchronous job. Accept the request, hand back an acknowledgement immediately, do the generation in the background, and deliver the result out of band. The caller never holds a connection open for the work, so length and timeouts stop being a constraint. For one-off long tasks like the summariser, an AWS Step Functions workflow or an SQS queue feeding a worker runs the model call off the request path and stores the summary in S3 or a database; the browser polls a status endpoint or gets a push over WebSockets or AppSync when it is ready. For bulk work like the nightly enrichment, Bedrock batch inference takes an S3 manifest of prompts, processes them offline, and writes results back to S3 at a lower per-token price, which suits forty thousand tickets far better than forty thousand real-time calls.

The three modes are not a ranking. Synchronous is the floor everything else is measured against, streaming is synchronous with the first-token experience fixed, and asynchronous is what you reach for once the caller can stop waiting.

Side by side

Read against the three features: the chat assistant wants streaming, the summariser wants an asynchronous job with a poll or push, and the nightly enrichment wants Bedrock batch inference. Only the shortest, snappiest synchronous calls should stay as they are.

The picks in depth

The chat assistant is the streaming case, and the fix is a change of both API and transport. Swap Converse for ConverseStream so tokens arrive as they are generated, then give them a path to the browser that does not buffer. Lambda response streaming through a function URL is the lightest option and forwards chunks as the model emits them; an API Gateway WebSocket API or AppSync subscriptions suit an app that already holds a socket for other live updates. The point to hold onto is that streaming does not make the model faster, it makes the wait visible: total generation time is the same, but time to first token collapses, and a reply that starts scrolling in a second reads as responsive even if it runs for fifteen. Do not try to stream through a standard REST or HTTP API Gateway integration, because it buffers the whole response and hands you a synchronous call by another name.

The summariser is the asynchronous case, and the tell is that it keeps hitting a timeout on a request a user triggered but does not need to hold a connection for. Accept the upload, return a job identifier straight away, and run the model call off the request path in a Step Functions workflow or an SQS-driven worker. Write the finished summary to S3 or a table, and let the browser either poll a status endpoint against that identifier or receive a push over WebSockets or AppSync when it lands. Now the twenty-to-forty-second generation happens with no connection held open, so the gateway timeout stops applying and the 504s disappear. The cost is the async machinery: a place to run the work, a place to store the result, and a way to signal completion. For a task that reliably exceeds a comfortable synchronous budget, that machinery is the price of not fighting the timeout.

The nightly enrichment is the batch case, and it is the clearest waste in its current shape. Forty thousand real-time calls pay full rate and serialise behind Lambda limits for no benefit, because nothing is watching and nothing needs the answers before morning. Bedrock batch inference takes a manifest of prompts from S3, processes them offline as a managed job, and writes results back to S3 at a lower per-token price than on-demand invocation. It trades immediacy, which the job does not need, for throughput and cost, which it does. The fragile hand-rolled chunking goes away because the batch service owns the fan-out.

Across all three, notice what did not change: the model and the prompt can be identical in every case. The delivery mode is a decision about the transport and the wait, made from the interactivity and length of the workload, and it is largely separable from the prompt engineering that shapes the answer itself. That separation is what lets one application serve a live chat, a triggered long job, and an overnight batch without pretending they are the same request.

A worked example: the summariser, from timeout to job

Today the summariser is a synchronous call behind API Gateway and Lambda. The browser posts the contract and waits; Lambda calls Converse and blocks for the whole generation; the response goes back through the gateway. On a long contract the completion runs past the roughly thirty-second integration timeout, the gateway returns a 504, and the user sees a failure even though the model was still working.

Reshaped as an asynchronous job, the request path does almost nothing. The endpoint validates the upload, drops a message on a queue or starts a Step Functions execution, and returns 202 Accepted with a job identifier in well under a second. A worker picks up the message, calls Converse, and writes the summary to S3 keyed by that identifier. The browser then polls a status endpoint:

POST /summaries        -> 202 { "job_id": "sum_9f21", "status": "processing" }
GET  /summaries/sum_9f21 -> 200 { "status": "processing" }
GET  /summaries/sum_9f21 -> 200 { "status": "done", "url": "s3://.../sum_9f21.txt" }

or, if the app already holds a WebSocket, the worker pushes a “done” event with the same identifier and skips the polling entirely. Either way the twenty-to-forty-second generation no longer sits inside a single held request, so there is no connection to time out. The user experience is a progress indicator instead of a spinner that sometimes dies at thirty seconds, and the same reshape gives the operations team a retryable, observable job instead of an opaque blocking call. The model invocation in the middle is unchanged; only the delivery around it moved.

What’s worth remembering

1. Pick the delivery mode from the workload’s interactivity and output length, not from whichever pattern the first feature happened to use.
2. For interactive chat, perceived speed is time to first token, so streaming with ConverseStream feels fast even though total generation time is unchanged.
3. Streaming needs a transport that forwards chunks as they arrive; Lambda response streaming, WebSocket APIs, AppSync subscriptions, or server-sent events work, standard REST and HTTP API Gateway integrations buffer and defeat it.
4. Synchronous InvokeModel or Converse is the simplest pattern and the right default for short answers, but it holds a connection open for the whole completion.
5. A long completion behind API Gateway risks the roughly thirty-second integration timeout, and an extreme one approaches Lambda’s fifteen-minute cap; that is a signal to go asynchronous.
6. An asynchronous job lets the caller disconnect: acknowledge immediately, generate in the background with Step Functions or SQS, store the result, and notify by poll or push.
7. For high-volume offline work, Bedrock batch inference processes an S3 manifest of prompts at a lower per-token price than real-time calls, which suits bulk enrichment far better than looping synchronous invocations.
8. The three modes are not a quality ranking; asynchronous is what you reach for once no one is waiting on a held connection, not an upgrade over synchronous.
9. The delivery decision is largely separable from the prompt: the same model and prompt can serve a stream, a background job, and a batch.
10. When a feature keeps timing out on a request the user triggered but does not need to hold open, the fix is to move the work off the request path, not to raise the timeout.