The situation
A team runs a customer-support copilot on Amazon Bedrock, backed by a Claude model through the Converse API. A conversation is a list of messages; on every turn the whole list goes back to the model as input, because the model is stateless and remembers nothing between calls. For a five-turn exchange this is invisible. For the long sessions the copilot actually gets, a subscriber working through a delivery problem across forty or fifty turns, it is neither invisible nor cheap.
Two things go wrong as the transcript grows. The bill climbs, because input tokens are charged on every call and the transcript is resent in full each time, so turn fifty pays for turns one through forty-nine all over again. And eventually the accumulated history plus the next user message plus room for a reply exceeds the model’s context window, at which point the call fails or the oldest turns have to be dropped mid-conversation, and the copilot forgets the subscriber’s name and the ticket reference it was told twenty turns ago.
The team wants long conversations that stay coherent without the per-turn cost growing without bound. The problem underneath is which history actually has to travel on every call, and what to do with the history that does not.
What actually matters
The first thing to name is what a conversation costs. The model holds no state, so continuity is an illusion the application maintains by resending the transcript. Every message you keep in the list is input you pay for on this turn and every turn after it. Cost is not proportional to conversation length; if you replay everything it is closer to proportional to length squared, because each new turn re-sends a transcript that is one turn longer than the last. The context window is the hard ceiling on top of that spend: a fixed number of tokens the model can attend to at once, and when the transcript approaches it there is no next turn to pay for.
So the real design choice is not whether to trim history but how, and every option trades one of three things. The first is fidelity. The verbatim transcript is the highest-fidelity record there is; anything that shrinks it, a summary or a dropped turn, loses detail, and the detail it loses might be the one the subscriber needs three turns later. The second is what the compaction itself costs. Summarising is not free: it is an extra model call, with its own input and output tokens and its own latency, run periodically to rewrite old turns into something shorter. You are spending tokens now to save more tokens later, which pays off on long sessions and is pure overhead on short ones. The third is durability. Some facts have to outlive the window and even the session, the subscriber’s address, that they are lactose-intolerant, the reference number of an open complaint, and those want a home outside the transcript entirely.
That third point is the distinction the whole topic turns on. Short-term memory is the recent transcript, the last several turns kept verbatim so the model has the immediate thread. Long-term memory is everything durable, summaries of what came before and discrete facts pulled out and stored, that gets retrieved and reinjected when relevant rather than carried on every call. A managed conversational memory, like the memory feature on Bedrock Agents, is exactly this split productised: the agent keeps the live session transcript for the turn, and at the end of a session it summarises what happened and stores that summary under a memory identifier, so a later session starts with a recap instead of a blank slate. Knowing the boundary is the point; the strategies below are just different ways of drawing it.
What we’ll filter on
- History sensitivity, does answering the next turn need the exact wording of old turns, or just their gist?
- Cost shape, does the per-turn input grow without bound, stay flat, or grow only slowly?
- Fidelity loss, how much detail does the strategy discard, and can a dropped detail break an answer?
- Compaction overhead, does the strategy add its own model calls or storage, and does the session run long enough to earn them back?
- Durability, do some facts need to survive past the window or past the session into the next one?
The memory-compaction landscape
Replay everything. Keep the full message list and resend it every turn. Highest possible fidelity, zero compaction machinery, and it is the right default for short conversations. The failure is baked in: input tokens and latency grow with every turn, cost grows faster than that, and a long enough session walks straight into the context-window ceiling. This is the baseline the other strategies exist to fix, not a strategy to reach for on a copilot that gets long sessions.
Sliding window of recent turns. Keep only the last N turns (or the last N tokens) and drop anything older before each call. Per-turn cost stops growing; it plateaus at whatever the window holds, which makes spend predictable and keeps you clear of the ceiling. The cost is amnesia with no ceremony: once a turn falls out of the window it is gone, so the copilot forgets the ticket reference from turn three the moment turn three ages out. A sliding window is cheap, simple, and correct only when old turns genuinely stop mattering.
Running summarisation. Periodically replace the older turns with a model-written summary. Keep the recent turns verbatim for immediacy, and every so often, when the transcript crosses a token threshold, make a separate model call that condenses the older block into a paragraph or two, then carry that summary in place of the raw turns. Cost grows slowly instead of linearly, because the old history travels as a short summary rather than a long transcript, and nothing falls off a cliff the way it does with a bare window. What you pay is fidelity and an extra call: the summary is lossy by design, it can quietly drop or distort a detail, and each compaction spends its own tokens and adds latency. This is the workhorse for long single sessions.
Fact extraction to a store. Pull durable facts out of the conversation and write them to a store, then retrieve the relevant ones on later turns instead of carrying the whole history. When the subscriber says they are lactose-intolerant or gives an address, extract that as a discrete fact, persist it (a database, or a vector store when you want to fetch facts by semantic relevance), and inject only the facts that matter to the current turn. This is long-term memory proper: facts survive the window, survive the session, and can inform a conversation weeks later. The price is machinery, something has to decide what is a fact, write it, and retrieve it, and a retrieval step that can fetch the wrong facts or miss the right ones.
Managed conversational memory. Let the platform keep short-term and long-term memory for you. Bedrock Agents memory keeps the live session transcript during a session and, at session end, generates a summary it stores under a memory identifier for a configurable retention period, so a returning subscriber gets a session that opens with the relevant prior summaries already in context. You get the transcript-plus-summary split without building the summarisation loop or the store yourself, at the cost of the platform’s choices about what to keep and how long. It is running summarisation and cross-session recall as a managed feature rather than application code.
Most production copilots end up combining these rather than picking one: a sliding window for the immediate thread, running summarisation for the rest of the session, and fact extraction for the handful of things that must outlive it.
Side by side
| Strategy | Per-turn cost shape | Fidelity | Compaction overhead | Survives the window | Survives the session | Best for |
|---|---|---|---|---|---|---|
| Replay everything | Grows every turn | Full | None | ✗ | ✗ | Short conversations |
| Sliding window | Flat (capped) | Recent only | None | ✗ | ✗ | Threads where old turns stop mattering |
| Running summarisation | Grows slowly | Lossy on old turns | Extra model call | ✓ | ✗ (unless persisted) | Long single sessions |
| Fact extraction to a store | Flat + retrieval | Exact on stored facts | Extraction + store + retrieval | ✓ | ✓ | Durable facts across sessions |
| Managed memory (Bedrock Agents) | Managed | Summary-level | Handled by the platform | ✓ | ✓ | Cross-session recall without building it |
The columns that separate them are cost shape and durability. Replay is the only one whose cost grows every turn, and the only reason to keep it is that below a certain length it is simplest and loses nothing. Everything else caps or slows the growth by giving up some fidelity, and the two that reach across sessions do it by moving durable content out of the transcript entirely.
The picks in depth
For the support copilot, the answer is not one strategy but a layered one, and the layers map onto how long different pieces of information need to matter.
Keep a sliding window for the immediate thread. The last several turns carry the live back-and-forth, the thing the subscriber said two messages ago and the clarification you asked for, and those want to travel verbatim because their exact wording is what makes the next reply coherent. Size the window in tokens rather than turns, because a turn that pastes in an error log is worth ten short ones, and a token budget keeps the input predictable regardless of how chatty any single turn gets.
Add running summarisation for the rest of the session. When the transcript crosses a threshold, make a separate call that condenses everything older than the window into a short running summary, and carry that summary in place of the raw turns from then on. This is where fidelity is deliberately traded for room, so the summary prompt matters: instruct it to preserve concrete commitments, numbers, references, and unresolved questions, and to compress pleasantries and resolved detail. The overhead is real, an extra call with its own latency and tokens, which is why you summarise on a threshold rather than every turn, and why on a copilot that mostly gets short sessions you might not summarise at all.
Extract the durable facts to a store. A handful of things must outlive both the window and the session: the subscriber’s identity, delivery address, dietary constraints, the reference of an open complaint. Pull those out as discrete facts and persist them, and on later turns retrieve only the facts relevant to the current message rather than carrying all of them. A vector store earns its place here when you want to fetch facts by semantic relevance rather than by exact key, which is the same retrieval machinery behind a Bedrock RAG index, pointed at conversation-derived facts instead of documents. This is the layer that makes a returning subscriber feel known.
If you would rather not build the summarisation loop and the store yourself, the managed path is Bedrock Agents memory, which keeps the session transcript during a session and stores a generated summary under a memory identifier at session end, retained for a configurable window and reinjected when the subscriber returns. You trade control over exactly what is kept for not having to write and operate the compaction yourself. It is the same short-term-plus-long-term split, drawn by the platform.
Two mistakes are worth calling out. Reaching for summarisation on conversations that are never long enough to need it just adds a model call and latency for no saving; a plain sliding window, or even replaying everything, is cheaper and lossless below the length where compaction pays off. And leaning on a bare sliding window for a copilot that needs continuity means it will confidently forget the ticket reference the moment that turn ages out, because a window has no long-term memory at all; if facts must persist, they have to be summarised or extracted, not merely windowed.
A worked example: a fifty-turn delivery complaint
A subscriber opens a session about a missed delivery. Turn three, they give the ticket reference and their address. Turns four through forty are back-and-forth: what happened, what the substitution policy is, whether they get a credit. By turn forty-five the raw transcript is large, and replaying it every turn is both the biggest line on the bill and a slow crawl toward the context ceiling.
Under the layered approach, three things are happening at once. The sliding window holds roughly the last eight turns verbatim, so the model always has the immediate thread in full. Everything older has been folded, by a summarisation call that fired when the transcript first crossed the threshold and again later, into a short running summary: “Subscriber reports a missed delivery on the 28th, ticket GB-44821; agreed a credit for the missed box is being processed; substitution policy explained; subscriber still wants confirmation of the redelivery date.” And the two durable facts, ticket GB-44821 and the delivery address, were extracted to the store on the turn they were first mentioned, so they are retrievable exactly even if they never appear in the window or the summary again.
On turn forty-six, what the model receives is not fifty turns. It is the running summary, the ticket reference and address retrieved as facts, the last eight turns verbatim, and the new message. The input is a fraction of the full transcript, the cost per turn has stopped climbing, the window is nowhere near its ceiling, and the copilot still answers the redelivery question correctly because the reference it needs was preserved as a fact rather than left to age out of a window or blur inside a summary. When the subscriber comes back a week later about the same complaint, the stored summary and facts mean the new session opens already knowing the history, instead of starting cold.
What’s worth remembering
- The model is stateless; continuity is the application resending the transcript, so every message you keep is input you pay for on this turn and every turn after it.
- Replaying the whole transcript makes per-turn cost grow with the conversation, and cost overall grows faster than length, until a long enough session hits the context-window ceiling.
- A sliding window caps the per-turn cost by keeping only recent turns, and it forgets everything that falls out of the window with no ceremony.
- Running summarisation trades fidelity for room: it replaces old turns with a model-written summary, so cost grows slowly, at the price of a lossy summary and an extra model call.
- Fact extraction moves durable details into a store and retrieves them on demand, so they survive the window and the session; the cost is the machinery to extract, store, and retrieve.
- Short-term memory is the recent verbatim transcript; long-term memory is summaries and stored facts that are retrieved and reinjected rather than carried on every call.
- Size the recent window in tokens, not turns, so one log-pasting turn cannot blow the input budget and the per-turn cost stays predictable.
- Write the summariser to preserve concrete commitments, numbers, and open questions and to compress the resolved chatter, because that is exactly the detail a later turn is likely to need.
- Bedrock Agents memory is the managed version of the split: it holds the session transcript live and stores a generated summary under a memory identifier for cross-session recall, trading control for not building the loop yourself.
- Match the strategy to session length and durability: replay or a plain window for short threads, summarisation for long single sessions, and extracted facts for anything that must outlive the session.