Choosing Where to Store Conversation State

The situation

A team is running a customer-facing chat assistant on Amazon Bedrock. Each user turn is a stateless model call: the runtime holds no memory between requests, so the application has to gather the running transcript, any session facts (the user’s plan, the current basket, what the assistant already asked), and hand the whole context back to the model on every turn. Right now that state lives in a process-local dictionary keyed by session id, which worked in the prototype and falls over the moment there is more than one container behind the load balancer, because the next turn lands on a different instance and the conversation has amnesia.

The traffic is spiky. A quiet afternoon is a few sessions; a promotion spikes it to thousands of concurrent conversations, some of them thirty or forty turns long, with users firing messages a couple of seconds apart. Conversations should survive a container restart mid-chat, but they do not need to live forever; after a day of inactivity the session is dead and keeping it is a privacy liability, because the transcript contains names, addresses, and order details. On top of that, product wants the assistant to remember a returning user across sessions (“last time you asked about the annual plan”), which is a different kind of memory from the in-flight transcript.

Nobody wants to run a database for the sake of it, and nobody wants to hand-build a memory layer that Bedrock might already offer. The question underneath all of it is the same: where should conversation state live, given how durable it has to be, how fast it has to be read and written, and how much of it the team wants to own.

What actually matters

Conversation state is not one thing, and the first useful move is to split it. There is short-term, in-flight state: the current transcript and the session’s working facts, read and written on every single turn, worthless once the conversation ends. And there is long-term memory: durable facts or summaries about a user that outlive the session and get retrieved when they return. The two have almost opposite storage profiles, and trying to serve both from one store is where designs go wrong.

For the in-flight state, the dividing property is durability against latency. Every turn does a read-modify-write of the session: fetch the context, append the new turn, hand it to the model, write it back. If that round trip to the store adds tens of milliseconds it is invisible next to the model’s own latency; if the conversation is high-turn-rate and the store is slow or contended, it starts to show. But durability pulls the other way. An in-memory store is the fastest option and the least safe, because a node failure can take the live conversation with it. The honest question is how bad it is to lose a conversation mid-flight: for a casual chat, a dropped session is an annoyance; for a booking or a support case with a transaction attached, it is a real failure, and that pushes toward a durable store even at some latency cost.

Scale and cost shape decides the second axis. The load is spiky and unpredictable, so a store that scales with the traffic and bills for what you use fits better than one you provision for peak and pay for at trough. A durable key-value store that autoscales and charges per request suits bursty session traffic; an in-memory cluster you size to a node count is priced on the cluster, not the calls, which is the right shape when the turn rate is relentlessly high and the wrong shape when it is mostly idle with occasional spikes.

Expiry and privacy are the same concern from two directions. The state is transient by nature and sensitive by content, so it should expire on its own rather than relying on a cleanup job that might not run, and it should be encrypted and access-controlled the whole time it exists. A store with a built-in time-to-live that deletes the session automatically after a period of inactivity does the privacy work and the housekeeping work in one setting; encryption at rest and in transit, plus tight access policies, are non-negotiable because the transcript is personal data.

The last axis is build versus buy. Everything above assumes the team assembles the memory layer: pick a store, key it by session, manage the TTL, and write the read-modify-write loop. The managed alternative is Bedrock Agents memory, where the agent runtime retains short-term conversation context within a session and can persist long-term memory (a running summary of the user across sessions) without the team standing up a store at all. That trades control and portability for a great deal less to build and operate. It is the right default when the assistant is built as a Bedrock agent and the memory needs fit what the service offers; it is the wrong fit when the state model is unusual, the assistant is not agent-shaped, or the data has to live in the team’s own stores for governance reasons.

And the cross-cutting one: long-term memory is a retrieval problem, not a session problem. Durable facts and summaries about a user are stored to be searched later, sometimes by meaning rather than by key, which is why long-term memory often lands in a separate durable store or a vector store, queried when the user comes back, rather than sitting in the same hot per-session table as the live transcript.

What we’ll filter on

1. State lifetime, in-flight transcript that dies with the session, or durable memory that outlives it?
2. Durability, how bad is losing a live conversation on a node failure?
3. Latency and turn rate, is this relentless high-frequency chat or occasional bursts?
4. Scale and cost shape, does the bill track spiky per-request traffic or a provisioned cluster?
5. Expiry and privacy, does the state self-delete on inactivity, and is it encrypted and governed as personal data?
6. Build versus buy, does managed agent memory fit, or does the team need to own the store?

The state-store landscape

1. Amazon DynamoDB. A fully managed, serverless key-value and document store. Key the table by session id, store the transcript and session facts as an item (or a small set of items), and read-modify-write it each turn. It is durable by default (replicated across Availability Zones), scales to spiky traffic with on-demand capacity that bills per request, and has a native time-to-live: set a TTL attribute and DynamoDB deletes the expired item automatically, which is exactly the “session dies after a day of inactivity” behaviour. Single-digit-millisecond reads and writes are fast enough to disappear behind the model call. This is the common default for durable per-session state, and DynamoDB Accelerator (DAX) sits in front as a read cache if a particular access pattern needs microsecond reads.

2. Amazon ElastiCache (Redis OSS / Valkey). A managed in-memory data store, the classic choice for very low-latency session state. Reads and writes are microseconds because the data lives in RAM, which suits high-turn-rate chat where the read-modify-write happens many times a second. Redis and Valkey have native key expiry, so per-session TTL is built in. The catch is durability: ElastiCache is a cache first, so a node or cluster failure can lose data, and even with replication it is not designed as a system of record. It shines as a hot layer for live conversation state where the odd lost session is tolerable, or in front of a durable store.

3. Amazon MemoryDB. A Redis- and Valkey-compatible, in-memory database that adds durability the cache does not have: writes are persisted to a multi-Availability-Zone transaction log, so it delivers in-memory read latency while surviving node failure as a system of record. This is the answer when the conversation is both high-turn-rate and cannot afford to be lost, giving microsecond reads and single-digit-millisecond durable writes with the same Redis/Valkey API (and the same native key expiry) as ElastiCache. It costs more than a plain cache, which is the price of the durability.

4. Amazon Bedrock Agents memory. The managed option: build the assistant as a Bedrock agent and let the service handle memory. It retains short-term context within a session automatically, and it can persist long-term memory, a summary of the user carried across sessions, so the agent recalls a returning user without the team running any store. You configure retention rather than operate infrastructure. The trade is control and portability: the memory model is what the service offers, and the state lives inside the managed agent rather than in your own tables.

5. A separate durable or vector store for long-term memory. Whatever holds the live transcript, the durable cross-session memory (facts, preferences, running summaries) is usually kept apart, because it is read on return rather than on every turn and is often searched by meaning. That can be a DynamoDB table of per-user facts, or a vector store (for example an OpenSearch Serverless vector collection, or a vector-enabled Aurora PostgreSQL) when retrieval is semantic. Keeping it separate stops the cold, occasionally-read memory from crowding the hot per-session path.

Side by side

Reading the table against the scenario: the live transcript wants a durable per-session store with automatic expiry, which is DynamoDB unless the turn rate is high enough and the loss-cost harsh enough to justify MemoryDB; the “remember me next time” feature is long-term memory, which belongs in a separate store or in Bedrock Agents memory; and the whole thing collapses into far less code if the assistant is built as a Bedrock agent and the managed memory fits.

The picks in depth

DynamoDB is the default for the in-flight transcript, and it is the default for good reasons that line up with the scenario one for one. The traffic is spiky, so on-demand capacity that bills per request beats a cluster sized for peak. The conversation must survive a container restart, so a store that is durable and multi-AZ by default beats an in-memory cache that can lose the session with a node. The session should die after a day of inactivity for privacy, so a native TTL attribute that deletes the item automatically beats a cleanup cron. And the latency, single-digit milliseconds, is invisible next to the model call, so the durability costs nothing perceptible. Key the table by session id, keep the item small (trim or summarise very long transcripts rather than letting one item grow without bound), turn on encryption at rest, and set the TTL. If one read path turns out to be genuinely latency-critical, DAX caches in front without changing the durable store underneath.

MemoryDB is the pick when the in-flight state is both high-turn-rate and unloseable, and the distinction from ElastiCache is the whole point. A plain ElastiCache cluster gives you the microsecond latency but is a cache, so a node failure can drop the live conversation; for a casual assistant that is a fine trade and the cheapest fast option. MemoryDB keeps the microsecond reads and adds a durable multi-AZ transaction log, so a failover does not lose the session. Reach for it when losing a mid-flight conversation is a real failure (a transaction attached, a support case in progress) and the turn rate is high enough that DynamoDB’s millisecond writes actually matter, which is a narrower case than people assume. If the turns are seconds apart, not milliseconds, DynamoDB’s latency is already invisible and the in-memory speed buys nothing.

Bedrock Agents memory is the build-versus-buy pick, and it is worth taking seriously before standing up any store at all. If the assistant is built as a Bedrock agent, the service retains short-term context within a session and can persist a long-term summary of the user across sessions, so both the in-flight transcript handling and the “remember me” feature come from configuration rather than code. That removes the store, the TTL management, and the read-modify-write loop from the team’s plate. The reasons to build it yourself instead are concrete: the assistant is not agent-shaped, the memory model needs to be something the service does not offer, or governance requires the personal data to live in the team’s own encrypted, access-controlled stores where their existing retention and audit tooling applies. When none of those bite, managed memory is less to build and less to get wrong.

Long-term memory is the pick that is really a separate decision. The returning-user feature is not the same problem as the live transcript, and stapling it onto the hot per-session table is a mistake: it is read on return, not every turn, and it is often searched by meaning (“what has this user cared about before”) rather than by exact key. So it lands in its own durable store, a per-user DynamoDB table for plain facts, or a vector store when the recall is semantic and you want to retrieve the most relevant past context rather than all of it. Whatever holds it, it is still personal data, so the same encryption, access control, and a retention policy apply; long-term does not mean forever.

Across all four, the privacy posture is not optional. Conversation state contains PII by default, so encrypt it at rest and in transit, scope access tightly with IAM, prefer stores whose TTL deletes stale sessions without a human in the loop, and set a real retention limit on the long-term memory too. The store you pick decides latency and cost; how you govern it decides whether a transcript full of names and addresses becomes a breach.

A worked example: routing the two kinds of state

The team splits the assistant’s memory in two and routes each half to the store that fits.

In-flight transcript. Turns arrive a couple of seconds apart, spiking to thousands of concurrent sessions during a promotion, and a conversation with a booking attached must survive a container restart. Seconds-apart turns mean DynamoDB’s single-digit-millisecond latency is already invisible, so the microsecond speed of an in-memory store buys nothing here, and the spiky load makes per-request billing the right cost shape. The pick is a DynamoDB table keyed by session id, with the transcript stored as an item, encryption at rest on, and a TTL attribute set to expire the session a day after the last turn:

Table: chat_sessions
  session_id   (partition key)
  transcript   (list of turns, trimmed to the last N + a summary)
  session_facts(map: plan, basket, pending_question)
  expires_at   (number, epoch seconds; TTL attribute)

Each turn: GetItem(session_id) -> append turn -> PutItem with
expires_at = now + 86400. DynamoDB deletes the item automatically
once expires_at passes with no further writes.

If load-testing later shows one hot read path needs microsecond latency, DAX goes in front without changing the durable table. If the turn rate turned out to be milliseconds-apart and losing a live booking were unacceptable, this is exactly the case that would justify MemoryDB instead; it is not this case.

Cross-session memory. The “last time you asked about the annual plan” feature is long-term memory, read only when the user returns and best matched by relevance. It goes in a separate store, a per-user vector collection holding short summaries of past conversations, queried on the user’s return to pull the most relevant prior context into the new session’s opening turn. It never touches the hot per-session table, it carries its own encryption and retention policy, and it is populated by summarising a session at the point the in-flight transcript expires.

Had the team built the assistant as a Bedrock agent from the start, both halves could have come from the service’s managed memory instead, short-term within the session and a long-term summary across sessions, with no table and no TTL to operate. They kept their own stores here because governance required the transcript’s PII to stay in tables their existing audit and retention tooling already covers. That is the build-versus-buy call made on a real constraint, not a reflex.

What’s worth remembering

1. Split conversation state before choosing a store: in-flight transcript that dies with the session, and long-term memory that outlives it, have almost opposite storage profiles.
2. DynamoDB is the common default for the durable per-session transcript, because it is durable by default, autoscales with spiky traffic, bills per request, and has a native TTL that expires dead sessions automatically.
3. Set the TTL attribute so stale sessions self-delete; it does the privacy housekeeping and the cleanup in one setting, with no cron to forget.
4. ElastiCache (Redis/Valkey) gives microsecond latency for high-turn-rate chat but is a cache, so a node failure can lose the live conversation; fine when a dropped session is only an annoyance.
5. MemoryDB gives the same in-memory speed with multi-AZ durability, for the narrower case where the chat is both high-turn-rate and cannot afford to lose a conversation.
6. If turns are seconds apart, DynamoDB’s millisecond latency is already invisible next to the model call, and the in-memory stores buy nothing worth their cost.
7. Bedrock Agents memory is the managed option: build the assistant as an agent and the service retains short-term context and can persist long-term memory, removing the store and the read-modify-write loop from your plate.
8. Build your own store when the assistant is not agent-shaped, the memory model is unusual, or governance needs the PII in your own encrypted, audited tables; buy the managed memory when none of those bite.
9. Long-term cross-session memory is a retrieval problem, so keep it in a separate durable or vector store, searched on the user’s return, not stapled to the hot per-session table.
10. Conversation state is PII by default: encrypt it at rest and in transit, scope access with IAM, expire it automatically, and give the long-term memory a real retention limit too.