The situation
A retailer is building a customer assistant on Amazon Bedrock. It has to do three quite different jobs from one chat surface. It answers policy questions (“how long do I have to return an item?”), which live in a few hundred pages of help-centre articles and terms documents that change a handful of times a year. It answers order questions (“where is order 55130, and when will it arrive?”), which live in the orders database and change minute to minute. And it answers account questions (“what is my current store-credit balance?”), which are specific to the signed-in customer and have to be exactly right.
The team’s first build put everything through one Amazon Bedrock Knowledge Base. The policy answers are good. The order answers are a disaster: the knowledge base was last synced overnight, so it happily tells a customer their parcel is “preparing to ship” when it was delivered two hours ago. The balance answers are worse, because there is no document anywhere that contains a live per-customer number, so the model either refuses or, alarmingly, invents a plausible figure.
The instinct is to sync the knowledge base more often. That is the wrong lever. Some of these answers should never have come from a retrieval index at all.
What actually matters
The first thing to name is that a retrieval index is a cache of documents, and every cache has a staleness bound. A Bedrock Knowledge Base answers from whatever was present at the last ingestion job; between syncs, the index is a photograph of the past. For a returns policy that changes twice a year, that bound is invisible and retrieval is close to perfect. For an order status that changes every few minutes, the same bound guarantees wrong answers, and no sync frequency short of “continuously, per request” closes it. Once you need per-request freshness, you are describing a tool call, not an index.
The second axis is the shape of the answer. Retrieval is built to return passages: spans of text that a document contains, ranked by relevance, handed to the model as grounding context. That is exactly right when the answer is explanatory (“here is what the policy says, in its own words”) and exactly wrong when the answer is a single precise value that no document contains as prose. A live order’s delivery estimate, an account balance, today’s price: these are computed or looked up, not written down in an article. A tool call, function calling against an API or a database, returns that value directly, and the model quotes it rather than paraphrasing a passage.
The third is who the data belongs to. Policy documents are shared: one corpus serves every customer, so indexing it once and retrieving many times is efficient and safe. A balance is per-user, and per-user data has no business sitting in a shared retrieval index, both because it is volatile and because it raises an access-control problem you do not want to solve inside a vector store. A tool call carries the signed-in customer’s identity to a system that already enforces who can see what, which keeps the authorisation where it belongs.
The fourth is latency and cost shape. Retrieval adds an embedding lookup and some context tokens; it is cheap and predictable, and it amortises the ingestion cost across many queries. A tool call adds a round-trip to a live system and, in an agentic flow, a second model turn to read the result, so it costs more per answer and its latency depends on the downstream service. That is a fair price for a fact that has to be current and exact, and a waste for a policy that a cached passage answers just as well.
None of this makes retrieval and tools rivals. The strong build uses both: retrieve the policy passage that explains the returns window, and in the same conversation call a tool for the live order status, then let the model compose one answer from the shared document and the per-user fact. The design question is not “which one”, it is “which one for this piece of the answer”.
What we’ll filter on
- Data volatility, does the underlying fact change by the year, or by the minute?
- Answer shape, is the answer a document passage the model paraphrases, or a precise current value it must quote exactly?
- Ownership, is the data shared across all users, or specific to one signed-in user?
- Source of truth, does the value live as prose in documents, or in an API or database that computes it on demand?
- Latency and cost tolerance, can the answer absorb a live round-trip, or does it need to come from a cheap cached lookup?
The grounding landscape
Retrieval-augmented generation (RAG). An ingestion job chunks and embeds a document corpus into a vector store; at query time the question is embedded, the nearest ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. are retrieved, and they are passed to the model as grounding context. On Bedrock this is a Knowledge Base, backed by a vector store such as Amazon OpenSearch Serverless, Aurora PostgreSQL with pgvector, or Amazon Neptune Analytics. Its strength is a large, slowly changing body of unstructured text: policies, manuals, help articles, contracts. Its hard limit is freshness, because the answer can only be as current as the last successful sync, so it is the wrong tool for a fact that moves faster than you ingest.
Live tool calling (function calling). The model is given a set of tools with typed schemas; when a question needs live data, it emits a call with arguments, the runtime executes it against an API or database, and the returned value comes back into the context for the model to answer from. On Bedrock this is the Converse API tool-use flow, and Amazon Bedrock Agents wrap it with orchestration and action groups (typically AWS Lambda functions that reach the live system). Its strength is exactly retrieval’s weakness: a volatile, precise, per-user value fetched at request time. Its cost is a live round-trip and, usually, an extra model turn to read the result.
Text-to-SQL. A specific and useful tool pattern for structured data: instead of hitting a hand-written API, the model translates the natural-language question into a SQL query, the query runs against the database, and the rows come back as the grounding value. Bedrock Knowledge Bases support this natively as structured-data retrieval, generating SQL against a connected store such as Amazon Redshift or the AWS Glue Data Catalog over Amazon Athena. It suits questions whose answer is a live aggregate or lookup over a relational source (“how many orders shipped today”, “this customer’s current balance”) where writing a bespoke API per question would be tedious. It is a tool call in spirit; the value is computed at request time, not read from a stale index.
Retrieval plus tools together. The two combine in one conversation. Retrieve the shared, slow-moving passage; call a tool for the volatile, per-user number; compose one answer. This is the normal shape for an assistant that spans reference material and live state, and Bedrock Agents can hold both a Knowledge Base and action groups so a single agent does both.
Side by side
| Property | RAG (Knowledge Base) | Live tool call | Text-to-SQL |
|---|---|---|---|
| Fast-moving facts | ✗ (bounded by last sync) | ✓ | ✓ |
| Slow-moving document corpus | ✓ | ✗ | ✗ |
| Answer is a text passage | ✓ | ✗ | ✗ |
| Answer is a precise current value | ✗ | ✓ | ✓ |
| Per-user, access-controlled data | ✗ | ✓ | ✓ |
| Structured/relational source | ✗ | ✓ (via API) | ✓ (native) |
| Per-answer latency and cost | Low, predictable | Higher, live round-trip | Higher, query round-trip |
| Freshness at answer time | Last ingestion | Request time | Request time |
Reading the table against the three jobs: the returns policy is a slow-moving shared document, so RAG; the order status is a fast-moving per-user value from a live system, so a tool call; the store-credit balance is a precise per-user number in the database, so a tool call or, if you would rather not maintain a bespoke API, text-to-SQL. None of the three is fixed by syncing the knowledge base more often.
The picks in depth
The returns policy stays on RAG, and the earlier build already had this part right. A few hundred pages of help articles and terms is precisely what a Bedrock Knowledge Base is for: a large, mostly static, unstructured corpus where the answer is a passage the model paraphrases. The staleness bound is real but invisible here, because a nightly or even weekly sync is faster than the documents change. The only discipline worth adding is treating the sync as a first-class step, so a policy edit triggers an ingestion job rather than waiting for the next scheduled run, which keeps the invisible bound invisible.
The order status moves to a live tool call, and this is the fix the team kept avoiding. Order state changes every few minutes and is specific to the signed-in customer, so it fails both the volatility test and the ownership test for an index. Declare a tool such as get_order_status(order_id), back it with a Lambda that reads the orders service, and let the model call it mid-conversation through the Converse API tool-use flow, or wrap the whole assistant in a Bedrock Agent whose action group is that Lambda. The value comes back at request time, the model quotes it, and “delivered two hours ago” is now something the assistant can actually say. The cost is a round-trip and an extra model turn, which is the correct price for a fact that has to be current.
The balance is the same shape as the order status, with one extra choice about how to reach the data. It is a precise per-user value that lives in a relational store, so it is a tool call; the question is whether you hand-write a get_balance(customer_id) API or let text-to-SQL generate the query. If you already expose a clean balance endpoint, call it. If the questions are open-ended over structured data (“how much did I spend last quarter”, “how many open orders do I have”), the native structured-data retrieval in Bedrock Knowledge Bases can translate the question to SQL against a connected Redshift or Athena source, which saves writing an API per question. Either way the value is computed at request time and carries the customer’s identity to a system that enforces access, so the authorisation stays out of the vector store where it never belonged.
The composed answer is where the two patterns meet. “Can I still return order 55130, and how long do I have?” needs the shared policy passage (retrieved) and the live per-user order date (a tool call) in the same turn. A Bedrock Agent that holds both a Knowledge Base and an action group can gather both and let the model write one grounded reply, quoting the current fact and paraphrasing the policy. The failure to avoid is forcing everything through one mechanism: pushing live state into the index gives stale answers, and pushing the policy through a bespoke tool throws away the cheap, shared, well-understood retrieval path for no gain.
A worked example: three questions, one assistant
Take three questions arriving at the same chat surface, and route each by volatility and answer shape.
“What is your returns window for electronics?” The fact changes maybe twice a year, the answer is a passage, and the corpus is shared. This is RAG: the Knowledge Base retrieves the relevant clause from the terms document, and the model paraphrases it. No live call, low latency, and the answer is as current as the last ingestion, which is plenty.
“Where is my order 55130?” The fact changes by the minute and belongs to one customer. Retrieval cannot help; there is no document that holds a live tracking state, and even if there were it would be stale by the time it was indexed. The model calls get_order_status(55130), the Lambda reads the orders service, and the reply quotes the returned status and estimate. Request-time freshness, per-user identity carried to the source.
“How much store credit do I have right now?” A precise per-user number in the database. The model either calls get_balance(customer_id) or, if the assistant leans on text-to-SQL, the structured-data retriever generates SELECT balance FROM store_credit WHERE customer_id = :id against the connected store and returns the row. Nothing about this answer wants a passage; it wants the exact current figure, quoted, and it must be right, which is why it never came from a document.
Now stack them. “Can I return 55130, and how long have I got?” pulls the policy passage from the Knowledge Base and the order’s delivery date from the tool in one turn, and the model composes: the window from the shared document, the clock started by the per-user fact. One assistant, three grounding routes, each chosen by how fast the data moves and what the answer actually is.
What’s worth remembering
- A model answers from a frozen snapshot, so any fact that has changed since training has to be grounded from outside; the choice is retrieval, a live tool, or both.
- RAG suits a large, slowly changing corpus of documents where the answer is a passage the model paraphrases, and its freshness is bounded by the last ingestion or sync.
- That staleness bound makes RAG the wrong tool for fast-moving facts; syncing more often narrows the gap but never closes it for data that changes by the minute.
- A live tool call, function calling against an API or database, fetches the value at request time, so it fits volatile, precise, per-user data that no document holds as prose.
- Route by volatility and answer shape: slow document passage to RAG, fast or precise current value to a tool, per-user data to a tool that carries the user’s identity.
- Per-user data does not belong in a shared retrieval index, both because it is volatile and because access control belongs in the source system, not the vector store.
- Text-to-SQL is a specific tool pattern for structured data: the model generates a query, the database runs it, and the row is the request-time value; Bedrock Knowledge Bases support this natively over Redshift or Athena.
- The two patterns combine in one answer: retrieve the shared policy passage, call a tool for the live number, and let the model compose a single grounded reply.
- On Bedrock, RAG is a Knowledge Base over a vector store, live tools are the Converse API tool-use flow or a Bedrock Agent action group, and one agent can hold both.
- Pay the extra latency and cost of a live call only where the answer must be current and exact; let cheap cached retrieval carry the slow, shared, explanatory questions.