Tracing an Agent's Decisions in Production

The situation

The subscriber help desk runs on a Bedrock Agent. A message comes in, the agent reasons about it, calls an action groupAction groupThe bundle of API operations a Bedrock agent is allowed to call, described by a schema so the model knows what each one does. to fetch the account, checks a knowledge base for the refund policy, calls another action group to compute a figure, and writes a reply. Most of the time it is right. This morning it told a subscriber they were owed nothing when they were owed a fortnight’s box, and the only thing anyone can see is that closing sentence.

The final answer is the one artefact that carries none of the information you need. A wrong number at the end could come from a bad tool argument, a stale knowledge-base document, a tool that returned the right value that the model then misread, or reasoning that skipped a step entirely. From the reply alone you cannot tell which, so you cannot fix it, and you cannot tell the subscriber what went wrong with any honesty.

An agent run is a chain of decisions: read the request, pick a tool, pass it arguments, read what comes back, reason to the next step, repeat until done. Debugging one, or proving to an auditor what it did, means reconstructing that chain for a single request. What matters is what to capture, where it lands, and how you pull one run back out of a day’s traffic.

What actually matters

The first thing worth naming is that tracing is a different job from evaluation and from guardrails, and the three are easy to conflate. Tracing answers “why did it do this,” reconstructing one run step by step. Evaluation answers “is it any good,” scoring outputs across many runs against references or a judge. Guardrails answer “stop it doing harm,” blocking or filtering content before it reaches anyone. You need all three, but a guardrail that blocked a response tells you nothing about the reasoning that led there, and an evaluation score tells you the agent is worse this week without telling you which step rotted. Only a trace reconstructs the decision path.

The second is the difference between the model’s reasoning and the plumbing around it. A Bedrock Agent trace records the agent’s own thinking: the orchestration steps, the rationale for each action, which action group or knowledge base it chose, the arguments it sent, and what came back. That is the layer that explains why. Underneath, each action group is usually a Lambda function calling real systems, and that layer explains what happened when the tool ran: the API it hit, the latency, the error it swallowed. A bad tool call and a bad piece of reasoning look identical from the final answer and completely different once both layers are captured.

The third is verbatim capture versus summary. The trace gives you the shape of the run, but for a real audit or a subtle bug you often need the exact prompt the model saw and the exact completion it produced, byte for byte. Model invocation logging is the feature that records those, the full request and response for each model call, delivered to your own logs and storage. A trace tells you the model called the billing tool; the invocation log tells you the precise text it generated to do so.

The fourth is production reach: getting to one run out of thousands, being alerted when something drifts, and keeping the record long enough to matter. A trace you can only see by re-invoking the agent with tracing switched on in the console is fine for a repro and useless for the incident that already happened. What you want in production is traces, metrics, and logs landing in a store you can query after the fact, with retention you control, tied together by an identifier so one subscriber’s bad morning is a single search rather than a manual hunt.

Underneath all of it: capture is a decision you make before the run, not after. The agent trace has to be requested, invocation logging has to be enabled, and distributed tracing has to be instrumented on the tools. None of it is retroactive. The cheapest time to turn it on is before you need it.

What we’ll filter on

1. Can you reconstruct a single run end to end: the reasoning, each tool call, its arguments, and what it returned?
2. Are the exact prompts and completions captured verbatim, not just summarised?
3. Does the picture join up across the agent and the Lambda tools underneath it, or does it stop at the agent boundary?
4. Is it queryable and alertable in production after the fact, or only visible while you re-run the agent?
5. How long is it retained, and can it stand up as an audit record of what happened?

The observability landscape

These are layers that stack, not rivals you choose between. A production setup usually runs several at once.

1. Bedrock Agent trace. When you invoke an agent you can ask it to return a trace, and the response then carries the step-by-step detail: the orchestration reasoning, the model’s rationale for each step, which action group or knowledge base it invoked, the parameters it passed, the observations it got back, and any pre-processing or post-processing steps. This is the one signal that explains the model’s decisions rather than just their effects. Its limit is that it describes the agent’s view of the run; it does not, on its own, see inside the Lambda that an action group called, and you have to request it per invocation.

2. Model invocation logging. A Bedrock account-level setting that captures the full input and output of model calls, the prompts and completions, and delivers them to CloudWatch Logs, Amazon S3, or both. This is your verbatim record: exactly what the model was asked and exactly what it said. It is the backbone of an audit trail and the thing you reach for when a trace summary is not precise enough to explain a subtle failure. It captures text, and can be configured for larger payloads like images to S3, but it is a firehose, so retention and access controls matter.

3. CloudWatch metrics and logs. Bedrock publishes operational metrics (invocation counts, token counts, latency, throttling, errors) to CloudWatch, and your action-group Lambdas write their own logs there. This is where you watch aggregate health and set alarms: a spike in errors, latency creeping up, token spend climbing. Metrics tell you something is wrong across the fleet; they do not, by themselves, reconstruct one run.

4. Distributed tracing across the tools. The action groups behind an agent are Lambda functions calling downstream systems. Instrumenting them with AWS X-Ray, or OpenTelemetry through the CloudWatch Application Signals collector, gives you a trace of the tool execution: the services it touched, the latency of each hop, and where an error was thrown. Correlate those tool traces with the agent trace and you can follow a request from the model’s decision into the API call that decision triggered. This is the layer that turns “the tool returned a bad value” into “the tool timed out against the billing service and returned a default.”

5. Agent observability for production monitoring. The newer tooling treats an agent run as traces, spans, and metrics in the observability sense: a parent span for the run, child spans for each reasoning turn, tool invocation, and knowledge-base lookup, with timing and status on each, landing in CloudWatch (its GenAI observability surface) or an OpenTelemetry-compatible backend. This is what makes a run queryable and alertable after the fact, so an incident is a search over stored spans rather than a re-run of the agent. It is the production form of the same information the invoke-time trace exposes.

6. Not tracing, but next to it. Bedrock model evaluation and RAG evaluation score quality across many runs; Bedrock Guardrails block or filter content in flight. Both produce useful signals and neither reconstructs a decision path. Keep them in the mental map so you do not reach for an evaluation job when what you actually need is one run’s trace.

Side by side

Signal	Reconstructs one run	Verbatim prompts and completions	Spans the Lambda tools	Queryable and alertable after the fact	Audit-grade retention
Bedrock Agent trace	✓ (the reasoning path)	✗ (summarised steps)	✗ (stops at the agent)	✗ (per-invocation by default)	✗
Model invocation logging	Partial (per model call)	✓	✗	✓ (query the logs or S3)	✓
CloudWatch metrics and logs	✗ (aggregate)	✗	Partial (tool logs)	✓	✓ (as configured)
Distributed tracing (X-Ray / OTel)	Partial (the tool side)	✗	✓	✓	✓ (as configured)
Agent observability spans	✓	✗ (references, not full text)	✓ (with instrumented tools)	✓	✓ (as configured)
Evaluation / Guardrails	✗	✗	✗	n/a	n/a

Reading it for this incident: the agent trace tells you which steps the model took and why, invocation logging gives you the exact text it generated, and tool tracing tells you whether the billing Lambda actually returned what the model acted on. No single row does the whole job; the debuggable, auditable setup is the trace for the reasoning, the invocation log for the verbatim record, and tool tracing for the plumbing, correlated by a run identifier.

An agent run as a trace

The picks in depth

Bedrock Agent trace, for the reasoning. Turn on the trace when you invoke the agent and the run comes back annotated with its own thinking: each orchestration step, the rationale the model gave for choosing an action, the action group or knowledge base it picked, the parameters it sent, and the observation it received, plus any pre-processing and post-processing. This is the signal that answers why, and it is the one you read first when an answer is wrong for no obvious reason. Its two limits shape how you use it. It describes the agent’s view, so it shows that the model called calcRefund with a particular weeks argument but not what happened inside the Lambda that served the call. And a trace attached to a live invocation is a repro tool, not an incident record; for production you want those same steps captured as stored spans (the agent-observability layer) so the run that already failed is still there to read.

Model invocation logging, for the verbatim record. Enable it once at the account level and every model call thereafter delivers its full request and response to CloudWatch Logs or S3. When the trace summary says the model “decided no refund was due” and you need to know exactly what it generated to reach that, the invocation log has the literal text. This is the backbone of an audit: it is precise, it is complete, and it lives in storage you control with retention you set. Treat it accordingly. It captures prompts and completions verbatim, which can include personal data, so it wants tight access controls, sensible retention, and encryption, and it is voluminous enough that you plan for its volume rather than discover it on the bill.

CloudWatch metrics and logs, for health and alarms. Bedrock’s published metrics (invocations, tokens, latency, throttles, errors) and your Lambda logs are where you watch the fleet and get told when something moves. You alarm on an error-rate spike or a latency climb and it points you at a window; you then pull the individual traces and invocation logs to see what actually happened in that window. Metrics start investigations; they do not finish them.

Distributed tracing across the tools, for the plumbing. Instrument the action-group Lambdas with X-Ray, or with OpenTelemetry via the CloudWatch Application Signals collector, and each tool execution becomes a trace of its own: the downstream services it called, the latency of each, and where an exception was thrown. Correlate those with the agent’s run and the two failure modes finally separate. “The model passed the wrong argument” shows up in the agent trace; “the tool returned a default because the billing service timed out” shows up in the tool trace. Propagating a shared identifier from the agent invocation into the tool calls is what lets you stitch one request together across both.

Agent observability spans, for production. The production shape of all this is the run as a span tree in your observability backend: a parent span for the invocation, children for each reasoning turn, tool call, and knowledge-base lookup, each with timing and status, landing in CloudWatch’s GenAI observability surface or an OpenTelemetry-compatible store. That is what makes “show me run 8c2f from this morning” a query rather than a re-run, and what lets you alert on a step failing or a tool-call rate climbing. It carries references and timings more than full payloads, so it pairs with invocation logging, which holds the verbatim text, rather than replacing it.

A worked example: the refund that was not

The subscriber wrote in; the agent replied that no refund was due; the subscriber was owed two weeks. The reply is the only thing the help desk agent could see, so the investigation starts by pulling the run.

The agent trace for the run lays out the chain. Orchestration turn one: fetch the subscription, getSubscription(id=8c2f), observation paused=true. Turn two: consult the refund policy knowledge base, two chunks returned about the refund window. Turn three: compute the refund, calcRefund(id=8c2f, weeks=?). Turn four: compose the reply. The trace shows the shape and already narrows the field: the model did reach the calculation step, so the failure is not a skipped step. The suspect is the weeks argument it passed.

The invocation log for that model call has the verbatim completion, and there it is: the model generated weeks=0, having read the pause as covering the current week only, when the subscriber had been paused for two delivery cycles. The exact text of the reasoning shows the misread. The tool did nothing wrong, it faithfully computed a zero-week refund as zero.

To be sure the tool was innocent, the tool trace for calcRefund, correlated by the run identifier, confirms it: the Lambda received weeks=0, called the billing service, which responded in 40ms with £0.00, no error, no timeout. The plumbing was healthy. The defect was upstream, in how the model turned the pause history into an argument.

That is a fix you can now make with confidence: the pause data the model reads is ambiguous about multi-cycle pauses, so you tighten what the account tool returns and add an instruction about counting cycles. Without the trace you would have been guessing between a tool bug, a stale policy document, and a reasoning error. With it, the wrong step is named, the verbatim proof is on file, and the tool is cleared. If an auditor asks later what the agent did for subscriber 8c2f on this date, the same three artefacts answer them.

What’s worth remembering

1. The final answer is the one artefact that hides the reason; debugging or auditing an agent means reconstructing the run behind it, not reading the reply.
2. An agent run is a chain of decisions, which tool, what arguments, what returned, how it reasoned onward, and you have to capture the chain to see any of it.
3. Tracing (why did it do this), evaluation (is it any good), and guardrails (stop it doing harm) are three different jobs; only a trace reconstructs a decision path.
4. The Bedrock Agent trace is the layer that explains the model’s reasoning: orchestration steps, chosen actions, arguments, and observations.
5. Model invocation logging is the verbatim record, the exact prompts and completions, delivered to your own logs or S3, and it is the backbone of an audit trail.
6. The agent trace stops at the agent; distributed tracing on the action-group Lambdas shows what the tools actually did, and correlating the two separates a bad argument from a bad tool.
7. CloudWatch metrics and logs are for fleet health and alarms; they start an investigation but do not reconstruct a single run.
8. Production monitoring wants the run captured as stored spans, queryable and alertable after the fact, not a trace you can only see by re-invoking the agent.
9. Capture is a before-the-run decision: the trace has to be requested, invocation logging enabled, and the tools instrumented; none of it is retroactive.
10. Verbatim logs carry the prompts and completions, so treat retention, access, and encryption as part of turning observability on, not an afterthought.

For a run whose whole shape you need to defend, from the model’s reasoning down into the tool it triggered, no single signal is enough. The trace explains the decisions, the invocation log proves them word for word, and the tool trace clears or convicts the plumbing, all stitched together by one identifier. That combination, closer to the way a supervisor over collaborators multiplies the number of runs you will one day have to explain, is what turns a wrong answer from a mystery into a fix.