The situation
The assistant answers customer-support questions over a knowledge base. It’s a RAGA pattern where you retrieve relevant documents at query time and stuff them into the prompt so the model can ground its answer on them. app on Bedrock: a subscriber’s message goes in, relevant docs get retrieved, a PromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot. gets assembled, and a completion comes back. It has been in production for four months and the support team likes it.
Now there’s a compliance review, and the reviewers arrive with a checklist. Who invoked this model, and when? What exactly did the model receive as input, and what did it return? Which model and which version produced each answer? Who approved putting this into production, and what was it approved to do? And can you prove no customer PII ended up somewhere it shouldn’t, and that the logs themselves haven’t been quietly edited since?
The app has none of this. It logs application errors and a request count to CloudWatch, and that’s it. Nothing records the prompt-and-completion pairs, nothing records who changed the guardrail configuration last month, and there’s no document anywhere stating what the assistant is allowed to do. Getting this wrong means a failed audit and, depending on the sector, regulatory exposure. Getting it right means standing up an evidence trail that answers each question with an artifact instead of a shrug.
What actually matters
The trap is to think of this as one feature (“turn on logging”) when it’s really several concerns that live in different places and answer different questions.
The first is the record of what the model actually did: every invocation, with the full prompt and the full completion, stored durably. This is the transaction log of the AI itself. Without it, “show me what the assistant told this customer on Tuesday” has no answer.
The second is the record of who changed the system around the model. Someone enabled logging; someone configured a GuardrailA filter or rule applied to an LLM’s inputs or outputs to keep it inside safe, legal, or on-brand behaviour. ; someone granted model access. Those are control-plane changes, and an auditor treats “who turned off the PII filter, and when” as seriously as any completion. Application logs won’t have it because these actions happen through the AWS API, not through the app.
The third is documentation of the solution itself. What is this assistant intended to do? What’s its risk rating? What are its known limitations, what data trained or grounds it, what did evaluation show? An auditor reviewing an AI system expects a governance document, not a code walk-through.
The fourth is the bridge from raw evidence to a compliance framework. Auditors don’t want a pile of logs; they want controls mapped to a standard (accuracy, privacy, fairness, resilience, responsible use, governance) with evidence attached to each and a report they can read. Collecting evidence is one job; organising it against a framework is another.
The fifth is the integrity of the evidence: retention long enough to satisfy the standard, encryption so the logs don’t become their own leak, tamper-evidence so nobody can claim the record was doctored, and tight access control so only the right people can read prompt-and-completion pairs that may themselves contain PII.
And a sixth, quieter one: the data-handling question the vendor answers, not you. Bedrock does not use your prompts or completions to train the base models, and your data stays in the region you call. That’s a property of the platform you cite, not a control you build, but the auditor will still ask.
What we’ll filter on
- Invocation payload capture, does it record the full prompt and completion of each call?
- Control-plane capture, does it record who changed configuration and access, through the AWS API?
- Framework mapping and report, does it map evidence to a compliance standard and produce an assessment?
- Model documentation, does it produce a governance artifact describing intended use, risk, and evaluation?
- Tamper-evidence and retention, are the records immutable, encrypted, and kept long enough?
- Managed effort, how much of this is configuration versus code we own?
The audit-trail landscape
Bedrock model invocation logging. Off by default, enabled per region. Once on, Bedrock delivers the full request and response of each invocation (text, image, and EmbeddingA fixed-length vector of floats that represents a piece of text (or image, or other thing) in a space where similar meanings sit close together. data) plus metadata to an S3 bucket, CloudWatch Logs, or both. This is the “what was asked and what was answered” record, the transaction log of the model. It captures payloads; it says nothing about who changed configuration.
AWS CloudTrail. Records Bedrock control-plane management events: PutModelInvocationLoggingConfiguration, guardrail create and update calls, model-access changes. It can also record InvokeModel as a data event, which captures that the call happened (identity, timestamp, model) though not the payload itself; the payload comes from invocation logging. CloudTrail gives you tamper-evidence through log-file validation, and delivering a multi-region trail to an S3 bucket with Object Lock makes the record immutable. This is the “who did what, and when” layer.
AWS Audit Manager. Ships a prebuilt “AWS generative AI best practices” framework that maps controls (accuracy, fairness, privacy, resilience, responsible use, governance, and more) to evidence, and auto-collects that evidence from CloudTrail and Config. It turns raw logs into an assessment report structured the way an auditor reads. This is the layer that produces the deliverable.
Amazon SageMaker Model Cards. Document the deployed model or solution: intended use, risk rating, training-data provenance, evaluation results, and caveats. This is the governance artifact that answers “what is this approved for, and what are its limits.”
AWS Config. Tracks whether the controls stay on over time. Managed and custom rules, gathered into a conformance pack, answer “is invocation logging still enabled, is the log bucket still encrypted, is the trail still running” continuously, not just on the day someone checked. This proves the posture holds, rather than that it held once.
Bedrock Guardrails plus CloudWatch metrics. The enforcement half: guardrails apply the PII and topic policy, and CloudWatch surfaces the operational metrics and alarms. Worth naming here because a redaction policy at the guardrail can strip PII before it ever reaches the logs, but the enforcement side is really its own subject, covered when you’re configuring guardrails for PII, topics, and grounding.
Side by side
| Option | Invocation payload | Control-plane | Framework mapping | Model doc | Tamper-evidence & retention | Managed effort |
|---|---|---|---|---|---|---|
| Invocation logging | ✓ full prompt + completion | ✗ | ✗ | ✗ | via S3 (KMS, lifecycle) | Low, per-region toggle |
| CloudTrail | partial (call, not payload) | ✓ | ✗ | ✗ | ✓ log-file validation + Object Lock | Low-medium |
| Audit Manager | ✗ (consumes evidence) | ✗ (consumes evidence) | ✓ GenAI framework + report | ✗ | inherits sources | Medium, framework setup |
| Model Cards | ✗ | ✗ | ✗ | ✓ intended use, risk, eval | versioned in SageMaker | Low, authoring effort |
| AWS Config | ✗ | ✗ (config state, not calls) | partial (conformance pack) | ✗ | ✓ config history | Medium, rules |
| Guardrails + CW | ✗ (can redact before logging) | ✗ | ✗ | ✗ | ✗ | Low-medium |
No row does the whole job. Invocation logging owns the payloads, CloudTrail owns the actions, Audit Manager turns both into a report, Model Cards documents the solution, and Config proves the controls stay on. The audit-ready answer is the stack, not any single service.
The evidence stack, illustrated
The pick in depth
The audit-ready shape is a stack, and each piece answers a distinct question. Invocation logging delivers the payload record. A multi-region CloudTrail with log-file validation, landing in an S3 bucket under Object Lock, gives the immutable who-and-when for every call and configuration change. Audit Manager’s generative-AI framework maps that evidence to controls and produces the report. A Model Card documents the solution. Config rules prove the whole arrangement stays switched on. Guardrails, if a PII redaction policy is applied, keep sensitive data out of the payload logs in the first place, which turns the “prove no PII leaked” question from a hope into a control.
The gotchas are where audits actually fail. Invocation logging is off by default and configured per region, so a Bedrock call in us-east-1 and another in eu-west-1 need logging enabled in each; a single toggle doesn’t cover the account. Cross-region inference makes this sharper, because a request can execute in a region other than the one you called, and if logging isn’t enabled there, those invocations vanish from the record. Enable it everywhere the model might run.
The logs themselves are a liability. Prompt-and-completion pairs can contain exactly the PII the audit is worried about, so encrypt them with a customer-managed KMS key, lock the S3 bucket and CloudWatch log group down to a named set of principals, and consider a guardrail redaction policy that strips PII before the completion is logged. An audit trail that leaks is worse than none.
Where the logs land is a real trade-off. CloudWatch Logs gives you search and alarms and fast lookup, which suits “find every prompt from last Tuesday” and “alarm if a completion trips a filter.” S3 gives you cheap long-horizon retention with lifecycle policies, and Athena over the bucket answers analytical questions across months. Most audit-ready setups send to both: CloudWatch for the operational window, S3 for the retention horizon. And because invocation logging, CloudTrail, and their buckets all stay in their configured region, data residency is a property you can state plainly, which is often the point of the review.
A worked example: five questions, five artifacts
The reviewer sits down and asks five concrete things. Each one maps to exactly one part of the stack.
“Show me every prompt this assistant received last Tuesday.” Model invocation logging. The full request-and-response records for that day are in CloudWatch Logs (queried by time range) and in S3 (queried with Athena for anything older than the CloudWatch retention window).
“Who turned off the PII filter, and when?” CloudTrail. The guardrail-update API call is a management event with the caller identity, the timestamp, and the before-and-after, sitting in the validated multi-region trail. Config corroborates it by showing the exact moment the resource’s compliance state changed.
“What is this model approved for?” The SageMaker Model Card. Intended use, risk rating, known limitations, and the evaluation results that supported sign-off, in one versioned document with the approver on it.
“Prove these logs weren’t altered.” CloudTrail log-file validation confirms the trail’s own integrity, and S3 Object Lock on the invocation-log and trail buckets means the objects couldn’t have been overwritten or deleted inside the retention period, whatever anyone’s credentials allowed.
“Does the vendor train on our data?” A platform property, not a log: Bedrock does not use your prompts or completions to train the base foundation models, and your data stays in the region you invoked. You cite it; you don’t have to build evidence for it.
Then the reviewer wants the whole thing as one document, and that’s Audit Manager: an assessment built on the generative-AI best-practices framework, with the CloudTrail and Config evidence already attached to each control, exported as the report they take away. The point of the earlier work on making a chatbot audit-ready with guardrails and watermarks was the enforcement side; this is the evidence side, and the two meet in that report.
What’s worth remembering
- Audit-readiness is several concerns, not one toggle: payload record, control-plane record, documentation, framework mapping, and evidence integrity each live in a different service.
- Bedrock model invocation logging is off by default and configured per region, so enable it in every region the model runs, cross-region inference included, or those calls leave no trace.
- CloudTrail records who changed the configuration; invocation logging records what the model said. The audit wants both, and application logs give you neither.
- Log-file validation plus an Object-Lock S3 bucket is how you answer “prove the record wasn’t altered” instead of asking to be believed.
- Audit Manager’s generative-AI framework turns a pile of evidence into a control-mapped assessment report, which is the artifact an auditor actually reads.
- A SageMaker Model Card is the governance document that answers “what is this approved for,” and it needs an approver and a risk rating, not just a description.
- The logs can contain the very PII under review, so encrypt with a customer-managed key, restrict access tightly, and redact at the guardrail before logging.
- AWS Config proves the controls stay on over time, which is a different claim from proving they were on the day someone checked.
- Bedrock not training on your prompts and completions, and keeping data in-region, is a platform property you cite rather than a control you build.
The app didn’t change. What changed is that every question the reviewer can ask now lands on a specific artifact, the artifacts are immutable and encrypted, and Audit Manager assembles them into the one document the review was really after.