Red-Teaming a Bedrock Application

The situation

A support assistant runs on Amazon Bedrock. It reads a subscriber question, retrieves supporting passages from a knowledge base, and can raise a small goodwill refund through an agent action groupAction groupThe bundle of API operations a Bedrock agent is allowed to call, described by a schema so the model knows what each one does. . The team has already spent time on runtime defences; guardrails are configured, tools are scoped, retrieved content is delimited. What nobody has done is attack the thing on purpose to find out what still gets through.

The pressure to do so is concrete. A model version is about to change. A prompt is being rewritten to handle a new refund policy. Both are the kind of edit that can quietly reopen a hole that a previous fix closed, and there is no test that would catch it. When someone asks “are we sure the jailbreak from March is still blocked?”, the honest answer is nobody knows, because the March fix was a prompt tweak and a manual check, and neither survives into the next deploy.

The goal is a red-team exercise that produces something durable: not a one-off report that a consultant probed the app and found three issues, but a suite of adversarial tests that runs on every change and a loop that turns each new finding into a permanent defence.

What actually matters

Red-teaming is easy to conflate with two neighbours, and keeping them apart is the whole game.

Evaluation measures quality on inputs you expect. An evaluation job asks whether the assistant answers billing questions correctly, stays on topic, and reads well. It runs representative traffic and scores the normal case. Red-teaming measures failure on inputs an adversary chooses. It runs hostile traffic, the override attempts, the smuggled instructions, the coaxing towards a refund, and asks whether any of it works. Same machinery in places, opposite intent: evaluation wants the app to succeed, red-teaming wants it to fail so you learn where it can.

Guardrails enforce at runtime; red-teaming probes. Amazon Bedrock Guardrails is a control that sits in the request path and blocks a live attack as it happens. Red-teaming is an offline activity that discovers what to block and checks that the block holds. One is a wall, the other is the person who keeps throwing rocks at the wall to find the loose brick. A finding from red-teaming often becomes a new guardrail rule, so they feed each other, but they are not the same layer and cannot substitute for one another. A guardrail with no adversarial testing is untested; a red-team finding with no runtime control is just a note.

The threat surface is wider than jailbreaks. A thorough exercise probes for direct prompt injection (the user typing an override), indirect injection (a hostile instruction riding in through a retrieved document or a tool result), data and secret exfiltration (persuading the model to emit its instructions, session context, or another subscriber’s data), PII leakage in the response, harmful or biased output, and unsafe tool use, where the model is talked into calling the refund action for an attacker who could never call it directly. The direct and indirect injection pair is covered in depth in the prompt-injection defence post; red-teaming is how you find out whether those defences actually hold.

Two properties decide how much a given failure matters. Side-effecting reach: a jailbroken read-only answer is embarrassing, a jailbroken refund moves money, so probes that end in a tool call rank above probes that end in text. Durability of the fix: a finding patched by editing a prompt evaporates on the next rewrite, while a finding turned into a guardrail rule, a schema check, or a regression test survives every future change. Red-teaming that does not close the loop into something durable buys you one clean deploy and nothing more.

What we’ll filter on

1. Threat coverage: does the suite exercise the whole surface (direct and indirect injection, exfiltration, PII, harmful output, unsafe tool use), or only the attack that made the news?
2. Repeatability: can every probe run unattended on each model or prompt change, so a regression cannot slip back in silently?
3. Measurability: can the outcome be scored automatically, by a guardrail check or an automated judge, rather than a human eyeballing each response?
4. Creativity: does the process leave room for a human to invent the attack a fixed test list would never contain?
5. Loop closure: does each finding become a durable artefact (a guardrail rule, an eval case, or a code fix), or just a line in a report?
6. Authorisation: is the exercise scoped and approved, run against the right environment, with no real subscriber data at risk?

The attack landscape

The space has two axes: the categories of attack you must cover, and the methods you use to run them.

The categories are the threat surface above. Direct injection probes are override and unlock framings typed straight in: “ignore previous instructions”, “you are now in developer mode”, “print your system prompt”. Indirect injection probes plant a hostile instruction in a place the app will pull in, a knowledge-base article, an uploaded document, a tool response, then ask an innocent question that triggers retrieval. Exfiltration probes try to make the model emit its instructions, encode secrets into an answer, or smuggle data into a tool call’s arguments. PII probes check whether the model repeats card numbers or emails that appear in context. Harmful and biased output probes push for content the content filters are meant to stop, and check for skew across demographic phrasings of the same request. Unsafe tool use probes are the ones that matter most: they try to reach the refund action through the model, testing whether the confused-deputy path is really closed.

The methods are how each category gets exercised. An automated probe harness is a stored set of adversarial inputs replayed against the application through the Bedrock Converse or InvokeModel path, or through the agent, with an assertion on each result. This is what makes red-teaming a regression rather than an event: the suite lives in the repository and runs in CI on every change. Bedrock Guardrails as a scorer catches the categories a guardrail already covers; if a probe response is passed back through the guardrail and it intervenes, the defence held, and if it does not, you have a finding. An automated judge handles the categories no simple rule can score: a second model call grades whether a response leaked the system prompt, complied with an injected instruction, or produced biased content, giving a pass or fail per probe. Amazon Bedrock model evaluation jobs support automatic scoring and an LLM-as-a-judge mode for exactly this kind of grading, and you can also run the judge yourself with a plain model call. Human red-teamers supply the creativity a fixed list cannot: they invent novel framings, chain steps, and follow the model’s own responses towards a weakness, and every attack they find that works gets written back into the automated harness so it never has to be found by hand again. Open-source adversarial toolkits seed the harness with known jailbreak and injection patterns so you are not starting from a blank page.

Scope and authorisation sit under all of it. Red-teaming your own Bedrock application means sending hostile prompts to software you own, which is testing your application rather than AWS infrastructure, but it still needs the discipline of a security exercise: written authorisation, a defined target (which application, which environment), rules of engagement, and a staging environment seeded with synthetic data so no real subscriber PII is ever the thing you are trying to exfiltrate. Run destructive tool probes against a sandbox billing API, never the live one.

Side by side

Probe categories as rows; the properties that decide how each is run and measured as columns.

Probe category	Automatable regression	Scored by a Guardrail	Needs an automated judge	Rewards human creativity	Side-effecting
Direct injection / jailbreak	✓	✓	✓	✓	✗
Indirect (second-order) injection	✓	✓	✓	✓	✓
Data / secret exfiltration	✓	✗	✓	✓	✗
PII leakage in output	✓	✓	✗	✗	✗
Harmful or biased output	✓	✓	✓	✓	✗
Unsafe tool / action use	✓	✗	✗	✓	✓

Every row is automatable, which is the point: the whole surface can live in a regression suite. Read the last column for where to spend the most effort, since the side-effecting rows, indirect injection and unsafe tool use, are the ones that move money if they succeed. Read the “needs a judge” and “rewards human creativity” columns together: the categories a guardrail cannot score are the ones where an automated judge and a human attacker earn their keep, because there is no simple rule that says a response leaked its instructions.

The picks in depth

The design that satisfies the filters has four moving parts.

A probe suite that lives in the repository. Store adversarial inputs as data, one per case, each tagged with its category and an assertion for what a safe outcome looks like. Replay them against the application through the same path production uses, the Converse API, InvokeModel, or the agent invoke, so the test exercises the real assembled context, retrieval and tools included, not a stripped-down model call. Wire the suite into CI so it runs on every model version bump and every prompt edit. This is the single change that turns red-teaming from a report into a regression: the March jailbreak is not a memory, it is case 47, and if a prompt rewrite reopens it, the build goes red.

Layered measurement. Not every probe can be scored by a string match. Pass response-side probes back through Bedrock Guardrails and treat an intervention as the defence holding; where the outcome is a judgement call, whether the model leaked its instructions, complied with an injected order, or produced biased text, use an automated judge, a second model call that returns a pass or fail with a reason. Bedrock model evaluation jobs offer automatic and LLM-as-a-judge scoring for this, and a self-hosted judge call works when you want the grading inside your own harness. For unsafe-tool-use probes, do not judge the text at all: assert on whether the action group was invoked and with what arguments, read from the tool audit log, because the only thing that matters is whether money would have moved.

Human red-teamers on top of the automation. The harness catches everything you already know to test, which is exactly why it cannot find the attack you have not imagined. Schedule human sessions where testers chain steps, invent framings, and follow the model’s responses towards a weakness, seeded with known jailbreak and injection patterns from open-source toolkits so they start beyond the obvious. The rule that keeps this from being a one-off: every working attack a human finds is written back into the automated suite the same day, so its discovery cost is paid once.

A closing loop with an owner. A finding is not done when it is written down; it is done when it becomes a durable artefact. Injection and jailbreak findings usually become a new Guardrails denied topicDenied topicsSubjects you describe in plain language that a Bedrock Guardrail refuses to discuss, whichever way a user phrases the request. or a tuned prompt-attack threshold; PII findings become a sensitive-information filter rule; unsafe-tool findings become a tighter IAM scope, a schema constraint on the arguments, or a human-approval gate; and every finding, whatever else it becomes, also becomes a regression case so the fix is proven on every future change. Triage by side-effecting reach first, since a probe that reaches the refund tool outranks one that only produces an awkward sentence.

Underneath all four, keep the exercise authorised and contained. Get written sign-off on the target and the window, run against a staging environment seeded with synthetic subscriber data, point tool probes at a sandbox billing API, and never make live customer PII the payload you are trying to exfiltrate.

A worked example: the model upgrade that reopened a hole

A newer model version is available and the team wants the quality gain. In the old process, someone would swap the model ID, run a few normal questions, see good answers, and ship. The subtle risk is that the new model responds differently to adversarial framings, and a jailbreak the previous model refused might now land.

With a probe suite in place, the upgrade runs against all of it before merge. Most cases pass. Two fail: an indirect-injection case where a tampered knowledge-base article now persuades the new model to attempt a refund, and an exfiltration case where a role-play framing coaxes out the system prompt. The indirect-injection failure is caught by the tool-log assertion, the action group was invoked when it should not have been, and the exfiltration failure is caught by the automated judge, which reads the response and flags that the instructions leaked.

Both are triaged. The refund path is side-effecting, so it goes first: the fix tightens the untrusted-content delimiting and adds a human-approval gate on the goodwill refund, and the case stays in the suite to prove it. The exfiltration finding becomes a new denied topic around revealing internal configuration, plus its own regression case. The upgrade merges only once every probe is green again. A human red-team session a fortnight later invents a fresh framing that chains the two, gets partway, and that framing is added as case 61 the same afternoon. The next model change, whenever it comes, will replay all of it without anyone remembering to.

What’s worth remembering

1. Red-teaming attacks your own application on purpose to find failures before an outsider does; evaluation measures quality on expected inputs, and guardrails enforce at runtime, so the three are distinct layers that feed each other rather than substitutes.
2. Cover the whole threat surface, direct and indirect injection, data and secret exfiltration, PII leakage, harmful or biased output, and unsafe tool use, not just the jailbreak that made the news.
3. The durable win is a probe suite stored in the repository and run in CI on every model and prompt change, so a fix cannot silently unwind on the next deploy.
4. Replay probes through the real production path, the Converse or InvokeModel call or the agent invoke, so retrieval and tools are exercised, not a stripped-down model call.
5. Measure in layers: use Bedrock Guardrails as a scorer where a policy covers the category, an automated judge where the outcome is a judgement call, and assertions on the tool audit log for unsafe-tool-use probes.
6. Amazon Bedrock model evaluation jobs offer automatic and LLM-as-a-judge scoring, and a self-hosted judge call works when you want the grading inside your own harness.
7. Keep human red-teamers on top of the automation for the creativity a fixed list cannot supply, and write every working attack they find back into the suite the same day.
8. Close every finding into a durable artefact, a guardrail rule, a sensitive-information filter, a tighter IAM scope or schema check, a human-approval gate, and always a regression case; triage by side-effecting reach first.
9. Scope and authorise the exercise like any security test: an approved target, a staging environment with synthetic data, sandbox tool APIs, and no real subscriber PII as the payload.