Monitoring a Bedrock Pilot in Production

When keeping an AI pilot honest laid out what moves after launch (the model, the inputs, the guardrails) it stayed at the level of principle. This is the build underneath it: the AWS plumbing that catches each of those moving, for the two pilots Lodgewise runs out of its envisioning session, maintenance triage and tenant question-answering.

If you’ve set up SageMaker Model Monitor for a classical model, the shape will feel familiar, because we’re going to copy it.

The shape, borrowed from Model Monitor

Model Monitor watches a SageMaker endpoint with four moving parts. The endpoint captures every request and response to an S3 prefix. A baseline describes what good looked like at training time. A scheduled processing job reads the captured window, compares it to the baseline, and emits two outputs: CloudWatch metrics to alarm on, and a violation report in S3 to read later. That’s the whole pattern: capture, baseline, scheduled job, metrics plus report.

None of it attaches to Bedrock. Claude is a hosted foundation model behind an API, not an endpoint you own, so there’s no DataCaptureConfig to switch on and no built-in monitor to schedule. What there is, instead, is everything you need to assemble the same four parts yourself, and the assembling is mostly a day’s work. The contents differ from the tabular case (text instead of feature vectors, an LLM judge instead of Deequ statistics, human verdicts instead of a label join) but the skeleton is identical, and naming it that way keeps the build honest.

Capture: turn on model invocation logging

Bedrock’s equivalent of data capture is model invocation logging, a per-account, per-region setting that writes every request and response, with the guardrail trace, token counts, and latency, to S3 and CloudWatch Logs. It’s the same switch that backs a compliance audit trail; here we point it at monitoring instead.

import boto3

bedrock = boto3.client("bedrock", region_name="ap-southeast-2")

bedrock.put_model_invocation_logging_configuration(loggingConfig={
    "s3Config": {"bucketName": "lodgewise-bedrock-logs", "keyPrefix": "invocations"},
    "cloudWatchConfig": {"logGroupName": "/lodgewise/bedrock/invocations",
                         "roleArn": "arn:aws:iam::...:role/lodgewise-bedrock-logging"},
    "textDataDeliveryEnabled": True,
    "embeddingDataDeliveryEnabled": False,
})

That gets you the raw model traffic, but the raw log doesn’t know your application. It can’t tell you that a particular call was a handoff, or which tenancy it belonged to, or whether a coordinator later overturned it. So alongside the Bedrock log, write a small per-request envelope of your own, the fields the monitor will actually reason about:

import boto3, json, datetime

s3 = boto3.client("s3", region_name="ap-southeast-2")

def capture(pilot, result, meta):
    # One line per request, partitioned by pilot and day so the
    # scheduled job can read a single prefix for the window it wants.
    day = datetime.datetime.utcnow().strftime("%Y/%m/%d")
    record = {
        "ts": meta["ts"],
        "tenancy_hash": meta["tenancy_hash"],   # hashed, never the raw id
        "action": result["action"],             # answer | handoff
        "confidence": result.get("confidence"),
        "category": result.get("category"),     # triage only
        "citations": result.get("citations", []),
        "guardrail_intervened": result.get("guardrail_intervened", False),
        "model_id": meta["model_id"],
        "latency_ms": meta["latency_ms"],
        "tokens": meta["tokens"],
    }
    s3.put_object(
        Bucket="lodgewise-bedrock-logs",
        Key=f"envelopes/{pilot}/{day}/{meta['request_id']}.json",
        Body=json.dumps(record).encode(),
    )

Two disciplines matter here. Hash the tenancy_id before it lands anywhere; the monitor needs to count distinct tenancies and spot a leak, not to hold a directory of who asked what. And let the Bedrock guardrail mask PII before the text is logged, so a card number a tenant pasted into a question (one of the adversarial cases from the honesty post) never reaches the bucket in the clear. Capture is the one part you can’t backfill: if it isn’t on, there’s nothing to monitor, so it’s the first thing to verify and the first thing to alarm on when it goes quiet.

The baseline: a golden set and an input profile

Model Monitor splits its baseline in two, and so do we. Model-quality monitoring needs a labelled reference to score against; ours is the golden eval set that earned the launch, the one from the tenant Q&A build, with its faithful, cited-right, handed-off-when-it-should, and isolation-leak checks. Data-quality monitoring needs a description of the inputs the model was happy with; ours is a profile of the traffic that golden set stands in for.

{
  "version": "2026-07-31",
  "tenant_qa": {
    "category_mix": {"notice": 0.18, "repairs": 0.22, "pets": 0.08,
                     "bond": 0.15, "rent": 0.20, "other": 0.17},
    "handoff_rate": 0.12,
    "question_len_p50": 14, "question_len_p95": 41,
    "embedding_centroid_uri": "s3://lodgewise-bedrock-logs/baselines/qa-centroid-2026-07-31.npy"
  }
}

Both baselines live in S3 with a date on them, because both go stale. The golden set grows as incidents and human review add cases; the input profile is recomputed from a known-good window of real traffic whenever the team agrees the world has legitimately moved (a new category that’s here to stay, not a blip). Versioning them means every monitoring run records which baseline it judged against, so a shift in the numbers is never ambiguous about whether the model moved or the ruler did.

The scheduled job

EventBridge Scheduler is the cron. It fires a Lambda each night (a SageMaker Processing job if the work outgrows Lambda’s limits, which for replaying a few hundred eval cases it won’t for a while). The job does the two halves of the baseline in turn.

First, the output half: replay the golden set against the pinned model id and recompute the metrics, exactly the eval that gates a merge, now run on a schedule against an unchanged model so a drift in the provider’s hosting shows up even when nothing in your repo changed.

import boto3, json
cw = boto3.client("cloudwatch", region_name="ap-southeast-2")

def run_output_checks(pilot, golden_set):
    results = evaluate(golden_set)          # the eval harness from the build post
    emit(pilot, {
        "faithful": results["faithful_rate"],
        "cited_right": results["cited_right_rate"],
        "isolation_leaks": results["leaked"],          # must be zero
        "handoff_recall": results["handoff_right_rate"],
    })
    return results

Then the input half: read the day’s envelopes from S3 and measure them against the profile. Distribution shift in the category mix is a Population Stability Index; semantic shift is the distance from the question-embedding centroid; the operational signals (hand-off rate, unclear-bucket rate) come straight off the envelopes.

import math

def psi(expected, actual):
    # Population Stability Index: how far this window's mix has moved
    # from the baseline mix. >0.2 is a meaningful shift.
    total = 0.0
    for bucket, e in expected.items():
        a = max(actual.get(bucket, 0.0), 1e-4)
        total += (a - e) * math.log(a / max(e, 1e-4))
    return total

def run_input_checks(pilot, envelopes, baseline):
    mix = category_distribution(envelopes)
    emit(pilot, {
        "category_psi": psi(baseline["category_mix"], mix),
        "handoff_rate": mean(e["action"] == "handoff" for e in envelopes),
        "guardrail_rate": mean(e["guardrail_intervened"] for e in envelopes),
        "embedding_drift": centroid_distance(envelopes, baseline),
        "p95_latency_ms": percentile([e["latency_ms"] for e in envelopes], 95),
    })

emit is just put_metric_data into a lodgewise/ai namespace, dimensioned by pilot, the same namespace the live request path already writes to, so the scheduled signals and the per-request signals share one dashboard. And like Model Monitor, the job also writes the full numbers as a dated JSON report to S3: CloudWatch is the 3am version, the report is the post-mortem version, and the report is what you read when an alarm fires and you want to know which eval cases regressed.

Alarms with teeth

Metrics nobody alarms on are decoration. The thresholds are the ones from the honesty post, now wired to CloudWatch, and they fall into two tiers. The failures the whole design exists to prevent page: an isolation leak above zero, a missed emergency above zero. Everything else warns: faithfulness under 0.95, hand-off rate outside its band, category PSI over 0.2, p95 latency or spend-per-resolved-request creeping up.

cw.put_metric_alarm(
    AlarmName="lodgewise-qa-isolation-leak",
    Namespace="lodgewise/ai",
    MetricName="isolation_leaks",
    Dimensions=[{"Name": "pilot", "Value": "tenant_qa"}],
    Statistic="Maximum", Period=86400, EvaluationPeriods=1,
    ComparisonOperator="GreaterThanThreshold", Threshold=0.0,
    AlarmActions=["arn:aws:sns:ap-southeast-2:...:lodgewise-oncall"],
    TreatMissingData="breaching",      # no run is itself a problem
)

TreatMissingData="breaching" is the small detail that catches the worst silent failure: the monitoring job not running at all. A green dashboard because nothing reported is the one outcome worse than a red one. The cost signals get the same treatment, because a prompt that quietly grew three revisions or a retrieval returning more chunks than it needs regresses spend without anyone deciding to (the levers for pulling it back are in cutting a Bedrock bill without hurting quality).

Closing the loop: capture, review, baseline

This is the part Model Monitor can’t hand you, because ground truth for a tabular model arrives on its own (the loan defaulted or it didn’t) while ground truth for an answer is a human judgement. The captured envelopes are the raw material; a person turns them into labels.

Route three streams off the capture path into an SQS review queue: a small random sample of everything, every low-confidence or handed-off case, and every suggestion a coordinator overturned. The overturns are the richest, because they’re the cases the model got confidently wrong. A reviewer’s verdict appends to the golden set in S3 as a new permanent case, the baseline version ticks forward, and the next scheduled run measures against a tougher bar than the last. That’s the flywheel: capture feeds review, review hardens the baseline, the baseline raises the gate. A pilot without it stays exactly as good as launch day; a pilot with it compounds. Treat the eval set itself as versioned configuration, the same way you’d treat the prompts (see managing prompts across thirty services).

When the real Model Monitor is the answer

One honest caveat on the parallel. If a pilot ever runs on an open-weight model you host on a SageMaker endpoint, rather than a hosted model on Bedrock, you don’t rebuild any of this: you switch on data capture and schedule the actual Model Monitor, because now there is an endpoint to attach to. The hand-rolled version here exists precisely because Claude on Bedrock is an API, not an endpoint. What stays constant either way is the shape, capture to S3, a dated baseline, a scheduled job, CloudWatch metrics beside an S3 report, so a team that knows one can read the other.

A cadence, and someone who owns it

The machinery decays into unread dashboards without a rhythm and a name on it. The scheduled job runs nightly. Weekly, someone owns a look at the signals that move: hand-off rate, category PSI, guardrail interventions, latency, cost. Monthly, the baselines get refreshed, the golden set from the review queue and the input profile from a clean window of traffic. And the autonomy rung from the envisioning session climbs only when the numbers hold across a review period, never on a single good week. The pilots aren’t finished when they ship; they’re finished when there’s a loop watching the model, the inputs, and the guardrails all move, and an owner who’d notice the day it stopped.