Cheat Sheet: Evaluation, Cost, and Operations

A fast revision pass over evaluating, monitoring, costing, and operating a generative AI app on Bedrock. Skim the tables, drill the decision rules, watch the traps.

Levers at a glance

Decision rules

• If you need one number to compare model changes, then run a Bedrock model evaluation job against a fixed golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. .
• If quality is fuzzy and subjective, then use LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. for scale, and sample to human review for the final word.
• If the app retrieves documents, then use a RAG evaluation job and read faithfulness and correctness separately.
• If the answer is well-written but invents facts not in the context, then faithfulness is failing, not correctness.
• If the answer is grounded in the context but the context is wrong, then correctness is failing, not faithfulness.
• If you need labelled data or structured human ratings, then use SageMaker Ground Truth.
• If some live responses must be checked by a person, then route them through Amazon A2I.
• If you can’t see what the model was sent, then enable Bedrock model invocation logging to S3 or CloudWatch first.
• If an agent gives a wrong answer, then read its trace to find which tool call or retrieval went wrong before touching the prompt.
• If volume is steady and high, then buy Provisioned ThroughputProvisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. ; if it is spiky, stay on-demand.
• If the work is offline and can wait, then use batch inference for roughly half price.
• If a long system prompt or document repeats every call, then turn on prompt caching.
• If identical or near-identical prompts recur, then add response or semantic cachingSemantic cachingServing a cached answer when a new question is close enough in embedding space to one you’ve already answered. in front of the model.
• If prompts vary in difficulty, then use Intelligent Prompt Routing to send easy ones to a cheaper model.
• If the bill is a mystery, then apply cost allocation tags and application inference profilesInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. , and alert with AWS Budgets.
• If first-token feel matters, then stream the response and optimise time to first token, not just total time.

Traps

• Provisioned Throughput is billed by the hour whether or not you send traffic; idle reserved capacity still costs money.
• Batch inference is cheaper but not real-time; never reach for it when a user is waiting.
• Faithfulness and correctness are different axes. A grounded answer can still be wrong, and a correct answer can still be unfaithful to bad context; don’t collapse them into one score.
• LLM-as-a-judge is cheap and consistent but inherits the judge model’s blind spots; anchor it to human review, don’t treat it as ground truth.
• A golden set with only easy, in-scope cases hides regressions. Include hard cases and out-of-scope prompts the app should refuse.
• Model invocation logging is off by default; if you didn’t turn it on, there is nothing to investigate after an incident.
• Prompt caching helps only when a stable prefix repeats; a prompt that changes at the top every call caches nothing.
• Output tokens usually cost more than input tokens, so trimming a rambling response can beat trimming the prompt.
• Averages hide tail latency. Track p50 and p99; a good mean with an ugly p99 still fails real users.
• Cross-region inference profiles spread load and add resilience, but consider where data is allowed to be processed.
• Throttling is expected under load. Without retries and back-off, throttles surface to users as hard errors.
• Fewer, tighter retrieval chunks cut both cost and latency; stuffing the context window wastes tokens and can dilute the answer.

Say it in one line

1. The golden set is the ruler; it must carry hard and out-of-scope cases, not just happy paths.
2. Automatic metrics scale, LLM-as-a-judge scales with nuance, humans decide the hard calls.
3. Bedrock has evaluation jobs for both plain models and RAG pipelines.
4. Faithfulness is “supported by the context”; correctness is “right about the world”.
5. SageMaker Ground Truth builds labels and human evaluations; A2I inserts human review into production.
6. Turn on Bedrock model invocation logging to S3 or CloudWatch before you need it.
7. Agent trace and X-Ray tell you which step failed; CloudWatch metrics tell you how often.
8. You pay per input and output token, and the tier sets the rate; output usually costs more.
9. Provisioned Throughput is hourly and committed; batch is about half price for work that can wait.
10. Prompt caching reuses a stable prefix; response and semantic caching skip the model for repeat prompts.
11. Intelligent Prompt Routing sends easy prompts to cheaper models; smaller models cut both cost and latency.
12. Tame the bill with Budgets, cost allocation tags, application inference profiles, Service Quotas, and rate limiting.
13. Stream, measure time to first token, and report p50 and p99, never just the average.
14. Ship staged rollouts with versions and aliases, keep retries and throttling in place, and use cross-region inference profiles for failover.