LLM-as-a-Judge: Designing a Rubric You Can Trust

The situation

A team has a support-summarisation feature on Amazon Bedrock: it turns a long ticket thread into a three-sentence summary for the next agent to read. They already run Bedrock evaluation jobs with programmatic metrics, and ROUGE against a reference summary tells them nothing useful, because a good summary that shares few words with the reference scores badly and a bad summary that echoes the reference wording scores well. They want a quality signal that tracks what an agent would actually say about the summary, and they want it on thousands of examples, not a handful.

So they reach for a second model as the judge. Point one Bedrock model at the summaries and ask it to score them. The first cut is a prompt that says “rate this summary from 1 to 10”. It runs, it produces numbers, and the numbers are useless: nearly everything lands between 7 and 9, longer summaries score higher whether or not they are better, and when they use the same model family for the judge and the candidate, the candidate looks suspiciously strong.

The judge is confidently producing a score. The open question is whether the score means anything, and what it would take to trust it before wiring it into a release gate.

What actually matters

An LLM judge is a measuring instrument, and an instrument you have not calibrated is a random number generator with good manners. The first thing worth naming is that the judge does not measure quality; it measures agreement with whatever standard the rubric encodes. If the rubric is vague, the judge fills the gap with its own priors, and those priors are exactly the biases you are trying to avoid.

The most consequential design choice is what the judge is asked to produce. Pointwise scoring rates one answer against a rubric on its own: accuracy 4 of 5, completeness 3 of 5, and so on. It is easy to aggregate, it gives per-dimension signal, and it maps cleanly onto a threshold you can gate on. Its weakness is that “4 out of 5” has no fixed anchor across examples; the judge’s internal sense of a 4 drifts, so absolute scores are noisier than they look. Pairwise comparison asks a narrower question: given answer A and answer B, which is better? Models are markedly more reliable at ranking two things than at pinning an absolute number on one, because the comparison gives the judge a concrete reference point instead of an imagined scale. The cost is that pairwise gives you an ordering, not a level, and comparing every pair is quadratic, so at scale you compare against a fixed baseline rather than all-against-all.

Then there are the biases, which are specific and documented, not vague worries. Position bias: in a pairwise prompt the judge tends to favour whichever answer is presented first (or sometimes last), regardless of content. Verbosity bias: judges reward longer, more elaborate answers even when the extra length adds nothing, which is precisely the failure the team is seeing. Self-preference (or self-enhancement) bias: a judge tends to score outputs from its own model family higher, so a model grading its own siblings flatters them. Each of these has a matching control. Randomise the order of A and B across the run, and ideally score both orders and average, so position cancels out. Control for length, either by holding the two candidates to similar lengths or by explicitly instructing the judge to ignore length and reward concision. Use a judge from a different model family than the one under test, so self-preference has nowhere to land. And ask the judge to write its reasoning before it commits to a score, not after, because a score stated first becomes a conclusion the rationale then rationalises, whereas reasoning first makes the score follow from stated criteria.

The last thing that matters, and the one most often skipped, is calibration. Before you trust a judge on ten thousand examples, you check it against a few hundred that humans have already labelled. If the judge’s scores correlate strongly with the human scores, you have earned the right to scale it. If they do not, the judge is measuring something other than what you care about, and running it on more data just produces more wrong numbers faster. Calibration is what turns “the model said 8” into “the model said 8 and we know that means what we think it means”.

What we’ll filter on

1. What you are scoring: one answer against a standard (pointwise), or two answers against each other (pairwise)?
2. Rubric concreteness: explicit named criteria on a fixed, anchored scale, or a bare “is this good”?
3. Bias exposure: which of position, verbosity, and self-preference does this setup invite, and is each one controlled?
4. Rationale ordering: does the judge reason first and score second, or emit a bare number?
5. Judge independence: is the judge a different model family from the candidate under test?
6. Calibration: has the judge been checked against a human-labelled set before it grades at scale?

The judging landscape

Pointwise LLM scoring against a rubric. The judge sees one answer and the rubric, and returns a score per criterion plus a rationale. Strong for per-dimension diagnostics (“completeness is fine, faithfulnessFaithfulnessWhether every claim in an answer is actually supported by the source it was given, regardless of whether it happens to be true. is the problem”) and for gating on an absolute threshold. Weak on cross-example consistency, because the scale is anchored only in the judge’s head. Best when you need to know why an answer is weak, and when a rough absolute level is good enough.

Pairwise LLM comparison. The judge sees two answers to the same input and picks the better one, or declares a tie. More reliable than pointwise on the core “which is better” question because ranking is easier than absolute scoring, which makes it the honest choice for comparing two models or two prompt versions. It carries position bias hard, so order randomisation is mandatory, and it gives an ordering rather than a level, so you cannot read an absolute quality bar off it directly. At scale you compare each candidate against a fixed reference answer rather than all pairs.

Reference-based programmatic metrics. BLEU, ROUGE, BERTScore, exact-match: cheap, deterministic, reproducible, and they need a gold reference. They measure overlap with that reference, not quality, so a good-but-differently-worded answer scores badly. Fine as a cheap regression tripwire, poor as the primary signal for open-ended generation. This is the tool the team already found wanting.

Human review. The highest-fidelity signal and the standard everything else is calibrated against. Expensive, slow, and not perfectly self-consistent (inter-rater disagreement is real), so it runs on a representative sample, not the full set. Its job in a mature setup is to calibrate the judge and to adjudicate the outliers, not to grade everything.

Bedrock model evaluation, LLM-as-a-judge mode. Amazon Bedrock’s model evaluation offers three flavours: programmatic (automatic) metrics, human evaluation through a work team, and an LLM-as-a-judge option where you pick a judge model and it scores the candidate against built-in quality metrics (correctness, completeness, faithfulness, helpfulness, coherence, relevance, and responsible-AI checks like harmfulness) or criteria you supply. It scores pointwise per example with a rationale, aggregates for you, and can compare two models by scoring each. It is the managed path to running the pointwise pattern below at volume without building the harness yourself; you still own the rubric design, the bias controls, and the calibration against human labels.

Side by side

Approach	Answers	Needs a reference	Bias exposure	Scales	Trust lever
Pointwise LLM judge	How good, per criterion	✗	Verbosity, self-preference	✓	Concrete anchored rubric
Pairwise LLM judge	Which of two is better	✗	Position (high), verbosity	✓ (vs baseline)	Order randomisation
Programmatic metrics	Overlap with a gold answer	✓	None (deterministic)	✓	Reference quality
Human review	Ground-truth judgement	✗	Inter-rater spread	✗	Representative sample
Bedrock LLM-as-a-judge	Pointwise quality, managed	✗	Same as pointwise	✓	Rubric plus calibration

No row is trustworthy on its own from a standing start. The pattern that works is pointwise or pairwise LLM judging for breadth, with the rubric and bias controls designed deliberately, calibrated against a human-labelled sample before it gates anything.

The picks in depth

Write a concrete rubric, not a vibe. “Rate this summary from 1 to 10” gives the judge nothing to anchor on, so it defaults to a lazy 7 to 9 and to its own biases. Replace the single vague scale with named criteria and a fixed, described scale for each. For the summariser: faithfulness (does every claim in the summary appear in the ticket? 1 means invents facts, 5 means fully grounded), completeness (does it capture the resolution and the open action? 1 means misses both, 5 means both present), and concision (is it three sentences of signal? 1 means padded, 5 means tight). Describe what each score level looks like, in one line, so a 3 means the same thing on Tuesday as it did on Monday. An anchored rubric is the difference between a judge that measures and a judge that emits.

Prefer pairwise when you need to know which of two is better. Comparing the current prompt against a candidate prompt, or model A against model B, is a ranking problem, and judges rank far more reliably than they score. Show the judge both answers to the same input and ask which better satisfies the rubric, allowing an explicit tie. The catch is position bias, and it is strong enough to flip verdicts, so randomise which answer is A on every example, and for the ones that matter run both orders and keep the result only when the two orders agree; a flip when you swap the order is the judge telling you it is reading position, not quality. Pointwise still earns its place when you need a per-criterion diagnostic or an absolute gate (“ship nothing below faithfulness 4”), and the two compose: pointwise for the level, pairwise for the head-to-head.

Control the biases you cannot design out. Verbosity is the one biting this team, so instruct the judge to reward concision and penalise padding, and where you can, hold the compared answers to similar lengths so length is not a free variable. Self-preference is handled by picking a judge from a different family than the candidate; a model grading its own siblings tilts the table. And always make the judge state its reasoning against each criterion before it gives the number, because a number first is a verdict the rationale then defends, while reasoning first makes the number a conclusion drawn from stated evidence, which also gives you an auditable trail when a score looks wrong.

Calibrate before you scale, every time. Take one to a few hundred examples, have humans score them on the same rubric with the same scale, then run the judge on the same set and measure how well the two agree, per criterion. Strong agreement earns the judge a release gate; weak agreement on a criterion means the rubric is underspecified there, so you tighten the wording and re-check rather than shipping the judge as-is. On Bedrock this is a natural split: run the LLM-as-a-judge evaluation for breadth and a human evaluation job on a stratified sample for the calibration set, then compare. Recalibrate when the model, the prompt, or the data distribution changes, because a judge calibrated against last quarter’s traffic is only assumed-good against this quarter’s.

A worked example: fixing the summariser judge

The first judge prompt was “rate this summary 1 to 10”, judged by the same model family that wrote the summaries. Scores clustered at 8, longer summaries won, and the candidate model looked great against itself. Three biases stacked: no anchor, verbosity, and self-preference.

The rebuild changed four things at once. The rubric became three named criteria (faithfulness, completeness, concision), each 1 to 5 with a one-line description of every level. The judge model moved to a different family than the candidates, killing self-preference. The prompt now asks for a sentence of reasoning per criterion before the scores, and explicitly says to reward concision and ignore length as a virtue. And for the model-versus-model question, the setup switched to pairwise: show both summaries of the same ticket, randomised order, ask which better meets the rubric, run both orders on the tie-break set and discard disagreements.

Before trusting any of it, they pulled 150 tickets, had two support leads score the summaries on the same rubric, and ran the judge on the same 150. Faithfulness and completeness correlated well with the humans; concision correlated weakly, because the rubric’s level descriptions were mushy about what “padded” meant. They rewrote those level descriptions with concrete cues (a summary that restates the ticket verbatim is a 2; three sentences of new synthesis is a 5), re-ran, and the correlation came up. Only then did the judge move to the full set as a Bedrock LLM-as-a-judge evaluation, with the pairwise comparison reserved for the model-swap decision and the human sample kept as the recurring calibration check. The number the release gate now reads is one they have a reason to believe.

What’s worth remembering

1. An LLM judge measures agreement with the rubric, not quality; a vague rubric just launders the judge’s own biases into a number.
2. Pointwise scoring gives per-criterion diagnostics and an absolute gate but drifts across examples; pairwise comparison is more reliable because ranking two answers is easier than scoring one.
3. Write named criteria on a fixed scale with each level described, so a 3 means the same thing every time; replace “rate this 1 to 10” with anchored dimensions.
4. Position bias can flip a pairwise verdict, so randomise the order of the two answers and, where it matters, score both orders and keep only the agreements.
5. Verbosity bias rewards longer answers for nothing; instruct the judge to reward concision and hold compared answers to similar lengths.
6. Self-preference bias means a model flatters its own family, so pick a judge from a different family than the candidate under test.
7. Make the judge write its reasoning before the score, not after, so the score follows from stated criteria instead of being rationalised backwards.
8. Calibrate the judge against a human-labelled sample and check per-criterion correlation before you trust it at scale; weak correlation means fix the rubric, not run more data.
9. Recalibrate whenever the model, prompt, or data distribution changes, because a judge is only trustworthy against the traffic it was checked on.
10. Amazon Bedrock model evaluation runs LLM-as-a-judge alongside programmatic and human evaluation; it handles the scoring at volume, but the rubric, the bias controls, and the calibration are still yours to design.