The judge is also on trial
For subjective quality — tone, helpfulness, faithfulness — you'll end up using an LLM to grade an LLM. It works, it scales… and it has documented biases. Trusting a judge you never tested is the eval version of shipping untested code.
8 steps · 6 minWhy judges at all
'Is this reply helpful and polite?' has no regex. Humans grade it best but don't scale to 500 outputs per deploy. An LLM judge — a model prompted with grading criteria — scores in seconds for pennies. The catch: you've added a second non-deterministic system and pointed it at the first.
Rubrics, not vibes
A judge prompted 'rate this reply 1–10' produces confident noise. A rubric turns judgment into checkable sub-questions: Does it address the actual question? Does it invent facts not in the source? Is the tone professional? Each yes/no is far more reliable than one global score.
Write rubrics like acceptance criteria — if a sub-question can’t clearly pass or fail, it isn’t a criterion yet. Week-two thinking, new address.
Your judge gives a support reply 9/10. The reply confidently cites a warranty policy that doesn’t exist. What went wrong?
The documented biases
LLM judges have failure patterns testers should know cold. VERBOSITY bias: longer answers score higher. POSITION bias: in A/B comparisons, the first option wins more. SELF-PREFERENCE: models rate their own family's style higher. None of these are exotic — they show up in your first hundred grades.
Counters: grade pairs in both orders and average; cap length effects in the rubric; use a different model family as judge than the one being judged.
Comparing prompt A vs prompt B, the judge prefers A 70% of the time. You swap presentation order and rerun: now it prefers B 65% of the time. What have you learned?
Calibrate before you trust
Before a judge gates anything, make it prove itself: take 50–100 outputs, grade them yourself (or with the team), then compare the judge's grades to yours. High agreement on a labeled set is the judge's own passing test. Disagreements are gold — each one is either a rubric gap or a bias showing.
This is test-the-tests thinking: you’d never trust a test suite that had never caught anything. Same standard for judges.
Your judge agrees with human labels 94% of the time — but on the “model invents facts” cases specifically, only 60%. The team wants to start gating deploys on it. Your call?
The loop closes
You now test the model with an eval, and test the eval's judge against humans. That recursion isn't a problem — it's the job at its most senior: every measurement instrument earns trust before it gets a vote. Next up: attacking the model on purpose.
Lesson complete.
Rubrics over vibes, known biases countered, and a judge calibrated against human labels before it gets a vote. The grader is part of the system under test — always was.