Evals in the pipeline
An eval you run by hand is a demo with paperwork. The finale of this pathway is wiring quality measurement into the machinery: every prompt change gated, every model upgrade regression-tested, production watched for drift.
8 steps · 6 minPrompts are code now
A prompt change can break behaviour as thoroughly as a code change — silently, and with a one-word edit. So it gets code's discipline: version control, review, and a CI gate that runs the eval before merge. 'Tweaked the prompt directly in prod' should sound as alarming as 'edited the server live'.
A PM improves the support-bot prompt and the demo looks better. The eval, run in CI, drops from 95% to 88% — mostly on trap cases where the bot should refuse. What happened?
Stage the depth
Full evals cost real money and minutes — so stage them like any pipeline. A smoke eval (20 canary cases, seconds) on every change; the full set before merge; the expensive judge-graded sweep nightly or before release. Cheapest truth first — the same shape as week four, with a token bill attached.
Canary cases are your sharpest 20: the traps, the past incidents, the attacks that once worked. Small set, maximum signal.
Production drifts
A green eval measures yesterday's questions. Users invent new ones; upstream models change under APIs; data shifts. So production gets sampled: score a slice of real (consented, anonymized) traffic on the same rubrics, and watch the trend. When live scores sag below eval scores, your dataset has gone stale — harvest the new failures into it.
The loop: production failures → eval cases → gates that prevent their return. The “escapes teach the pack” rule, now with drift.
Eval: steady at 94%. Production sample scores: drifting down four straight weeks. Nobody changed the prompt or the model. What’s your first move?
Quality has a bill now
Two dimensions your old suites never had: latency and cost. A reply that's perfect in 30 seconds is a failed reply; a pipeline that spends $400 per run stops being run. Treat both as first-class metrics with budgets in the eval — p95 latency and cost-per-run sit next to the pass rate, gated the same way.
Final call of the pathway. A model upgrade: pass rate 94→96%, cost per run ×3, p95 latency 2s→9s for a live-chat assistant. Ship it?
The pathway, closed
Non-determinism took your assertions and left your judgment. You rebuilt the instruments: datasets, graders, calibrated judges, adversarial suites, gates, drift monitors. Every one of them is a testing idea you already knew, pointed at a system that rolls dice. The tools were never the job. The judgment was — and it still is.
Lesson complete.
Gated changes, staged eval depth, drift watched, failures read before verdicts shipped. That’s the full discipline: the tester’s mind, running continuously, pointed at systems that roll dice.