When software rolls dice
Your whole toolkit assumes the same input gives the same output. AI features break that assumption on purpose. Before you can test them, you need to understand exactly what died — and what survived.
7 steps · 5 minThe assumption you never noticed
Every test you've ever written contains a hidden promise: same input, same output, forever. assertEquals lives on that promise. An LLM feature samples from a probability distribution — ask it twice, get two different answers, both 'correct'.
This isn’t a bug to file. It’s the feature working as designed — and your assertions dying as designed.
Your test asserts the AI summary equals last week’s saved output. The model was updated overnight and the test fails — the new summary is different but arguably better. What actually failed?
What survives
Not everything melts. Deterministic parts stay deterministic: the API contract, the retrieval query, the guardrail code, the UI. Test those exactly as before. Only the model's judgment calls need new tooling.
A classic mistake is eval-ing everything. Save evals for the part that rolls dice; keep unit tests for the parts that don’t.
An AI support-reply feature: (a) fetches the customer’s order via API, (b) drafts a reply with an LLM, (c) blocks replies containing refund promises via a regex guardrail. Which part needs an eval rather than a normal test?
From verdicts to rates
One roll tells you almost nothing about a dice-rolling system. So AI testing swaps the unit-test verdict (pass/fail) for a measurement (pass RATE across many examples). 'It works' becomes '94% of 200 cases meet the bar' — a number you can track, compare, and gate releases on.
A teammate demos the feature on three prompts, all great, and calls it tested. What’s the sharpest reply?
Same tester, new instrument
Notice what didn't change: you still ask what matters, what breaks, and what's acceptable. Boundary thinking, severity judgment, reading failures — all of it transfers. The instrument changed from assertion to measurement; the judgment behind it is the one you already have.
Next: building the measurement instrument itself — your first eval.
Lesson complete.
You now know what non-determinism actually breaks (exact assertions) and what it doesn’t (your judgment about what matters). Everything else in this pathway builds on that split.