An AI feature is only as good as the loop around it.
An eval is a test for a feature whose output you can’t predict. You keep a set of inputs with known good answers, grade every output against them, and get a score. Then you change one thing and score it again.
Here the feature drafts replies to support emails for Halyard & Co., a made-up outdoor-gear shop. It runs on Workers AI. Turn the knobs, run all eight labelled emails through five graders, and compare runs to see which cases you fixed and which you broke.
Live evals demo
Variant
v1·8b·0prompt v1, 8b model, temperature 0
Judge
pinned at t 0,passes at 4
Concurrency
2 at a time
Allowance
3 runs / 10 min
The eval loop. The knobs set a variant. Each of the 8 golden tickets goes through the feature, then five graders: format, escalation, facts, guardrails and a pinned judge. Results add up on the scorecard, and the loop closes back to the knobs.
Tuning bench turn one knob between runs
Read prompt v1 in full
You answer customer support emails for Halyard & Co., an outdoor-gear shop.
Be helpful and friendly. Reply as JSON in this shape:
{"reply": "<the email you would send>", "escalate": <true or false>}
Set "escalate" to true if a person at the shop should handle the email instead of you.
Each run makes 16 model calls: 8 drafts and 8 judgements. You can start 3 every 10 minutes.
Try it one email, five graders
One draft, then all five graders on it.
Case matrix The golden set. Run it to fill each grader’s column.
The judge is pinned: Llama 3.3 70B, temperature 0. It never sees the label, and passes a reply that scores 4 or more out of 5.
History every visitor’s runs, newest first
Stored runs. Pick a baseline and a candidate to compare.
Base
Cand.
Variant
Cases
Passed
When
v2·70b·1.2
6 of 8 passed
6/8
v1·70b·1.2
0 of 8 passed
0/8
v1·70b·0.6
0 of 8 passed
0/8
v2·70b·0
6 of 8 passed
6/8
v1·8b·0
0 of 8 passed
0/8
v1·70b·1.2v2·70b·1.2+6
1 knob changed: prompt.
Fixed
Unused tent, two weeks old
Boots, changed their mind after 45 days
Engraved flask, no longer wanted
Stove leaking after three months
Tracking stuck for nine days
A day late, demands a 20% code
Regressed
None
0 still pass, 2 still fail. Above temperature 0 the same variant can score differently twice. Run it again before you believe a one-case change.
How it's wired.
One variant, eight cases, five graders
Variant
Your click sends the three knobs to /api/concepts/evals/runs. The Worker checks your allowance, 3 runs every 10 minutes, and the site’s daily cap of 300 runs, which it counts in Postgres.
Feature
Two tickets at a time, the Worker sends each email to Workers AI with your prompt, model and temperature, and asks for { reply, escalate } as JSON.
Graders
Four code graders run first, cheapest first: format, escalation, facts and guardrails. They’re plain functions and regular expressions, so they cost nothing and never change their minds. If the format fails, the rest are skipped.
Judge
A second model reads the policy, the email and the reply, but never the label, and scores it from 1 to 5. It’s pinned to one model at temperature 0, and a reply passes at 4.
Stream
Each case comes back as a Server-Sent Event the moment it’s graded, so the matrix fills in while the rest of the set is still running.
Stored run
When the eighth case lands, the scorecard is saved to Postgres, where every visitor’s runs share one history. A run that breaks is saved as failed, with whatever finished.
The pass rule · domain/graders.ts
// every grader that ran must passif (grades.some(g => g.verdict === "fail"))
returnfalse;
// on a labelled ticket, skipped is not a passif (labelled && grades.some(g => g.verdict === "skipped"))
returnfalse;
return grades.some(g => g.verdict === "pass");
The pinned judge · domain/judge.ts
export const JUDGE = {
workersAiId: "@cf/meta/llama-3.3-70b-…",
temperature: 0,
passAt: 4,
} as const;
// a constant, not a knob: changing it// would move every past score
Where evals earn their keep.
Any change to a model-backed feature
Before you change a prompt
A prompt edit that fixes one complaint can quietly break three other cases. A scorecard shows you both before your customers do.
Before you swap a model
A cheaper or newer model is a different feature. Run the same golden set through it and compare, case by case.
When a real failure comes in
Turn the bad reply into a labelled ticket. Now the suite remembers it, and that bug can’t come back unnoticed.
In CI, on every change
Code graders are cheap enough to run on every pull request. Save the model judge for a nightly run or a release.
The small print.
What evals don't promise
Eight tickets is a smoke test.
It’s enough to show the loop, not to trust a feature. Real suites have hundreds of cases, and they grow from production failures.
The judge is a model too.
Model judges tend to prefer longer answers and their own style. That’s why this one is pinned, never sees the label, and runs after the code graders rather than instead of them.
Above temperature 0, the score moves on its own.
The same variant can score differently twice. Run it again before you believe a one-case change. Even at 0, the output can drift.
Tune against eight tickets for long enough and you’ll overfit them.
The score is a proxy, not the goal. Keep a held-out set you never tune against, and check it before you ship.
A pass is not a guarantee.
Graders only check what someone thought to write down. A reply can pass all five and still be wrong in a way nobody labelled.