Evals.Tuning an AI feature.

An AI feature is only as good as the loop around it.

An eval is a test for a feature whose output you can’t predict. You keep a set of inputs with known good answers, grade every output against them, and get a score. Then you change one thing and score it again.

Here the feature drafts replies to support emails for Halyard & Co., a made-up outdoor-gear shop. It runs on Workers AI. Turn the knobs, run all eight labelled emails through five graders, and compare runs to see which cases you fixed and which you broke.

Live evals demo

Variant
prompt v1, 8b model, temperature 0
Judge
pinned at t 0, passes at 4
Concurrency
2 at a time
Allowance
3 runs / 10 min
then change one knobKNOBSthe variantpromptv1model8btemp0GOLDEN SET8 labelledTENTBOOTFLSKSTOVPRCLLAMPCHBKDISCFEATUREWorkers AI · 2 at a timellama-3.1-8b · t 0lane 1lane 2FMTESCFCTGRDJDGpinnedSCORECARD–/8no run yet
The eval loop. The knobs set a variant. Each of the 8 golden tickets goes through the feature, then five graders: format, escalation, facts, guardrails and a pinned judge. Results add up on the scorecard, and the loop closes back to the knobs.

Tuning bench turn one knob between runs

Prompt
Read prompt v1 in full
You answer customer support emails for Halyard & Co., an outdoor-gear shop.

Be helpful and friendly. Reply as JSON in this shape:
{"reply": "<the email you would send>", "escalate": <true or false>}

Set "escalate" to true if a person at the shop should handle the email instead of you.
Model
Temperature

Each run makes 16 model calls: 8 drafts and 8 judgements. You can start 3 every 10 minutes.

Try it one email, five graders

Email

One draft, then all five graders on it.

Case matrix The golden set. Run it to fill each grader’s column.
TicketFMTESCFCTGRDJDGCase
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
Format: not yet runEscalation: not yet runFacts: not yet runGuardrails: not yet runJudge: not yet runnot yet run
  • pass pass
  • fail fail
  • skipped skipped
  • running in flight
  • FMT format · ESC escalation · FCT facts · GRD guardrails · JDG judge

Scorecard no run yet

–/8cases pass when every grader that ran passes

Passes per grader
GraderCasesPassed
Format0 failed, 0 skipped—
Escalation0 failed, 0 skipped—
Facts0 failed, 0 skipped—
Guardrails0 failed, 0 skipped—
Judge pinned0 failed, 0 skipped—

The judge is pinned: Llama 3.3 70B, temperature 0. It never sees the label, and passes a reply that scores 4 or more out of 5.

History every visitor’s runs, newest first

Stored runs. Pick a baseline and a candidate to compare.
BaseCand.VariantCasesPassedWhen
v2·70b·1.26 of 8 passed6/8
v1·70b·1.20 of 8 passed0/8
v1·70b·0.60 of 8 passed0/8
v2·70b·06 of 8 passed6/8
v1·8b·00 of 8 passed0/8

v1·70b·1.2v2·70b·1.2+6

1 knob changed: prompt.

Fixed
  • Unused tent, two weeks old
  • Boots, changed their mind after 45 days
  • Engraved flask, no longer wanted
  • Stove leaking after three months
  • Tracking stuck for nine days
  • A day late, demands a 20% code
Regressed
None

0 still pass, 2 still fail. Above temperature 0 the same variant can score differently twice. Run it again before you believe a one-case change.

Start with the bare prompt on the small model. Run it, read the failures, then change one knob.

How it's wired.

One variant, eight cases, five graders
  1. Variant

    Your click sends the three knobs to /api/concepts/evals/runs. The Worker checks your allowance, 3 runs every 10 minutes, and the site’s daily cap of 300 runs, which it counts in Postgres.

  2. Feature

    Two tickets at a time, the Worker sends each email to Workers AI with your prompt, model and temperature, and asks for { reply, escalate } as JSON.

  3. Graders

    Four code graders run first, cheapest first: format, escalation, facts and guardrails. They’re plain functions and regular expressions, so they cost nothing and never change their minds. If the format fails, the rest are skipped.

  4. Judge

    A second model reads the policy, the email and the reply, but never the label, and scores it from 1 to 5. It’s pinned to one model at temperature 0, and a reply passes at 4.

  5. Stream

    Each case comes back as a Server-Sent Event the moment it’s graded, so the matrix fills in while the rest of the set is still running.

  6. Stored run

    When the eighth case lands, the scorecard is saved to Postgres, where every visitor’s runs share one history. A run that breaks is saved as failed, with whatever finished.

The pass rule · domain/graders.ts
// every grader that ran must pass
if (grades.some(g => g.verdict === "fail"))
  return false;
// on a labelled ticket, skipped is not a pass
if (labelled && grades.some(g => g.verdict === "skipped"))
  return false;
return grades.some(g => g.verdict === "pass");
The pinned judge · domain/judge.ts
export const JUDGE = {
  workersAiId: "@cf/meta/llama-3.3-70b-…",
  temperature: 0,
  passAt: 4,
} as const;

// a constant, not a knob: changing it
// would move every past score

Where evals earn their keep.

Any change to a model-backed feature
Before you change a prompt
A prompt edit that fixes one complaint can quietly break three other cases. A scorecard shows you both before your customers do.
Before you swap a model
A cheaper or newer model is a different feature. Run the same golden set through it and compare, case by case.
When a real failure comes in
Turn the bad reply into a labelled ticket. Now the suite remembers it, and that bug can’t come back unnoticed.
In CI, on every change
Code graders are cheap enough to run on every pull request. Save the model judge for a nightly run or a release.
What evals don't promise
Eight tickets is a smoke test.
It’s enough to show the loop, not to trust a feature. Real suites have hundreds of cases, and they grow from production failures.
The judge is a model too.
Model judges tend to prefer longer answers and their own style. That’s why this one is pinned, never sees the label, and runs after the code graders rather than instead of them.
Above temperature 0, the score moves on its own.
The same variant can score differently twice. Run it again before you believe a one-case change. Even at 0, the output can drift.
Tune against eight tickets for long enough and you’ll overfit them.
The score is a proxy, not the goal. Keep a held-out set you never tune against, and check it before you ship.
A pass is not a guarantee.
Graders only check what someone thought to write down. A reply can pass all five and still be wrong in a way nobody labelled.
Clear thinking. Deliberate engineering.© 2026 Owain Williams