The evaluation gap: why agent demos don't survive production

Every team that ships an LLM feature goes through the same arc. The demo is magical. The first week in production is fine. Then the support tickets start, and nobody can say whether the system got worse, the users got weirder, or it was always like this and the demo hid it. That uncertainty is the evaluation gap: the distance between "we saw it work" and "we know how often it works."

Closing it is not glamorous and it is not optional. It is also less work than people expect, because the first fifty examples do most of the job.

Why demos lie

A demo is a sample of one, drawn from the distribution of inputs the builder had in mind. Production draws from a different distribution: typos, half-formed questions, documents in the wrong format, users who want something adjacent to what you built. Four properties of language-model systems make the gap worse than it would be for ordinary software.

• Non-determinism. The same input can produce a different output. A demo that worked three times tells you less than three passing unit tests would.
• Compounding steps. An agent that is 95% reliable per step is about 60% reliable across ten steps. Demos rarely run ten steps.
• Silent failure. A wrong answer looks exactly like a right one. Nothing throws.
• Moving dependencies. The model behind the API changes on a schedule you do not control. Behavior that was true in your demo is a snapshot.

None of those are solved by a better prompt. They are solved by measurement.

What an eval actually is

Strip away the tooling and an eval is three things: a set of inputs, a way to score each output, and a threshold that says whether the score is acceptable. Everything else — dashboards, model graders, CI integration — is convenience on top.

I think about evals at three levels, and most teams need all three.

Level	What it checks	Example
Unit	One model call, one step	Given this ticket, does the classifier pick the right queue?
Trace	A whole multi-step run	Did the agent retrieve the right document, call the right tool with the right arguments, and stop when it should have?
Outcome	Did the user get what they needed	Was the refund issued correctly, with no manual correction afterward?

Unit evals are cheap and precise; they catch regressions in a single prompt. Trace evals catch the compounding problems — the agent that takes a correct action for the wrong reason, or loops. Outcome evals are the only ones that measure what the business cares about, and they are the hardest to automate, so they are usually sampled and reviewed by a person.

Building the golden set

The golden set is the dataset you run on every change. Here is how I build the first one.

1. Start with 30 to 50 real cases, not synthetic ones. Pull them from logs, support tickets, or the people who will actually use the system. Synthetic cases test your imagination; real ones test the product.
2. Over-sample the ugly ones. Missing fields, ambiguous requests, two documents that contradict each other, a user asking for something out of scope. The happy path will be fine. The set exists for everything else.
3. Label the expected behavior, not the exact text. "Should decline and offer to escalate" is a stable label. "Should say 'I'm sorry, I can't help with that'" is brittle and will fail the next time the model phrases it differently.
4. Record why each case is in the set. A one-line note — "customer gave the wrong order number" — turns a dataset into documentation of what the system is supposed to handle.
5. Grow it from production. Every incident that reaches a human becomes a case. The set should look more like your real traffic every month.

Fifty cases feels small. It is enough to catch most regressions, and it is small enough that a person can read every failure, which matters more than statistical power at this stage.

Scoring without fooling yourself

Use the strictest scorer the task allows. If the output is a category, a number, a JSON field, or a tool call with arguments, check it exactly. There is no reason to ask a model whether queue: "billing" matches queue: "billing".

When the output is free text — a summary, an explanation, a draft — you need a rubric, and in practice you need a model to apply it at scale. That works, with two rules I do not skip:

• Write the rubric as yes/no questions. "Does the summary mention the deadline?" grades reliably. "Is the summary good?" does not.
• Spot-check the grader. Have a person grade twenty of the same cases and compare. If the grader disagrees with the person more than it agrees, you have two unreliable systems, not one.

For agents, score the trace as well as the final answer. The questions I ask of a trace: Did it retrieve the right evidence? Did it call the tools it needed, and only those? Did it stop when the task was done? An agent that gets the right answer after calling six unnecessary tools is a cost problem today and a correctness problem the day one of those tools has a side effect.

Metrics that predict trust

Accuracy is where everyone starts, and it is rarely what users feel. The metrics I have found actually predict whether people keep using an AI feature:

• Task success rate — the outcome-level number, sampled and human-graded.
• Unnecessary-action rate — tool calls or steps that a reviewer would not have taken. This is the leading indicator of both cost and risk.
• Escalation rate — how often the system hands off to a person. A low number is not automatically good; a system that never escalates is probably guessing.
• Time to correct — when the output is wrong, how long does it take a person to fix it? A wrong answer that is easy to fix is a tolerable product. A wrong answer that is hard to spot is not.

I would trade a point of accuracy for a meaningful drop in unnecessary actions or time to correct almost every time.

Trace review as a habit

The single most useful practice I know is also the least technical: read ten traces a day. Not the failures the eval flagged — a random sample. Tag each one with a short category: wrong retrieval, misread instruction, tool argument error, over-eager, under-eager, fine.

After two weeks you have a failure taxonomy, and the taxonomy is your roadmap. It tells you whether to fix retrieval, tighten a tool schema, rewrite a system prompt, or add a human gate. Most "the model isn't smart enough" conclusions do not survive a week of reading traces; the problem is usually upstream.

Gate every change

Finally, wire the eval into the path every change takes. A prompt edit, a new tool, a retrieval tweak, a model version bump — each one runs the golden set before it merges, and a drop below threshold blocks it. The threshold does not need to be sophisticated. "No worse than main" is a fine place to start.

The most important case is the model upgrade. Providers now ship new tiers roughly every quarter, and their release notes include changes that will break code written against the previous version. An eval suite is the only tool that tells you whether the new model is better for you, as opposed to better on a leaderboard. Treat a model change like a dependency bump: pinned, tested, reviewed.