The evaluation suite. The unglamorous engineering behind agents that ship.

Offline evaluation. The spec the agent owes the team.

The offline evaluation suite is the closest thing an AI agent has to a specification. It is a growing set of real cases the agent must handle correctly, with expected outputs, that runs every time the team changes a prompt, swaps a model or updates a retrieval index. Writing it well forces the team to make explicit what good looks like, which prompt engineering on its own tends to leave implicit.

Golden sets. A representative sample of real cases drawn from production traffic or from operator examples, with the expected outputs documented. The golden set should grow over time as new cases are added, especially the cases where the agent surprised the team. We have walked into engagements where the team had a few dozen golden cases. We have walked into other engagements where the team had several thousand, organised by use case and difficulty. The right size depends on the breadth of the agent's scope, but in our experience the golden set is rarely too large and frequently too small.

Regression suites. Every change the team makes runs against the golden set automatically. A regression is a case that used to pass and now fails. The team should know within minutes of a deployment whether the new version regresses on cases the old version handled. The teams we work with who hold this discipline tend to keep their agents stable through model swaps and prompt iterations. The teams who run regression manually tend to discover the failures from users.

Adversarial probes. A separate set of cases designed to make the agent fail in instructive ways. Prompt injections, ambiguous instructions, contradictory inputs, edge cases that the operator might never produce on a normal day but that a determined adversary or an unusual situation would. The adversarial set is where the safety properties of the agent get tested, and it should grow over time as new attack patterns emerge in the broader ecosystem.

Online evaluation. What the agent does in the wild.

Offline evaluation gives the team confidence that the agent passes a defined set of cases. Online evaluation tells the team what the agent is actually doing in real use, which is always richer than any offline set can capture. The three patterns that matter are shadow traffic, A/B routing, and sampling for human review.

Shadow traffic. A new version of the agent runs in parallel with the production version, on the same inputs, without exposing the new version's outputs to operators. The team compares the two versions case by case and surfaces the divergences. Shadow traffic is the safest way to validate a major change before exposing it to real users, and the teams that build the infrastructure for it once tend to use it on every significant deployment.

A/B routing. When the team is ready to expose a new version, a slice of real traffic gets routed to it while the rest continues on the existing version. The metrics that matter, operator satisfaction, override rate, latency, cost per session, are compared across the two. A/B routing tells the team whether the change improved the experience for real users, which is the only test that finally matters.

Sampling for human review. A small fraction of real sessions are pulled aside for human review by operators, by domain experts, or by the build team. The reviewers tag the cases that surprised them, that the agent handled poorly, or that exposed a new pattern. Those cases feed the golden set and the adversarial probes. The loop closes, the evaluation suite grows, and the agent gets better at the cases that matter to real users.

Drift detection. The early warning system.

The third component of a serious evaluation suite is drift detection, and it is the one most often missing in the teams we audit. Drift comes in four flavours, each of which the suite should monitor explicitly.

Input drift. The distribution of inputs hitting the agent shifts. A new product line is launched, a season changes, the operators start using the agent for a different kind of task than originally designed. Input drift tends to be the early warning that the agent is being used in a way the team did not anticipate, and it should trigger a conversation rather than a silent degradation in quality.

Output drift. The distribution of the agent's outputs shifts. The agent is producing different kinds of answers than it used to, or the same answers in different proportions. Output drift can signal a problem with the underlying model, an issue with retrieval, or simply the input drift catching up with the output side. Either way, it is information the team needs to see.

Latency drift. Response times are creeping up. This is usually the first dimension to drift quietly, because the agent is still producing useful answers, just slower. The teams that track latency at the percentile level rather than the mean catch the long-tail problems before they become user-visible.

Cost drift. The cost per session is rising, sometimes because of increased context length, sometimes because of more tool calls, sometimes because of a model price change. Without runtime cost tracking, the first signal the team gets is a finance report at the end of the quarter, and that is not a conversation anyone wants to have.

The team and the tooling.

The composition of the team that owns evaluation matters more than the specific tools they use. In our experience, a capable engineer who owns evaluation full-time on a serious production agent, working alongside the build team and the operators, produces better results than a larger team without ownership. The discipline is the asset. The tools follow.

On the tooling question, our practice is deliberately neutral. The landscape includes open-source frameworks for evaluation orchestration, observability platforms with native LLM trace support, and commercial vendors for specific regulated-industry needs. The right tool for a given client depends on what they already run, what their security posture allows, and what the team can maintain over time. We tend to recommend building on top of what already exists rather than adding a parallel stack, unless the existing stack genuinely cannot meet the requirements.

What to build versus what to buy is the practical question. Our default recommendation is to buy the foundational layer, the trace ingestion, the dashboards, the basic alerting, and to build the use-case-specific layer, the golden sets, the regression suites, the operator review tooling. The team's expert time is better spent on the cases that are specific to their agent than on rebuilding infrastructure that exists in the market.

How the evaluation cost gets justified to the CFO.

Evaluation work is unglamorous and tends to be underfunded for the same reason. It does not produce demo-able output. It does not get applauded in steering committees. The conversation about its cost, when it comes up with a finance leader who is reviewing the AI budget, usually happens at exactly the wrong moment, after the suite has been running quietly for some time and saving the programme from problems that did not happen.

The way we recommend running this conversation is documenting the moments where evaluation has earned its keep. The regression caught before deployment. The drift detected before users noticed. The adversarial case found before a real adversary did. The cost spike caught the same week it started rather than at the end of the quarter. The teams that maintain a running log of these moments tend to defend their evaluation budget easily. The teams that do not tend to discover that the budget conversation has already happened in their absence.

The evaluation suite is the part of the AI programme that does its best work when nothing happens. The art is making the absence of failure visible to the people who fund the work.

Where the credibility of an AI programme is built.

When a regulator, an auditor or a board committee asks how the team can defend the safety and reliability of an AI capability in production, the answer the team needs to give is rooted in the evaluation suite. The golden sets, the regression histories, the drift reports, the override rates, the human review samples. These are the artefacts that distinguish an AI programme that operates seriously from one that operates on hope. They are also the artefacts that the AI Act and similar regimes will increasingly require explicitly, particularly for high-risk categories where the evidence burden falls on the deploying organisation.

Building the evaluation suite with regulatory and audit requirements in mind from the first sprint is significantly cheaper than retrofitting them later. The teams we advise in financial services, healthcare and other regulated contexts have learned this. The teams that are still discovering it tend to be in the middle of a painful conversation about evidence they cannot produce.

If your team is building AI agents and the evaluation discipline has not yet been treated as a first-class investment, the conversation we usually open with is short. What does your golden set look like today. Who runs it on every change. What does your drift dashboard show. If those three questions have clear answers, you are further along than most. If they do not, the right next step is to invest in the suite before the next pilot is funded.