Agent Evaluation: The Demo Worked. Can You Ship It?

The agent solved the demo. It read the request, found the right record, drafted an answer and updated the system. Everyone in the meeting watched it work. What nobody saw was whether it would behave correctly when the customer was ambiguous, the tool timed out or the same request arrived twice.

Agent evaluation is repeatable testing of the complete workflow: its inputs, tools, environment, final state and failure boundaries. A language model's plausible answer is only one part of that workflow. If the agent can write, send, delete or purchase, the thing to evaluate is what it actually did.

Begin with the acceptance contract

Write the task in business terms before choosing a model. For a hypothetical support agent: find the correct order, explain the applicable policy, propose an eligible action and escalate cases outside authority. “Be helpful” is not an acceptance condition. “Never issue a refund above the approved limit without confirmation” can be checked.

Keep correctness, permission, recovery, cost and time as separate dimensions. A fast wrong answer is wrong. A correct unauthorized action is still a failure. A perfect outcome after six human interventions is not autonomous completion. Compressing these into one average erases the product choices you need to make.

Anthropic's January 2026 article on agent evals describes tests that combine inputs with grading logic, and emphasizes the complications of multi-turn tool use and state changes. OpenAI's evaluation guidance likewise recommends task-specific tests, early and frequent evaluation, and numerical measures combined with human judgment. The common principle is evidence designed for the job, not a leaderboard borrowed from a different job.

Long chains amplify small failures

A model that succeeds at an individual step 95% of the time does not make a ten-step task 95% reliable. Under a simplifying independence assumption with equal step success and no recovery, the probability that every step succeeds is 0.95 to the tenth power: about 59.9%. That is arithmetic, not a measured claim about any vendor's agent.

Real failures are often correlated: one wrong retrieval can contaminate every later step. Recovery can also improve results. The purpose of the model is to stop a team from casually copying a component score into an end-to-end promise. Measure the whole task in the actual environment.

A clean sample is not proof of zero risk either. With no observed failures in one hundred independent, representative trials, the exact one-sided 95% upper confidence bound on failure probability is still about 2.95%. Change the sample size in the exhibit and watch the uncertainty shrink. More runs help only when they test the population and failure modes you actually care about.

Build a test set that can disagree with you

Start with representative tasks, then deliberately add boundary cases. Include incomplete information, conflicting instructions, unavailable tools, stale documents, adversarial content, expired permissions and duplicate requests. Keep a held-out set that is not used for prompt tuning, so repeated optimization cannot quietly turn into memorizing the exam.

Record how examples were sampled and which data was excluded. A test set made only from successful customer interactions will underrepresent the moments that require escalation. Sensitive examples need consent, minimization and controlled access; the retention ledger applies to evaluation data as well as live traffic.

Version the scenario, model, prompt, tool schemas, permissions and environment. Reset state between runs. If a cached result or a previously created record makes the next run easier, the score no longer describes the original task. Reproducibility is part of what makes an evaluation useful rather than ceremonial.

Grade the outcome, then inspect the trace

Use deterministic checks where the outcome is concrete: correct record changed, permitted amount, valid structure, no duplicate write. Human review is essential where meaning and context matter. A model-based grader can help triage or compare outputs, but calibrate it against human judgments and inspect disagreements. A second model's confidence is not ground truth.

The trace answers how the outcome happened: which documents were retrieved, which tool calls were attempted, what permissions were exercised and what the agent did after an error. A correct answer can mask an unsafe intermediate action. Conversely, a failed task can reflect a legitimate refusal rather than incompetence. Keep those categories separate.

Anthropic's earlier Building effective agents recommends simple, composable designs instead of unnecessary complexity. That remains a useful architectural principle even as tooling evolves. An agent that makes fewer consequential decisions is often easier to evaluate, monitor and constrain than a sprawling autonomous workflow.

A safety failure is not a rounding error

Suppose ninety-nine tasks succeed and one reveals another customer's record. Reporting “99% success” hides the decision that matters. Some failures require an unconditional hold for the tested release, regardless of the average. Define those red lines before results arrive, not when the team is trying to preserve a launch date.

The exhibit's all-pass rule is an intentionally conservative editorial example, not a universal threshold. Your release contract should reflect consequences, reversibility and the available human fallback. A read-only assistant and an agent authorized to transfer money should not inherit the same risk tolerance.

When a failure appears, name its class: retrieval, reasoning, authority, state, recovery or evaluator defect. Repair the underlying assumption and inspect every sibling path. Preserve a regression case. This is the practical extension of Harness²: a correction becomes durable only when the next run can expose the same mistake.

The product manager owns the release boundary

A product manager does not need to invent every grader. The essential work is deciding what counts as accepted, what must never happen, which users and tasks the evidence covers, and who can stop the rollout. Product judgment becomes more valuable precisely because building more capability is no longer the same as choosing the right capability.

Ship a bounded pilot: limited audience, minimal permissions, monitoring by failure category, a visible human escalation path and a rollback that actually works. Compare accepted outcomes and review effort against the current workflow. If reviewers are correcting every case, the agent may have moved work rather than removed it—the sign-off bottleneck in a new costume.

Run the suite again when a model, prompt, tool or data source changes. An old score describes an old system. Monitor live drift without pouring raw sensitive conversations into every dashboard. Product reliability and privacy belong in the same release discussion, not in separate cleanup meetings.

The best evaluation is a decision instrument. It tells you what works, where it fails and what evidence would change the release boundary. That is more valuable than a beautiful aggregate score, because it makes the next product decision less dependent on the room's enthusiasm for the demo.

Sources and limits

Primary sources support the standards and policy descriptions. The planning tools and proposed release practices are editorial models, not audit certificates, legal advice or measured production results. Provider documentation can change; verify the applicable agreement before procurement.

• Anthropic: Demystifying evals for AI agents (January 9, 2026)
• Anthropic: Building effective agents (December 19, 2024)
• OpenAI: Evaluation best practices