AI Development: Evidence Before Autonomy

The agent finishes in four minutes. The summary sounds meticulous. The demo works. A week later, someone opens the page through the path nobody tested, and the impressive change becomes an incident.

That is not primarily a prompting problem. It is an evidence problem. The team promoted a convincing account of success into a release decision without establishing what actually worked, for whom, and under which conditions.

Start with a task contract, not a prompt contest

A useful task contract names the desired change, the scope boundary, the acceptance rule, and the forbidden side effects. “Improve this flow” invites interpretation. “Repair the checkout error without changing prices or permissions, and prove both successful and rejected paths” creates an evaluable task.

This does not require a huge document. It requires enough specificity that a checker can distinguish correct work from plausible work. The contract should be written before the agent starts, so the definition of success does not drift toward whatever the model happened to produce.

The broader practice already lives in spec-driven, maker-checker, governed scope. The missing addition in many teams is a reusable evaluation set: not one friendly task, but a collection of real tasks that exposes the workflow’s recurring failure modes.

Why a chain of good steps can still be fragile

The reliability exhibit assumes independent steps with equal success probability. At 95% per step, six steps all succeeding is approximately 73.5%, not 95%. Real systems may have correlated failures, so the calculation is an illustration rather than a deployment forecast.

The useful lesson is structural. Retrieval, planning, editing, testing, interpreting test results, and release authorization are different points where a workflow can go wrong. Excellent performance at one point does not erase mistakes elsewhere. Adding more agents or more steps can add capability and additional failure opportunities at the same time.

A recovery gate changes the design. It may detect a wrong edit before publication, request missing evidence, or stop a dangerous action. The objective is not “every step always succeeds”; it is “important failure becomes visible while it is still containable.”

“The agent said it passed” is a report. “The shipped path passed” is evidence.

The dated study that should change how we measure

METR’s July 2025 report describes a randomized study using early-2025 tools with 16 experienced open-source developers across 246 tasks. Developers expected AI access to make them faster; measured completion time increased by 19% in that setting.

This is not a universal claim that AI coding slows people down. It concerns a particular population, tools, period, and repository familiarity. It is also not a current benchmark for Claude Code. Its durable contribution is the gap between expectation, subjective experience, and observed task completion.

That gap should influence your evaluation design. Do not ask only whether people felt more productive. Measure a task from start to accepted finish, including review and correction. Ask which classes of work improved and which got worse. A blended average can hide a useful tool for one task and a liability for another.

Build a small evaluation set you can afford to rerun

Choose tasks from actual work: a bounded bug repair, a content change, an unfamiliar area of the codebase, an external dependency failure, and a permissions-sensitive request. Add negative cases where the correct behavior is to stop or ask for missing information. A system that completes everything indiscriminately is not necessarily capable; it may simply lack boundaries.

For each task, record the starting state, allowed tools, model or product version, and acceptance evidence. Grade correctness, scope adherence, total elapsed time, review burden, and significant failure types separately. Avoid collapsing everything into a score before you understand what the score hides.

Use deterministic tests where they can express a rule. Use browser checks for actual journeys. Use expert review for questions that cannot be automated honestly. A model-based checker can be helpful, but agreement between two models is not a substitute for a ground-truth check.

Separate the maker from the evidence collector

The maker’s incentives point toward completing the task. The checker’s job is to find the mismatch. A maker-checker loop is useful because it changes that role, not because it automatically creates independence. Two agents relying on the same incorrect assumption can agree perfectly and still be wrong.

Give the checker the original contract and the actual result. Require it to test sibling paths, not only the path the maker touched. If a fix corrects one caller of a shared function, search for the other callers. If a gate mounts a reduced-motion or demo configuration, do not claim it proves the full production experience.

Make evaluation part of product design

The point of AI evaluations is not to slow a capable team down with ceremony. It is to prevent speed from becoming an unmeasured transfer of work into review and incidents. The quickest durable workflow is the one that makes its mistakes cheap to detect.

This changes unit economics and makes the writer-to-operator transition credible. An operator does not need to write every line personally. The operator does need to know what evidence authorizes the next action.