AI systems · Practice
AI evaluations
Also known as: Evals
What is Evals?
AI evaluations are repeatable tests of a model or agent workflow against defined tasks, acceptance criteria, and observable failure categories.
Measure correctness, scope adherence, elapsed time, review burden, and important failures separately on representative tasks.
Benchmark performance and subjective productivity are not interchangeable with accepted production outcomes.
Related entities
- Altitude, depth and rangeAltitude, depth and range are Felix Rieseberg’s three directions beyond speed for AI work: pursue outcomes, use expert instruments and combine disciplinary methods.
- Claude DuetClaude Duet is Anthropic’s proactive personal AI agent for delegated work in connected apps, introduced as an invitation-only research preview in October 2026.
- Digital accessibilityDigital accessibility is the design and implementation of digital experiences that people with disabilities can perceive, understand and operate.
- Product experimentationProduct experimentation uses bounded tests and predefined evidence to reduce uncertainty about customer behavior, outcomes or technical feasibility.
- AI subscriptionsAI subscriptions are recurring-fee access products whose usable capacity depends on the provider, plan, workload, and applicable usage limits.
- AI unit economicsAI unit economics measures the total cost of delivering an accepted outcome, including inference, orchestration, human review, and relevant failure costs.
- Claude CodeClaude Code is Anthropic’s agentic development tool for inspecting projects, editing files, using tools, and verifying work under configurable permissions.
- Maker-checker reviewMaker-checker review is a control pattern in which one agent or person produces work and a separate one verifies it against a specification before it ships.
- Synthetic usersSynthetic users are AI-simulated personas used to generate and rank hypotheses about how real customers might react, before real-user testing confirms them.
Field notes covering Evals
- Altitude, Depth, Range: More Ambitious AI PromptingHow to be more ambitious with AI: Felix Rieseberg’s altitude, depth and range framework, fourteen templates and a practical, bounded Prompt Elevator.
- When Building Gets Cheap, Product Judgment Gets ExpensiveClaude Code expands the space of possible products. The differentiator is no longer a convincing specification—it is choosing, testing and shipping the right experience.
- More Agents. One Person Who Signs Off.Leaders are using AI more often, but adoption is not ability. As execution accelerates, accepted work—not generated output—becomes the real productivity measure.
- Agent Evaluation: The Demo Worked. Can You Ship It?A product leader's release contract for AI agents: representative tasks, final-state checks, critical failure gates and monitored pilots—with a reliability microscope and an interactive release desk.
- AI Economics: The Cost of Work You Can Actually UseInference prices fall. Review, retries, and failures still compound. An interactive guide to the unit economics behind a useful AI outcome.
- From Content Writer to Website OperatorClaude Code changes the boundary of a writing job: from handing over a document to owning a governed website pipeline. The opportunity, the workflow, and the limits.
- AI Development: Evidence Before AutonomyA convincing demo is not a release gate. How to evaluate agent-built software, contain failure, and measure the whole delivery loop instead of the typing speed.
- Roadmapping When the Model Gets Better Every Seven MonthsMETR found AI task horizons doubling roughly every seven months. How to plan in capability bands, write the test before the feature, and stop building what the next model gives away.
- The Practice: Spec-Driven, Maker-Checker, Governed ScopeTwo credible trials on AI coding point in opposite directions — 55.8% faster in one, 19% slower in the other. The difference isn't the model. It's the practice. Three patterns, with interactive math and cited data.
Sources & identifiers
Search disciplinesSite structureAI systemsLocal searchDistribution & growthAgentic commerceHover to trace relationships · click to open