AI outside judge
An acceptance check outside an agent’s self-assessment, such as a deterministic test, held-out evaluation, measurement or separately briefed reviewer.
Define acceptance before execution and inspect actual artifacts or final state. A check should be able to reject the result.
A second model can share the first model’s blind spots. Separate context and different model identity reduce some coupling but do not create ground truth.
Connected
- Agent evaluationRepeatable testing of an AI agent's complete task, tool use, permissions and final system state against explicit acceptance criteria. A successful demo is not a production reliability estimate.
- Altitude, depth and rangeFelix Rieseberg’s framework for ambition with AI models: speed repeats work faster, altitude targets outcomes, depth uses expert instruments and range combines disciplines.
- PlaywrightMicrosoft's open-source browser automation framework, widely used by coding agents to verify web interfaces end to end.
- Claude Code subagentsSpecialised agents with their own instructions, tools and context window that the main Claude Code session can delegate tasks to.
Linked from
- Outcome-based promptingAsking an AI agent to pursue a specified change for a user or system, with evidence and boundaries, instead of merely producing a named deliverable.
- Multidisciplinary agent reviewA workflow in which specialist passes apply distinct methods and produce concrete evidence, rather than merely imitate expert personas.
Field notes that cover AI outside judge
More concepts & methods
- machine learningscientific study of algorithms and statistical models that computer systems use to perform tasks without explicit instructions
- deep learningspecialized form of machine learning that uses artificial neural networks to identify patterns in data
- genetic algorithmcompetitive algorithm for searching a problem space
- Prompt injectionAn attempt to make an AI system follow instructions embedded in untrusted material such as email, documents or websites instead of the authorized user’s task.
- Accessibility testingEvaluation of digital accessibility through automated checks, knowledgeable manual assessment and research with people with different access needs. Conformance evaluation and usability research provide complementary evidence.
- AI data retentionThe rules governing which AI inputs, outputs, logs and derived records remain in a system, for what purpose, and for how long. Not using data for training does not mean that it is not stored.