robots.txt
The Robots Exclusion Protocol file that tells crawlers, including AI crawlers such as GPTBot and ClaudeBot, which paths they may fetch; standardised as RFC 9309 in 2022.
- Standard
- RFC 9309 (2022)
Connected
Field notes that cover robots.txt
More agent tooling & protocols
- Claude CodeAnthropic's agentic coding tool that reads a codebase, edits files, runs commands and verifies its work from the terminal, IDE or web.
- agent-html-kitAn open-source template kit and Claude Code skill (kit.js) that lets agents emit content-only HTML components while design lives in one shared file, published with a 72-run HTML vs Markdown benchmark.
- CLAUDE.mdA Markdown file of standing instructions that Claude Code loads at the start of every session, at user, project and folder level.
- Karpathy-inspired CLAUDE.md rulesA widely starred public CLAUDE.md with four behaviour rules: think before coding, simplicity first, surgical changes and goal-driven execution.
- AGENTS.mdAn open, tool-neutral Markdown convention for giving coding agents project instructions, introduced by OpenAI in August 2025 and now stewarded by the Linux Foundation's Agentic AI Foundation.
- Claude Code hooksShell commands that Claude Code runs automatically at lifecycle events such as before or after a tool call, so rules can be enforced by code rather than requested in prose.