AI systems · Software agents
AI crawlers
Also known as: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended
What is GPTBot?
AI crawlers are the user agents AI companies use to fetch web pages — some for model training, some for live search and answers — each controllable separately in robots.txt.
OpenAI separates GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (user-initiated fetches). Google-Extended is a robots.txt token controlling Gemini use of content and does not affect Google Search. Blocking training while allowing search agents is a coherent policy.
Related entities
- Retrieval-augmented generationRetrieval-augmented generation (RAG) is a technique in which a language model retrieves relevant documents first and then writes its answer grounded in them.
- llms.txtllms.txt is a proposed plain-text file at a site's root that gives language models a concise, curated map of what the site is and which pages matter.
- Search Engine OptimizationSearch Engine Optimization (SEO) is the practice of making a website crawlable, relevant and trusted so it earns positions in organic search results.
Field notes covering GPTBot
- AI Search: From Retrieval to RecommendationThe web is becoming both an answer supply chain and a decision surface. What changes for query intent, extractable evidence, citations, and the click that remains.
- How AI Assistants Read a Website: Crawl to CitationThe five machines between your page and a recommendation — crawl, render, chunk, retrieve, compose — and an interactive robots.txt matrix for AI crawlers.
Sources & identifiers
Search disciplinesSite structureAI systemsLocal searchDistribution & growthAgentic commerceHover to trace relationships · click to open