How AI Assistants Read a Website: Crawl to Citation

Before an assistant can recommend you, five different machines have to succeed in order: a crawler fetches the page, a renderer builds it, a parser chunks it, a retriever picks a chunk, and a model decides whether to trust it. Most sites fail at a step their owners never look at.

Step 1 — Crawl: can the machine get in?

Each AI company runs several crawlers with different jobs, and your robots.txt decides which get in. Blocking a training crawler is a legitimate business choice; accidentally blocking a search crawler removes you from an answer surface. The distinction matters and most templates get it wrong.

User-agent: GPTBot
Disallow: /

Step 2 — Render: is the content in the HTML?

Googlebot renders JavaScript, though rendering can be deferred. Many AI fetchers are simpler and read the initial HTML. If your prices, services or FAQs only appear after client-side scripts run, assume some assistants never see them. Server-side rendering or prerendering removes the risk.

Step 3 — Chunk: does each passage stand alone?

Retrieval systems split pages into passages. A passage that says “we offer this in all three locations” is useless out of context; one that says “Acme Dental offers Invisalign in Boca Raton, Delray Beach and Boynton Beach” is retrievable on its own. This is the core of answer-first writing.

Step 4 — Retrieve: is your passage the best match?

In a retrieval-augmented system, the query is matched against candidate passages. Passages that use the words buyers use, sit under question-shaped headings, and come from pages that already rank have the advantage — which is why SEO remains the foundation.

Step 5 — Compose: does the model trust you enough to name you?

The model weighs what it retrieved against what it learned. Consistent facts across your site, Business Profile and directories reduce contradiction; brand mentions across the web raise familiarity. See brand mentions are the new backlinks.

An assistant cannot recommend a page it could not fetch, could not read, or could not tell apart from your competitor's.

A five-minute audit

1. Open /robots.txt and check every AI agent against the matrix above.
2. View page source (not Inspect) on your top service page. Is the key copy there?
3. Pick any paragraph. Does it name the business and the place without needing the paragraph before it?
4. Search your brand name in three assistants. Do they describe you the same way your homepage does?

Next: AI pages vs. AI architecture and building a site-wide entity graph.