robots.txt and llms.txt for AI crawlers

Two small text files at the domain root now show up in every “are we ready for AI search?” checklist: robots.txt and llms.txt. One of them is a decades-old access-control file. The other is a newer, unofficial map. Mixing them up wastes a sprint.

robots.txt is still a gate

robots.txt tells crawlers which paths they may request. Search engines have honored the convention for a long time. AI vendors added their own user agents on top of the usual Googlebot.

The names that matter in practice include:

• GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI)
• Google-Extended (Google’s generative / Gemini training-related crawler — separate from Googlebot)
• ClaudeBot (Anthropic)
• PerplexityBot
• Applebot-Extended
• CCBot (Common Crawl, which many models have used as source material)

A Disallow: / on those agents is a complete block for that crawler. Sites copy a “block AI” snippet from a privacy post, deploy it, and then wonder why generated answers never cite the service pages. If you do not want training, say so on purpose. If you want to be eligible for retrieval and citation, do not lock the door.

Googlebot and Google-Extended are not the same allow-list. You can stay in organic search and still opt out of extended use. Read the group you actually edited.

Also check the boring misses: a wildcard Disallow: / meant for staging that shipped to production, or a sitemap URL in robots.txt that 404s.

llms.txt is a map, not a ranking lever

llms.txt (and sometimes llms-full.txt) is a Markdown file at the site root. The emerging convention is a short, human-and-model-readable table of contents: what the organization is, which pages matter, how to describe products or service areas.

It is useful the way a clean sitemap and honest about page are useful. It is not a switch that inserts you into ChatGPT. Many crawlers will never fetch it. Some will. Absence is a gap in the AI-facing stack — alongside missing schema and thin entity pages — not a failed optimization tactic.

If you add one, keep it true:

• Use the same name, address, and phone you use everywhere else.
• Link to canonical service and location pages, not a junk drawer of parameters.
• Do not stuff keywords. A model that reads the file can also read that you stuffed it.

An empty or auto-generated llms.txt that disagrees with the homepage is worse than none.

What to check on Monday

1. Fetch https://yourdomain/robots.txt. Search for the AI user agents above. Confirm you did not disallow / unless that is the policy.
2. Fetch https://yourdomain/llms.txt. 404 is common. That is a gap, not an emergency.
3. Confirm sitemap.xml is live and lists the URLs you actually want retrieved.
4. Look at schema on the homepage and key service pages (LocalBusiness, Organization, FAQPage only where it matches the page).

Crawler access is eligibility. Citation is a different problem — content, listings, and whether anyone else on the web describes you the way you describe yourself. Fix the gate first so you are not debugging visibility with the crawler already turned away.