IMPLEMENT · Chapter 5

AI crawlers, robots.txt, and llms.txt for SMBs.

Every AI engine has two crawler categories: training bots that gather content for future model versions, and retrieval bots that power live answers. Citation share depends on the retrieval bots. Training bots are a licensing decision. Getting the robots.txt wrong is the single most common reason an otherwise GEO-ready site is invisible inside ChatGPT, Perplexity, and Claude answers.

KEY TAKEAWAYS

Allow retrieval bots — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot.
Decide on training bots — GPTBot, ClaudeBot, Google-Extended — based on content-licensing posture, not citation goals.
llms.txt is a one-file infrastructure bet, not a ranking signal. Ship it; do not depend on it.
"Block all AI bots" is the most expensive default an SMB can choose in 2026.

AI crawler decision matrix

Vendor	Training bot	Retrieval bot(s)	Recommendation
OpenAI	GPTBot	OAI-SearchBot, ChatGPT-User	Allow OAI-SearchBot and ChatGPT-User (retrieval and live browsing). Decide on GPTBot based on the licensing stance — blocking it does not affect citations.
Anthropic	ClaudeBot	Claude-SearchBot, Claude-User	Allow Claude-SearchBot and Claude-User. Block ClaudeBot only if the content is genuinely off-limits for training — and accept the trade-off in long-tail model exposure.
Perplexity	(no training bot — Perplexity does not train base models)	PerplexityBot, Perplexity-User	Allow both. PerplexityBot indexes content for answers; Perplexity-User fetches in real time when a user runs a query. Blocking either eliminates the site as a citation source.
Microsoft	Bingbot (dual role — indexes for Bing and Copilot)	adidxbot for ads, Bingbot for organic	Allow Bingbot — there is no separate Copilot crawler. Copilot grounds on Bing's index, so disallowing Bingbot eliminates the site from both Bing and Copilot.
Google	Google-Extended (training opt-out token)	Googlebot	Allow Googlebot. Google-Extended only opts content out of Gemini training — it does not affect AI Overviews citations, which still ground on the regular Google index.

A minimum-viable robots.txt for GEO

The block below is the starting point I deploy on new SMB sites that want broad AI-citation eligibility without committing content to training. Edit the training-bot lines based on the licensing position the business wants to take.

# Search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI retrieval bots — these power live citations. Allow.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# AI training bots — decide on licensing posture.
# Default below is allow; flip to Disallow if content is not for training.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# Default
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

llms.txt — ship it, but understand what it is

llms.txt is a proposed markdown file at the root of a domain that summarizes the site for language models — title, short description, and links to the most important pages.

The cost is one file. A few hundred lines of markdown that mirror the site map and highlight the pages that matter most.
No major provider has committed to it as of mid-2026. Not OpenAI, not Anthropic, not Perplexity, not Google. Treat llms.txt as developer-experience hygiene rather than a ranking signal.
The upside is asymmetric. If even one engine starts honoring it, the work is done. The downside of having it is zero.
Keep it honest. If llms.txt becomes a signal and it lists pages that no longer exist or misrepresents the site, the engine will downgrade trust. Treat it like the sitemap — auto-generated from the same source of truth.

Best for

SMBs starting from a robots.txt with no AI-bot rules and wanting broad citation eligibility.
Sites doing content licensing conversations and needing to demonstrate explicit training-bot opt-outs.
Founder-led businesses where the entity graph (author + organisation) is a real asset and citation share converts.

Fails when

Robots.txt blocks the retrieval bots — citation eligibility goes to zero immediately.
Training bots are blocked but the legal team forgets the retrieval bots also need an explicit Allow.
llms.txt is treated as a ranking play instead of an infrastructure bet.

Verify before

Test robots.txt with each AI bot user-agent in a fetch tool — confirm 200 responses.
Re-check the robots.txt after every CMS or framework upgrade — defaults change quietly.
Confirm llms.txt is regenerated whenever the sitemap is.

Frequently asked

Should I allow GPTBot in robots.txt?

GPTBot is OpenAI's training crawler — it gathers content to train future GPT models, not to power live ChatGPT answers. Allowing it does not directly affect citation share inside ChatGPT, which is driven by OAI-SearchBot and ChatGPT-User. Decide on GPTBot based on the licensing stance the business wants to take. The retrieval bots (OAI-SearchBot, ChatGPT-User) are the ones that matter for citations and should be allowed.

What is the difference between Perplexity-User and PerplexityBot?

PerplexityBot is the persistent crawler that indexes content for Perplexity's own search index. Perplexity-User is the live-fetch agent that retrieves pages in real time when a user runs a query. Both are needed for full Perplexity citation coverage — blocking either materially reduces the chance of being cited.

Is llms.txt actually used by any AI provider?

As of mid-2026 no major AI provider has publicly committed to honoring llms.txt as a ranking signal. The proposal is an Answer.AI initiative that summarizes a site for language models in a markdown file at the root. The implementation cost is one file; the upside is asymmetric. Ship it as a low-effort developer-experience win, not as a ranking play. Do not depend on it.

Does opting out of Google-Extended hurt my Google rankings or AIO citations?

No. Google-Extended is the opt-out token for Gemini's training on a site's content — separate from the standard Googlebot crawl and from AI Overviews grounding. Disallowing Google-Extended prevents future Gemini models from training on the content but does not affect Google rankings or AIO citation eligibility. The choice is purely a content-licensing decision.

What is the minimum viable robots.txt for GEO?

Allow Googlebot and Bingbot (table stakes). Allow the retrieval bots — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. Decide on the training bots — GPTBot, ClaudeBot, Google-Extended — based on the licensing posture. Disallow nothing else by default unless there is a specific reason.