AI crawlers, robots.txt, and llms.txt for SMBs.

GPTBot is OpenAI's training crawler, meaning it gathers content specifically to train future GPT models rather than to power live ChatGPT answers in real time. Allowing or blocking it does not directly affect citation share inside ChatGPT today, since that live citation process is driven by two separate bots, OAI-SearchBot and ChatGPT-User, not by GPTBot at all. The decision on GPTBot specifically should come down to the content-licensing stance a business wants to take toward AI model training in general. The retrieval bots, OAI-SearchBot and ChatGPT-User, are the ones that actually matter for citations and should be allowed regardless of the GPTBot decision.

PerplexityBot is the persistent crawler that indexes a site's content in advance, building it into Perplexity's own standing search index over time. Perplexity-User is a separate live-fetch agent that retrieves a specific page in real time at the exact moment a user runs a query that touches on that page's content. Both bots are needed for full Perplexity citation coverage on a given page, and blocking either one on its own materially reduces the overall chance of that page getting cited in a Perplexity answer.

As of mid-2026, no major AI provider has publicly committed to honoring llms.txt as an actual ranking or citation signal. The proposal originated as an Answer.AI initiative that summarizes a site's content for language models inside a plain markdown file placed at the site's root. The implementation cost is genuinely low, just one file to create and maintain, but the practical upside remains unproven and asymmetric against that low cost. The reasonable approach is to ship it as a low-effort developer-experience nicety rather than treat it as a real ranking play, and not to depend on it for any measurable citation gains.

Opting out of Google-Extended does not hurt Google rankings or AI Overviews citations. Google-Extended is specifically the opt-out token that controls whether Gemini can train on a site's content, and it is kept entirely separate from both the standard Googlebot crawl that drives ordinary search rankings and the AI Overviews grounding process that drives citations. Disallowing Google-Extended in robots.txt prevents a future Gemini model from training on that content, but it has no effect on either Google's regular search rankings or AI Overviews citation eligibility today. The choice is purely a content-licensing decision, not an SEO or GEO one.

A minimum viable robots.txt for GEO starts by allowing Googlebot and Bingbot, which is simply table stakes for being found at all. On top of that, it should allow the full set of retrieval bots that power live AI citations: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User. The separate training bots, GPTBot, ClaudeBot, and Google-Extended, are a different decision entirely, made based on the business's own content-licensing posture rather than on citation eligibility. The file should disallow nothing else by default unless there is a specific, documented reason to block something, and it should be revisited whenever a new AI crawler enters wide use across the industry.

Cloudflare's AI Crawl Control feature and similar CDN-level bot-management products can block a retrieval bot even when the site's own robots.txt file explicitly allows it, because that kind of block happens at the network layer before the request ever reaches the site's actual server. The reliable way to check is to look directly inside the CDN's own bot-management dashboard, not just at the robots.txt file sitting on the server. A quietly widened block list at the CDN level is one of the most common silent causes of an SMB losing AI citations suddenly after an otherwise unrelated CDN or firewall configuration change.

Verifying that a bot claiming to be GPTBot or PerplexityBot is genuinely real matters for any traffic that is going to inform a licensing decision. Bad actors regularly spoof AI crawler user-agent strings specifically to scrape a site's content without any real permission to do so. OpenAI, Anthropic, and Perplexity all publish either IP address ranges or reverse-DNS verification methods that confirm whether a given request actually came from their real, legitimate bot. Running server log analysis with reverse DNS lookups is the reliable way to separate verified crawlers from spoofers, and any unverified traffic merely claiming to be an AI bot should be treated as ordinary, unprivileged traffic until proven otherwise.

A slow server response rate does hurt AI crawler retrieval, often even more sharply than it hurts Googlebot. Retrieval bots such as ChatGPT-User and Perplexity-User fetch a page live at query time and frequently operate under a tighter timeout window than the crawl budget Googlebot is generally allowed to work with. A page that takes several seconds to respond risks a failed or truncated fetch from one of these live-retrieval bots, and that failure shows up specifically as a missed citation opportunity rather than as a simple ranking drop the way a slow Googlebot crawl typically would. Server response time has effectively become a GEO metric in its own right now, not only a Core Web Vitals metric that matters for human visitors.