Does blocking GPTBot remove me from ChatGPT?

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and pages opened on a user’s request use ChatGPT-User. Allow those two to be found and cited.

Does Google-Extended affect Google Search or AI Overviews?

No. Google-Extended is a token for Gemini’s use of your content. Google Search, AI Overviews and AI Mode crawl with Googlebot.

Do I need an llms.txt file?

It helps some assistants and tools find your key pages, but no major assistant requires it. Get crawler access, server-rendered content and structured data right first.

Which AI crawlers to allow in robots.txt: search, user and training bots

Three kinds of AI crawler

Each AI company runs more than one crawler, and they do different jobs. Treating them as one “AI bot” is how sites end up invisible in ChatGPT search by accident.

Search crawlers build the index an assistant searches when it answers: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude) and PerplexityBot (Perplexity). Block these and the assistant can’t find your pages to cite.
User-triggered fetchers read a page because a person asked about it, right now: ChatGPT-User, Claude-User and Perplexity-User. Block these and an assistant can’t open your page even when someone pastes your link.
Training crawlers collect text to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic) and the Google-Extended token (Gemini). Blocking them keeps your content out of training sets.

What to put in robots.txt

If you want to be recommended by assistants, allow the search and user-triggered agents. Whether to allow training is your call: it doesn’t affect whether ChatGPT search, Claude, Perplexity or Google’s AI Overviews can cite you. Google’s AI Overviews and AI Mode use Googlebot, not Google-Extended.

To allow AI search but opt out of training, add one group per training crawler:

User-agent: GPTBot followed by Disallow: /
User-agent: ClaudeBot followed by Disallow: /
User-agent: Google-Extended followed by Disallow: /

Everything else falls through to your User-agent: * group, so the search and user agents are allowed as long as that group allows them.

Check the CDN too

robots.txt is only half of it. Firewalls and bot-management settings at Cloudflare, Akamai and others can block AI agents before robots.txt is ever read, and some now block AI crawlers by default on new sites. Test your site with each agent’s user agent and look for a 403 or a challenge page instead of your content. The free SolidReach report runs this check for every agent above.

Make the page readable without JavaScript

AI crawlers mostly read the HTML your server sends. If your prices, rooms, services or answers only appear after JavaScript runs, many crawlers see an empty page. Server-side rendering, or at least the key facts in the initial HTML, fixes that.