AI crawlers and robots.txt: which bots to allow
Dravik teamUpdated 7 min read
Most B2B companies should allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) because they decide whether you appear and get linked in AI answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended) are a separate choice about model training. Blocking a training bot does not remove you from AI search, and vice versa.
Why the bot names matter
Each major AI company now runs more than one crawler, and each crawler has a different purpose. One gathers content for model training. Another builds a search index so the assistant can cite and link pages. A third fetches a page in real time because a user asked about it.
If you block by company instead of by purpose, you can end up invisible in ChatGPT search while thinking you only opted out of training. The fix is to read each vendor's documentation and decide bot by bot.
Everything below reflects what the vendors themselves document. Vendors update these pages, so check the source before you change a live file.
The bots, by purpose
| User agent | Company | What the vendor says it does | Effect of blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models | Signals your content should not be used for training |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they may still appear as navigational links |
| ChatGPT-User | OpenAI | Fetches pages when a user asks ChatGPT or a custom GPT something, or through GPT Actions | OpenAI says robots.txt rules may not apply because the request is user-initiated |
| ClaudeBot | Anthropic | Collects web content that could contribute to training Anthropic's models | Signals future content should be excluded from training data |
| Claude-SearchBot | Anthropic | Crawls to improve the quality of Claude's search results | Anthropic says your content will not be indexed for search, which may reduce visibility |
| Claude-User | Anthropic | Retrieves pages when a person asks Claude a question that needs them | Claude cannot fetch your page for user queries |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results; Perplexity says it is not used to train foundation models | You may not appear in Perplexity results |
| Perplexity-User | Perplexity | Visits pages to answer a specific user question | Perplexity says this fetcher generally ignores robots.txt because a user requested it |
| Google-Extended | A product token that controls whether content Google crawls can be used to train Gemini models and for grounding in Gemini Apps and Vertex AI | Google says it does not affect inclusion in Google Search and is not a ranking signal | |
| Applebot-Extended | Apple | A token that controls whether content Applebot crawls can be used to train Apple's foundation models | Does not stop Applebot from crawling for Apple's search features |
The Google-Extended misunderstanding
This is the one most teams get wrong. Google-Extended is not a separate crawler that visits your site. It is a token in robots.txt that tells Google how it may use content that Googlebot already fetched.
Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews are part of Google Search. If you want to limit how your content appears in Google's AI features, Google points to the normal Search controls, such as the nosnippet and max-snippet directives, not to Google-Extended.
So blocking Google-Extended is a decision about Gemini training and grounding. It is not a switch that removes you from AI Overviews, and allowing it is not a way to get into them.
What about Bing and Copilot
Microsoft Copilot draws on the Bing index. There is no separate Copilot crawler token to allow. If Bingbot can crawl and index your pages, you are eligible. If you block Bingbot, you lose Bing search and the AI experiences built on it.
A sensible default for B2B companies
Most B2B companies want to be found and cited. They sell services or software, their content exists to attract buyers, and the risk of a model learning their public blog posts is low compared to the cost of being invisible. For them we recommend allowing everything.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you have no rules for a bot at all, it is allowed by default. Listing them explicitly is still useful because it documents the decision and makes later changes obvious.
If you want search visibility but no training
Some companies, especially those publishing original research or proprietary data, want AI search to cite them but do not want their content used for training. Split the rules by purpose.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
This keeps you eligible for ChatGPT search, Claude search and Perplexity results while signalling a training opt-out to each vendor that documents one.
Protecting private sections
Some parts of a site should never be crawled by anyone: customer portals, staging copies, internal tools, gated downloads. Robots.txt is a request, not a lock. Put private content behind authentication. Use robots.txt only to keep well-behaved bots away from low-value paths like search result pages or cart URLs.
User-agent: OAI-SearchBot
Disallow: /portal/
Disallow: /search/
Allow: /
Repeat the same Disallow lines inside each bot's block, because a crawler follows only the group that names it most specifically.
Common mistakes
- Blocking by company. A single rule aimed at "OpenAI" does not exist. Each user agent needs its own group.
- Assuming user-initiated fetchers obey robots.txt. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Anthropic says its bots honour robots.txt. If you truly need a page hidden, use authentication.
- Copying a security plugin's default. Some CDN and security tools added AI bot blocking as a toggle. Check your CDN and firewall settings, not just the robots.txt file, because a firewall block overrides anything robots.txt says.
- Forgetting the sitemap line. Pointing crawlers to your sitemap helps every bot find your important pages.
- Treating llms.txt as a replacement. An llms.txt file is a separate, informal proposal for describing your site to language models. It does not control crawler access. See our explainer on llms.txt.
How to check what is actually happening
- Open yourdomain.com/robots.txt in a browser and read it line by line. Look for broad Disallow rules you did not intend.
- Check your CDN or firewall for bot management settings. Look for anything labelled AI crawlers, AI bots or verified bots.
- Search your server or CDN logs for the user agent strings in the table above. If a bot you allowed never appears, something upstream may be blocking it.
- Ask the AI tools directly. Search for your company and key service pages in ChatGPT and Perplexity. If they cannot see recent content, access is a likely cause.
- Recheck after any site migration, CDN change or security plugin update, since those are the moments rules change without anyone noticing.
Once access is right, the next question is whether those tools trust and cite you. That depends on content and authority, covered in our guide to getting cited by ChatGPT.
Where Dravik fits
A misconfigured robots.txt or firewall is one of the fastest fixes in AI search, and one of the most common problems we find. If you are not sure what your site allows today, book a business audit and we will check crawler access along with the rest of your search and AI visibility.
Frequently asked questions
- Should I block GPTBot?
Only if you have a specific reason to keep your content out of OpenAI's model training, such as proprietary research. Blocking GPTBot does not affect ChatGPT search, which uses OAI-SearchBot. Most B2B marketing sites gain little from blocking it.
- Will blocking Google-Extended remove me from AI Overviews?
No. Google states that Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. To limit how your content appears in Google's AI features, use Search controls such as nosnippet.
- How long does a robots.txt change take to work?
It depends on the vendor. OpenAI and Perplexity both say changes can take around 24 hours to be reflected for their search bots. Others recrawl on their own schedule, so allow some time before judging the result.
- Do AI crawlers respect robots.txt?
The training and search crawlers from OpenAI, Anthropic, Perplexity, Google and Apple are documented as following robots.txt. User-initiated fetchers are different, and OpenAI and Perplexity both say theirs may not follow it. Robots.txt is a convention, so use authentication for anything truly private.
- Is there one rule that allows all AI bots at once?
Any bot without a matching rule is allowed by default, so a site with no restrictive rules already allows them. Listing each AI bot by name is still the clearest approach, because it makes your choices visible and easy to change later.
Book a business audit
Find the leak. Then we hunt it down.
We look at your busywork, your Google rankings and how AI assistants describe your category, then hand you a prioritized plan.
- $199 fixed price
- 30 minute call
- A written plan you keep
- Full refund if not worth it