GPTBot, ClaudeBot, PerplexityBot: Which AI Crawlers to Allow

QUICK ANSWER
If you want AI engines to recommend you, allow the crawlers that power AI search answers, such as OAI-SearchBot, PerplexityBot, Claude’s search and user agents, and Bingbot and Googlebot. Crawlers used mainly for model training, such as GPTBot, ClaudeBot and Google-Extended, are a separate choice. Check your firewall, not just robots.txt.
Why AI crawlers matter for GEO
Before an AI engine can cite or recommend your page, something has to fetch it. Each AI company runs its own crawlers, and many websites block them by accident through robots.txt rules, security plugins or CDN firewalls.
If the crawler that powers a search feature cannot reach you, that engine has to rely on other sources to describe you, and may not mention you at all.
The main AI crawlers
- OAI-SearchBot (OpenAI): used to surface sites in ChatGPT search results. Blocking it can keep you out of ChatGPT search answers.
- ChatGPT-User (OpenAI): fetches pages when a user asks ChatGPT to visit a link or browse.
- GPTBot (OpenAI): collects content that may be used to train OpenAI models.
- ClaudeBot (Anthropic): collects content that may be used for model training. Anthropic also uses separate agents for search and user-requested fetching.
- PerplexityBot (Perplexity): indexes pages for Perplexity’s answers. Perplexity-User fetches pages for user requests.
- Google-Extended: a control token, not a separate crawler. It tells Google whether content can be used for Gemini model training and grounding. It does not affect Google Search or AI Overviews eligibility, which use Googlebot.
- Bingbot: powers Bing and Microsoft Copilot. See Copilot and Bing.
- Applebot-Extended: controls whether Apple can use content for training its AI models.
Crawler names and purposes change, so check each company’s documentation before you make changes.
Search crawlers vs training crawlers
There are two different decisions:
- Search and user crawlers fetch pages to answer questions now. Blocking them usually reduces your visibility in that AI’s answers.
- Training crawlers collect content that may shape future models. Some businesses allow them to help models learn about their brand. Others block them for content protection reasons.
For most businesses that want AI visibility, allowing search crawlers is essential. The training decision depends on your content and your comfort level.
How to check and fix access
- Read your robots.txt at yourdomain.com/robots.txt and look for Disallow rules aimed at these user agents.
- Check your CDN or firewall settings. Some block “AI bots” with a single switch.
- Review security plugins, which may block unknown user agents.
- Test by requesting a page with each crawler’s user agent and confirming a normal 200 response.
- Keep your sitemap listed in robots.txt.
A simple robots.txt that allows everything, except private areas such as admin pages, works for most sites. Pair it with a clear llms.txt if you like.
Then make the content worth fetching
Access is only the first step. Once crawlers can reach you, the content has to answer questions clearly. See how to write pages AI engines can quote and our GEO checklist.
Key takeaways
- Allow search crawlers if you want AI visibility.
- Training crawlers are a separate business decision.
- Firewalls and security plugins often block AI bots by accident.
- Check access with a real request, not just robots.txt.
Frequently asked questions
If I block GPTBot, will ChatGPT stop recommending me?
Not necessarily. ChatGPT search uses OAI-SearchBot. Blocking GPTBot mainly affects training. Check OpenAI’s current documentation.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search, including AI Overviews.
Can I allow only some pages?
Yes. robots.txt rules can allow or disallow specific folders for each crawler.
See what AI says about your business
Get a free AI Mirror: what ChatGPT, Gemini, Claude and Google AI say about you today, and who they name instead.