Answers · AI search · As of 14 November 2026
Which AI crawlers exist, and what does each one do?
Short answerEach large AI service sends up to three kinds of crawler. One collects for training. One builds the search index. One only comes when a person is asking right now. If you want to appear in AI answers, let the search crawlers in.
Your next step: check for free whether AI services may read your website. It takes half a minute.
Ten names, three jobs
| Name in your logs | Operator | Why it comes |
|---|---|---|
| OAI-SearchBot | OpenAI | Search: pages for ChatGPT search1 |
| GPTBot | OpenAI | Training: content for future models1 |
| ChatGPT-User | OpenAI | On a user's request: a person asks, ChatGPT fetches the page1 |
| Claude-SearchBot | Anthropic | Search: search index for Claude2 |
| ClaudeBot | Anthropic | Training2 |
| Claude-User | Anthropic | On a user's request2 |
| PerplexityBot | Perplexity | Search, no training3 |
| Perplexity-User | Perplexity | On a user's request3 |
| Googlebot | Search: the same index also feeds AI Overviews and AI Mode4 | |
| Google-Extended | Not a crawler of its own: a name in robots.txt that controls use for training Gemini5 |
What does that mean for your robots.txt?
Each name gets its own rule. OpenAI states that the rules for OAI-SearchBot and GPTBot are independent of each other1. So you can block training and allow search.
Fetches on a user's request are a special case. OpenAI writes that robots.txt may not apply to ChatGPT-User1. Perplexity writes that Perplexity-User generally ignores robots.txt3. The reason is the same: a person is asking, not an automatic collector.
What robots.txt can and cannot do: Does robots.txt keep AI crawlers off my website?
And Google?
Google needs no separate crawler for AI Overviews and AI Mode. Both draw on the normal search index that Googlebot fills4.
Google-Extended is a name you can use in robots.txt. It controls whether your content is used for training Gemini. It has no effect on inclusion in Google Search5.
What this looks like in a real log
On our own website, over 16 days, OAI-SearchBot came on 16 days, PerplexityBot and GPTBot on 15 days each, ClaudeBot on 12 days and ChatGPT-User on 11 days6. The search crawlers come daily. Whether they get through on your site, the check shows in half a minute.
Sources
- OpenAI: Overview of OpenAI Crawlers: developers.openai.com
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?: support.claude.com
- Perplexity: Perplexity Crawlers: docs.perplexity.ai
- Google Search Central: AI features and your website: developers.google.com
- Google Search Central: Google's common crawlers (Google-Extended): developers.google.com
- Metatron Marketing: our own server logs for metatron.marketing, 30 October to 14 November 2026, counted by user agent
Every figure on this page comes from the source named next to it and was read on 14 November 2026. Figures from studies describe the sample studied; they are no promise for an individual case.
