Answers · AI search · As of 14 November 2026

Which AI crawlers exist, and what does each one do?

Short answerEach large AI service sends up to three kinds of crawler. One collects for training. One builds the search index. One only comes when a person is asking right now. If you want to appear in AI answers, let the search crawlers in.

Your next step: check for free whether AI services may read your website. It takes half a minute.

Ten names, three jobs

Name in your logsOperatorWhy it comes
OAI-SearchBotOpenAISearch: pages for ChatGPT search1
GPTBotOpenAITraining: content for future models1
ChatGPT-UserOpenAIOn a user's request: a person asks, ChatGPT fetches the page1
Claude-SearchBotAnthropicSearch: search index for Claude2
ClaudeBotAnthropicTraining2
Claude-UserAnthropicOn a user's request2
PerplexityBotPerplexitySearch, no training3
Perplexity-UserPerplexityOn a user's request3
GooglebotGoogleSearch: the same index also feeds AI Overviews and AI Mode4
Google-ExtendedGoogleNot a crawler of its own: a name in robots.txt that controls use for training Gemini5

What does that mean for your robots.txt?

Each name gets its own rule. OpenAI states that the rules for OAI-SearchBot and GPTBot are independent of each other1. So you can block training and allow search.

Fetches on a user's request are a special case. OpenAI writes that robots.txt may not apply to ChatGPT-User1. Perplexity writes that Perplexity-User generally ignores robots.txt3. The reason is the same: a person is asking, not an automatic collector.

What robots.txt can and cannot do: Does robots.txt keep AI crawlers off my website?

And Google?

Google needs no separate crawler for AI Overviews and AI Mode. Both draw on the normal search index that Googlebot fills4.

Google-Extended is a name you can use in robots.txt. It controls whether your content is used for training Gemini. It has no effect on inclusion in Google Search5.

What this looks like in a real log

On our own website, over 16 days, OAI-SearchBot came on 16 days, PerplexityBot and GPTBot on 15 days each, ClaudeBot on 12 days and ChatGPT-User on 11 days6. The search crawlers come daily. Whether they get through on your site, the check shows in half a minute.

Sources

  1. OpenAI: Overview of OpenAI Crawlers: developers.openai.com
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?: support.claude.com
  3. Perplexity: Perplexity Crawlers: docs.perplexity.ai
  4. Google Search Central: AI features and your website: developers.google.com
  5. Google Search Central: Google's common crawlers (Google-Extended): developers.google.com
  6. Metatron Marketing: our own server logs for metatron.marketing, 30 October to 14 November 2026, counted by user agent

Every figure on this page comes from the source named next to it and was read on 14 November 2026. Figures from studies describe the sample studied; they are no promise for an individual case.

Related questions

All answers

Add as a preferred source on Google