Answers · AI search · As of 6 October 2026
Does robots.txt keep AI crawlers off my website?
Short answerOnly the crawlers that abide by it. robots.txt is an instruction to crawlers and not a technical block1. Conversely, an old rule in the file often blocks AI services without anyone having decided it.
What robots.txt achieves
The file tells crawlers which addresses of a website they may fetch. Google describes it as a means of controlling crawler traffic1.
Google itself points out the limitation: not every search engine supports the rules. And a blocked address can still appear in the search results, then without a description1.
Do AI providers comply?
Anthropic states that its bots honour the instructions in robots.txt and additionally support the Crawl-delay directive2.
Why the file alone blocks nothing
HasData analysed 592 websites that exclude GPTBot in robots.txt. 234 of these, or 39.5%, still served the page when GPTBot requested it3.
The file does not technically prevent retrieval. If you really want to block access, you need a rule on the server or in the firewall.
The more common question: are you accidentally blocking access?
Many businesses want to be mentioned in AI answers. It is then worth looking in the other direction: whether robots.txt, a firewall or the page itself is locking out the crawlers.
Sources
- Google Search Central: Introduction to robots.txt: developers.google.com
- Anthropic: Help article about the crawler: support.claude.com
- HasData: AI crawler block index: hasdata.com
Every figure on this page comes from the source named next to it and was read on the date shown above. Figures from studies describe the sample examined; they are no promise for an individual case.