VercatusAI crawler names and the purposes they serve
OpenAI and Anthropic each run separate robots.txt tokens per purpose. OpenAI: GPTBot crawls content that may be used for training; OAI-SearchBot surfaces sites in ChatGPT search (sites that disallow it are not shown in ChatGPT search answers, though they can still appear as navigational links); ChatGPT-User makes user-initiated visits for which robots.txt rules may not apply; OAI-AdsBot only visits pages submitted as ads. Each has a published IP list. Settings per token are independent, but if a site allows both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both purposes, and robots.txt changes take about 24 hours to affect search results. Anthropic: ClaudeBot collects content that may be used for training, Claude-SearchBot indexes for search quality, and Claude-User fetches when a person asks Claude a question; Anthropic states all three honour robots.txt and Crawl-delay, and that disabling Claude-SearchBot or Claude-User may reduce the site's visibility in Claude's search answers.
What we observe across audited sites
robots.txt disallowed GPTBot on 10 of 64 sites, OAI-SearchBot on 5, ClaudeBot on 13 and PerplexityBot on 7. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1
- 2Claude Help Center · 2026Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries