VercatusSearch crawlers and training bots have different purposes
Vendors separate training crawlers from search and user-initiated fetchers, and each robots.txt token is independent. GPTBot (OpenAI), ClaudeBot (Anthropic) and the Google-Extended token govern model training (Google-Extended also covers grounding in Gemini Apps and Vertex AI and does not affect inclusion or ranking in Google Search); OAI-SearchBot, Claude-SearchBot and PerplexityBot govern search visibility; ChatGPT-User, Claude-User and Perplexity-User fetch pages when a user asks. Disallowing a search bot removes or reduces visibility in that engine's answers and gives no training protection, because training is controlled by the separate token.
What we observe across audited sites
5 of 64 sites disallowed GPTBot, the training token, without disallowing OAI-SearchBot, the search token, and 5 disallowed both. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1
- 2Anthropic · 2026Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
- 3
- 4
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Answer engines, or search the whole base.
All Answer engines entries