VercatusCrawler access
An answer engine can only ground on pages its fetchers can retrieve. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers (though they can still appear as navigational links); Anthropic says blocking Claude-SearchBot or Claude-User may reduce a site's visibility in Claude's search results; Google names robots.txt for Googlebot as the control for AI features in Search and asks that crawling be allowed both in robots.txt and by any CDN or hosting infrastructure. User-initiated fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, so a WAF or challenge block is what stops them.
Sources
- 1
- 2Anthropic · 2026Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
- 3
- 4
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Answer engines, or search the whole base.
All Answer engines entries