VercatusHow user-requested AI fetches differ from crawling
User-initiated fetchers retrieve a page when a person asks an assistant about it, and most vendors say robots.txt may not bind them: OpenAI says robots.txt rules may not apply to ChatGPT-User, Google that its user-triggered fetchers (including Google-Agent) generally ignore robots.txt, Perplexity that Perplexity-User generally ignores it, Meta that Meta-ExternalFetcher may bypass it and Amazon that Amzn-User may not follow all directives. Anthropic is the exception: it states Claude-User honours robots.txt. Blocking these fetchers at the WAF makes shared links unreadable inside assistants, a UX cost with no training-protection benefit.
Sources
- 1
- 2
- 3
- 4
- 5
- 6Claude Help Center · 2026Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries