VercatusHow Claude retrieves web pages
Anthropic's help centre (April 2026) says all three of its bots, ClaudeBot (training), Claude-SearchBot (search indexing) and Claude-User (fetches at a user's request), honour robots.txt, including Crawl-delay, and do not bypass CAPTCHAs; disallowing Claude-User stops Claude retrieving the page for users. The Claude API web fetch tool fetches server-side and does not render JavaScript, accepts only text, HTML and PDF, refuses URLs blocked by robots.txt (error url_not_allowed), fetches only URLs that already appeared in the conversation or in earlier search or fetch results, and serves cached copies unless a caller sets use_cache false (tool versions from web_fetch_20260309). Citations are always on for web search results and optional for fetched pages.
Sources
- 1Claude Help Center · 2026Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
- 2
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries