VercatusAccess and crawlers
What decides whether a crawler is allowed to fetch the page at all: the tokens named in robots.txt, the control each vendor publishes for its own crawler, and what a server, CDN or bot manager does when one knocks.
- 18 published entries
- A16 grade A
- C2 grade C
- earliest re-check due 2026-12
A grade describes where an entry's sources come from, not how likely a change is to work. How grades are used
- Aofficial documentation or a standardR-021
Additional AI crawler tokens documented in 2026
Re-check due 2027-03 - Aofficial documentation or a standardR-003
AI crawler names and the purposes they serve
Re-check due 2027-03 - Aofficial documentation or a standardR-011
Bing and Copilot content-preview controls
Re-check due 2027-06 - Aofficial documentation or a standardR-005
Cloudflare controls for AI crawler access
Re-check due 2027-03 - Aofficial documentation or a standardR-001
Content signals policy
Re-check due 2027-03 - Aofficial documentation or a standardR-014
Crawler tokens used by other AI vendors
Re-check due 2027-03 - Aofficial documentation or a standardR-019
Google agent and non-search crawler controls
Re-check due 2027-06 - Aofficial documentation or a standardR-010
Google Search controls for generative AI features
Re-check due 2026-12 - Aofficial documentation or a standardR-020
How Claude retrieves web pages
Re-check due 2027-06 - Aofficial documentation or a standardR-004
How to verify a genuine Googlebot request
Re-check due 2027-06 - Aofficial documentation or a standardR-006
How user-requested AI fetches differ from crawling
Re-check due 2027-03 - Aofficial documentation or a standardR-009
OpenAI AdsBot
Re-check due 2027-03 - Aofficial documentation or a standardR-015
OpenAI cloud browser signed requests
Re-check due 2027-03 - Aofficial documentation or a standardR-012
Paywall markup and crawler access
Re-check due 2027-06 - Aofficial documentation or a standardR-002
TDM reservation EU
Re-check due 2027-09 - Aofficial documentation or a standardR-017
When ChatGPT may show a blocked URL as a navigation link
Re-check due 2027-03 - Creputable industry data or observationR-018
Evidence and disputed claims about Perplexity crawling
Re-check due 2027-03 - Creputable industry data or observationR-016
Why an agent browser can resemble ordinary browser traffic
Re-check due 2027-03
Other topics
- Agent readiness27
- Answer engines25
- Commercial12
- Content11
- Entities8
- Languages and regions7
- Measurement10
- Performance13
- Policies13
- Retrieval9
- Technical29
- Trust and provenance5
Every entry is searchable together in the full knowledge base. Vercatus plays cite entries by id. How the base is curated.