VercatusCloudflare controls for AI crawler access
Cloudflare's AI Crawl Control (formerly AI Audit), available on all Cloudflare plans, shows which AI crawlers request a zone's content and their request patterns, lets owners set allow or block rules per crawler, and tracks robots.txt compliance to identify violators. Its Pay Per Crawl feature (private beta) answers crawlers with HTTP 402 Payment Required and a crawler-price header unless they send a payment intent, and relies on Web Bot Auth signatures to verify crawler identity. It is the fastest way to reconcile declared policy with edge behaviour on Cloudflare-fronted sites.
Sources
- 1
- 2
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries