VercatusWhy AI crawler blocking depends on firewall configuration
On 1 July 2025 Cloudflare announced that it blocks AI crawlers accessing content without permission by default, with every new domain asked at signup whether to allow them, and launched Pay Per Crawl (private beta, HTTP 402). On 1 July 2026 it split AI traffic into Search, Agent and Training; from 15 September 2026 new domains block Training and Agent by default on pages that display ads while Search stays allowed, and multi-purpose crawlers that combine Search with Training (Cloudflare names Googlebot, Applebot and BingBot) are blocked for customers who chose to block Training. Sites can therefore drop out of answer engines without an owner's deliberate decision. Cloudflare Challenges need JavaScript execution, which command-line and other non-browser fetchers lack.
What we observe across audited sites
The home page was served to the GPTBot probe on 36 of 64 sites, challenged on 10 and refused on 14. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1Cloudflare · 2025Cloudflare just changed how AI crawlers scrape the Internet-at-large (press release, 1 July 2025)cloudflare.com
- 2
- 3
- 4
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Answer engines, or search the whole base.
All Answer engines entries