VercatusJavaScript challenges can hide the intended page
Cloudflare's interstitial challenge pages evaluate the visitor's browser by running JavaScript; Cloudflare lists command-line tools without JavaScript execution and automated browsers as unsupported, so a crawler that is not exempted by the site's rules receives the challenge page instead of the content. For Google, content from a 4xx response is not used or indexed, and a 2xx page that shows an error message is reported as a soft 404.
What we observe across audited sites
The home page answered the GPTBot probe with a challenge page on 10 of 64 sites, and with a refusal on 14. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1
- 2
- 3Google Crawling Infrastructure · 2026How HTTP status codes affect Google's crawlersdevelopers.google.com
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Technical, or search the whole base.
All Technical entries