VercatusHow to verify a genuine Googlebot request
User-agent strings are easy to spoof, so vendors publish ways to verify a crawler: Google by reverse DNS then forward DNS (googlebot.com, google.com or googleusercontent.com) or by matching published JSON IP lists (common crawlers, special crawlers, user-triggered fetchers and user-triggered agents); Bing through its Verify Bingbot tool; OpenAI through per-bot IP lists such as searchbot.json. A 403 to a spoofed user agent from an unlisted IP is correct security behaviour, so probes that only set a crawler's user agent are inconclusive without log or IP verification.
What we observe across audited sites
Probes that set only a crawler's user agent were refused or challenged on the home page of 27 of 64 sites for Googlebot and 27 for ClaudeBot: a copied user agent is not proof of what the real crawler receives. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1Google Crawling Infrastructure · 2026Verify requests from Google crawlers and fetchersdevelopers.google.com
- 2
- 3
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries