Machine identities
A site is not one thing. It is whatever each visitor is served, and machines are served differently from people, by design (rendering) and by accident (bot management). Vercatus requests each sampled page as several identities and compares the responses.
| Identity | What it represents | What its refusal means |
|---|---|---|
| Plain HTTP | A polite, identified HTTP client that does not execute JavaScript: most RAG fetchers, link-preview generators, uptime checkers | The site challenges anything that is not a browser; retrieval pipelines see a challenge page |
| Browser | A headless browser session, which is what Googlebot’s renderer and a person see | Rarely refused; if it is, the site is broken for everyone |
| Browser, no JS | The same browser with scripts disabled: what a non-executing crawler gets, from a client the WAF trusts | Used as the baseline for the JS-added text measurement when plain HTTP is challenged |
| Googlebot | Google’s published user agent | A refusal from an unverified origin is usually correct security; confirm with logs |
| GPTBot · OAI-SearchBot · ClaudeBot · PerplexityBot · Google-Extended | The AI vendors’ published crawler user agents; what each one is for is in Crawler tokens | Review the crawler purpose and verified request logs; blocking a training crawler does not establish a search or citation block |
Reading a refusal honestly
Section titled “Reading a refusal honestly”Identity probes carry a vendor’s user-agent string from Vercatus’s own network. A 403 proves the site
refuses that identity from this origin. Sites that verify crawlers by IP or reverse DNS will refuse a
spoofed user agent and admit the real one. Vercatus states this in every access finding and asks for
WAF or bot-management logs before you act on it.
Two patterns are unambiguous:
- Plain HTTP refused, browser served: a JavaScript challenge on non-browser clients. Every fetcher without a JS engine is affected, not only crawlers.
- AI crawlers refused, plain HTTP served: a rule targeting those identities specifically. Either an explicit AI-crawler block (on by default for new zones on some CDNs since 2025) or spoof verification. The logs settle which.
Identity probes run on a per-template sample rather than every page, and requests back off automatically
on 429; a 429 is reported as rate limiting, never as a refusal.
JS-added text
Section titled “JS-added text”For each page, Vercatus compares the words present in the non-executing view with the rendered DOM. The share that exists only after JavaScript runs is invisible to crawlers that do not execute scripts, which, as of 2026, includes the major AI crawlers (the evidence). Templates above 40% are flagged.
Check what the page actually contains
Section titled “Check what the page actually contains”Use the JavaScript content test to distinguish missing initial HTML from a crawler-access failure.