Machine identities
A site is not one thing. It is whatever each visitor is served, and machines are served differently from people — by design (rendering) and by accident (bot management). Vercatus requests each sampled page as several identities and compares the responses.
| Identity | What it represents | What its refusal means |
|---|---|---|
| Plain HTTP | A polite, identified HTTP client that does not execute JavaScript — most RAG fetchers, link-preview generators, monitoring tools | The site challenges anything that is not a browser; retrieval pipelines see a challenge page |
| Browser | A headless Chromium session — what Googlebot’s renderer and a person see | Rarely refused; if it is, the site is broken for everyone |
| Browser, no JS | The same browser with scripts disabled — what a non-executing crawler gets, from a client the WAF trusts | Used as the baseline for the JS-added text measurement when plain HTTP is challenged |
| Googlebot | Google’s published user agent | A refusal from an unverified origin is usually correct security; confirm with logs |
| GPTBot · OAI-SearchBot · ClaudeBot · PerplexityBot · Google-Extended | The AI vendors’ published crawler user agents | The engine cannot fetch, index or cite the page — unless the refusal is spoof verification |
Reading a refusal honestly
Section titled “Reading a refusal honestly”Identity probes carry a vendor’s user-agent string from Vercatus’s own network. A 403 proves the site
refuses that identity from this origin. Sites that verify crawlers by IP or reverse DNS will refuse a
spoofed user agent and admit the real one. Vercatus states this in every access finding and asks for
WAF or bot-management logs before you act on it.
Two patterns are unambiguous:
- Plain HTTP refused, browser served — a JavaScript challenge on non-browser clients. Every fetcher without a JS engine is affected, not only crawlers.
- AI crawlers refused, plain HTTP served — a rule targeting those identities specifically. Either an explicit AI-crawler block (on by default for new zones on some CDNs since 2025) or spoof verification. The logs settle which.
Identity probes run on a per-template sample rather than every page, and requests back off automatically
on 429; a 429 is reported as rate limiting, never as a refusal.
JS-added text
Section titled “JS-added text”For each page, Vercatus compares the words present in the non-executing view with the rendered DOM. The share that exists only after JavaScript runs is invisible to crawlers that do not execute scripts — which, as of 2026, includes the major AI crawlers. Templates above 40% are flagged.