Allow-list the crawler
A run’s page and file requests over plain HTTP, the identity probes and the access watch carry an RFC 9421 HTTP
Message Signature with the web-bot-auth tag and a Signature-Agent header naming the key directory. That is the
part worth trusting: user-agent strings can be sent by anyone. Pages that need a browser to render are fetched with
a headless browser, which names Vercatus in its user agent and is not signed: a rule that allows by signature alone
leaves the rendered fetch to your bot management, and the audit then reads the page without JavaScript only. A few
small requests name Vercatus in the user agent and are not signed either: the read of your home page that looks for
a domain-proof meta tag, the fetch of your home page and icon for the icon beside your site’s name in the app, the
checks of a commerce feed and its protocol files, the fetch of a brand logo for a report, and the crawler-traffic
connection test (VercatusBot/1.0).
Cloudflare
Section titled “Cloudflare”Add a WAF custom rule with action Skip (bot management, managed challenge) for requests where the
Signature-Agent header equals "https://vercatus.com". If your zone uses Verified Bots, the signed
requests validate against the published directory without a rule. To let the rendered fetch through as well, add
a second rule for requests whose user agent contains Vercatus/; that match rests on the string, which anyone
can send, so keep its action narrow (skip the managed challenge only).
Akamai, Fastly, self-managed
Section titled “Akamai, Fastly, self-managed”Verify the Signature-Input / Signature headers against the Ed25519 keys published at
https://vercatus.com/.well-known/http-message-signatures-directory, and exempt verified requests from
challenges. If signature verification is not yet available on your platform, match the user agent
Vercatus/1.0. The directory holds keys, not addresses: a customer who needs an address list for a self-managed WAF
gets a dedicated egress on request.
What the identity probes look like
Section titled “What the identity probes look like”To measure how your site treats other crawlers, Vercatus also sends a small number of requests carrying the published user agents of Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Over plain HTTP these are signed as Vercatus, so they are distinguishable from the real crawlers and from spoofing. Allow-listing by signature covers them; allow-listing by user agent does not, and that is intended; the probes exist precisely to see what an unallow-listed identity receives.
What the rendered fetch looks like to your WAF
Section titled “What the rendered fetch looks like to your WAF”The browser asks for the page, then for every file the page loads, with JavaScript on: scripts (your own and third
parties’), stylesheets, images and fonts, but no audio or video. All of it carries the user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36 Vercatus/1.0 (+https://vercatus.com/crawler)
and none of it is signed. The page’s scripts run, so your analytics and tag scripts run too and may count the load as a
visit; filter the user agent containing Vercatus/ where your analytics allows it. The files come as fast as the
browser asks, in a burst, not at the page pace, so a rate-limit rule on assets can trip on them: exempt the same user
agent there. The load with JavaScript off runs no scripts and loads no fonts, and is otherwise the same.
A run’s page requests, the access watch, the preview and the checks are paced at no more than about two a second per
host (a gap of 600 ms between requests by default), and slow down after a 429, a 5xx or a timeout. The files a
rendered page loads are not paced (see above). Lower on request.
See also the public crawler page, written for security teams.
When access is not the problem
Section titled “When access is not the problem”If the crawlers get in and your brand is still absent from the answers, the reason is usually elsewhere: the pages engines read for your topics name your competitors, or you are not in the results of the searches they run. A topic’s Why shows which. For a step-by-step check of discovery and readable content, see Why your website is not showing up in ChatGPT Search.