For security and infrastructure teams

The Vercatus crawler

Vercatus analyses sites at their owner's request. We crawl only domains a customer has verified they control, and we identify ourselves on every request — including cryptographically, so you never have to trust a user-agent string.

How to recognise us

User agent

Vercatus/1.0 (+https://vercatus.com/crawler)

Signature

Every request carries an RFC 9421 HTTP Message Signature with the web-bot-auth tag and Signature-Agent: "https://vercatus.com". Our Ed25519 public key is published at /.well-known/http-message-signatures-directory on this domain (key id Lo05gqn5…YN0).

Source addresses

We do not ask you to trust IP addresses: cloud egress changes, and a fixed list is the first thing an impostor copies. The signature is the identity. Customers who require an address allow-list receive a dedicated egress on request.

Identity probes

To measure how your site treats other crawlers, we also send a small number of requests carrying the published user agents of Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. These are always signed as Vercatus, so they are distinguishable from the real crawlers and from spoofing.

What we request

• robots.txt, your sitemaps, and a small set of well-known files (llms.txt, /.well-known/*).
• A sample of pages listed in your sitemaps — up to the customer's plan budget (1,000 to 10,000 per run), typically a few hundred.
• Each sampled page is requested a handful of times, once per machine identity being measured.
• Nothing else: no form submissions, no authenticated areas, no scraping of content for resale, and no use of your content to train models.

Crawl behaviour

Rate

Two requests per second per host at most, from a single origin, with automatic backoff on 429 or 5xx. Configurable lower on request.

robots.txt

Honoured for the Vercatus user agent. If you disallow us, we stop — and tell the customer their own site is blocking the analysis they asked for.

Re-fetching

A monthly full run samples afresh; weekly change runs re-request only the pages already known. Page bodies are compared by content hash, so unchanged pages are not re-analysed.

Rendering

A headless Chromium loads pages to measure what a JavaScript-executing crawler sees. Media and fonts are not downloaded.

Allow-listing

On Cloudflare, add a WAF skip rule for requests whose Signature-Agent header equals"https://vercatus.com", or allow the verified-bot signature directly. On Akamai, Fastly or a self-managed WAF, the same signature can be verified against the published keys. Matching on the user-agent string alone also works, but anyone can send that string — the signature is the part worth trusting.

Blocking us

Disallow Vercatus in robots.txt and we stop crawling within one run. If you believe we are crawling a site you own without your authorisation, write toabuse@vercatus.com and we will suspend it while we check the domain-verification record.