Skip to content

The crawler

User agent Vercatus/1.0 (+https://vercatus.com/crawler)
Signature RFC 9421 HTTP Message Signature, tag web-bot-auth, Ed25519; keys at /.well-known/http-message-signatures-directory on vercatus.com
Rate ≤ 2 requests/second/host, automatic backoff on 429 and 5xx
robots.txt Honoured for the Vercatus token
Rendering Headless Chromium for the browser perspectives; media and fonts are not downloaded
Scope robots.txt, sitemaps, a few well-known files, and a sample of sitemap URLs — no forms, no authenticated areas
Content use Analysis for the requesting customer only; never used to train models

The identity probes — requests carrying other crawlers’ published user agents — are always signed as Vercatus and limited to a per-template sample. See Allow-list the crawler and the public crawler page.