Skip to content
VercatusDocs

The crawler
User agent Vercatus/1.0 (+https://vercatus.com/crawler)
Signature RFC 9421 HTTP Message Signature, tag web-bot-auth, Ed25519; keys at /.well-known/http-message-signatures-directory on vercatus.com
Rate A run’s page requests, the access watch, the preview and the checks: no more than about 2 per second per host (a gap of 600 ms by default), slower after 429, 5xx and timeouts; the files a rendered page loads are not paced
robots.txt Honoured for the Vercatus token by runs, the access preview, the setup wizard’s domain check and the public access check. The home-page and icon fetch for a host’s icon in the app does not consult it
Rendering A headless browser for the browser perspectives. With JavaScript on it loads what the page asks for except audio and video, scripts included; with JavaScript off it runs no scripts and loads no fonts
Scope robots.txt, sitemaps, a few well-known files, and a sample of sitemap URLs; no forms, no authenticated areas
Content use Analysis for the requesting customer only; never used to train models

The identity probes (requests carrying other crawlers’ published user agents, sent on purpose) are signed as Vercatus over plain HTTP and limited to a per-template sample. See Allow-list the crawler and the public crawler page.

How to tell it is us

Section titled “How to tell it is us”

A run’s page and file requests over plain HTTP, the identity probes and the access watch carry an HTTP message signature over the request line and the headers, with the key id and the signature in Signature-Input and Signature. The public keys are published at https://vercatus.com/.well-known/http-message-signatures-directory, so a server or a WAF can verify a request without asking anyone: fetch the directory, find the key by its id, check the signature. The preview and the checks that read a domain before it is verified send the same signed requests.

Pages that need a browser to render are fetched with a headless browser. Those requests name Vercatus in the user agent, after the browser’s own, and carry no signature; so a rule that blocks every unsigned request that names Vercatus also blocks the rendered fetch. The same holds for a few small requests that name Vercatus in the user agent and are not signed: the home-page read that looks for a domain-proof meta tag, the fetch of a host’s home page and icon, the checks of a commerce feed and of its protocol files, the fetch of a brand logo for a report, and the crawler-traffic connection test (its user agent is VercatusBot/1.0). The identity probes and the daily access watch send other crawlers’ published user agents on purpose, to see how the site treats them; over plain HTTP they carry the same signature, so the signature, not the user agent, tells them from the real crawlers. An agent task a customer sets up on their own verified site runs a headless browser under an identity the customer picks: plain Chrome (unsigned), a Chrome that names Vercatus and signs its requests to the site, or an AI assistant’s declared user agent (unsigned, and not that assistant’s network).

The user agent alone is not proof, and we do not ask anyone to treat it as proof where a signature exists. That is the same standard we apply to the crawlers we measure: an unverified user agent is recorded as unverified.

What it never does

Section titled “What it never does”
  • It does not submit forms or follow links behind a login. It sends no cookies of its own; a page’s own cookies are kept only while pages are loading in the browser and cleared when it is idle.
  • It does not fetch audio or video when it renders. A render with JavaScript on loads everything else the page asks for, fonts, images and third-party scripts included, and runs the scripts, so your analytics and tag scripts run under a Vercatus user agent; a render with JavaScript off runs no scripts and loads no fonts.
  • It does not retry a page that answered 4xx, beyond the single request each identity makes, with two exceptions: a 429 is retried once, after the server’s Retry-After (three seconds when it gives none) and through the host’s pace; and the daily access watch asks again, once and two seconds later, when an identity that was served at the last check is turned away now (blocked, challenged or no answer), so a passing refusal is not reported as a change.
  • It does not use what it reads to train models, and it does not share one customer’s analysis with another.
  • It does not run a crawl or an analysis of a domain until that domain has been verified in the workspace that asked for it. What it reads before that is listed below.

The access preview before verification

Section titled “The access preview before verification”

Once the owner’s email address is confirmed, a new workspace gets a preview of the first domain it added, before the domain is verified and without a plan. It reads robots.txt first: when that disallows the Vercatus token the preview stops there and requests nothing else. Otherwise it requests the home page, llms.txt and the sitemaps the file names (or /sitemap.xml), and the home page again under the user agents of Googlebot, Google-Extended, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, signed as Vercatus: a few dozen requests. Nothing is analysed or measured and no page body is kept; the workspace sees the home page’s title and icon, how each crawler was answered, which AI crawlers robots.txt disallows, and how many pages the sitemaps list. It is repeated when the workspace asks, no more than once a day.

Other reads before verification

Section titled “Other reads before verification”
  • The setup wizard’s domain check. When a domain is typed in the wizard, robots.txt is read first; unless it disallows Vercatus, one request for the home page follows, to say whether Vercatus can read the site and to show its title and icon. Nothing is stored.
  • The public access check. The public API has an access check that needs no workspace (POST /api/public/access-check). It reads robots.txt first and requests nothing else when that disallows Vercatus; otherwise it requests the home page under the same identities as the preview, then robots.txt and llms.txt, one after another, and keeps the answer for a day.
  • The icon beside a host’s name. The app shows a mark for the sites, competitors and cited domains in a workspace. To make it, Vercatus reads that host’s home page and its icon, at most once a month per host, under the Vercatus user agent, unsigned, and does not consult robots.txt for it.
  • The domain proof by meta tag. Checking a ca-verify meta tag reads the home page of the domain being proven, under the Vercatus user agent, unsigned. The DNS check reads the domain’s TXT records, not the site.

If you would rather it did not come

Section titled “If you would rather it did not come”

robots.txt is honoured for the Vercatus token, so this is enough:

User-agent: Vercatus
Disallow: /

A workspace can only analyse a domain it has proven it controls, so in practice a blocked crawl means the site’s own team asked for the analysis and the site’s own rules refuse it. The run says so rather than working around it. To allow it instead, see Allow-list the crawler.

Load

Section titled “Load”

A run requests each sampled page once per identity, with page requests at no more than about two a second for the whole host (a gap of 600 ms by default), and backoff when a server answers 429 or 5xx or times out. The daily access watch, a manual check, the access preview and the checks that read a domain before it is verified send their requests one after another at the same pace. The files a rendered page loads come from the browser as it asks for them and are not paced. A 200-page run is a few thousand requests spread over several minutes, which is less traffic than a single visitor session with images. The page budget is the customer’s choice and the run dialog states the request count before anything starts.