Vercatus
Sign inGet started
Get started

Security · Our crawler

The Vercatus crawler

Vercatus analyses a customer's site only after the customer has verified they control it, and reads the public pages answer engines cite, politely, to see which brands they name.

How to recognise usBlock us

How to recognise us.
By the signature.

It names itself Vercatus in the user agent of its own requests, except where it measures how your site treats other crawlers, and signs its page and file requests over plain HTTP cryptographically, so you can verify them without trusting a user-agent string.

  • User agent

    Vercatus/1.0 (+https://vercatus.com/crawler)

  • Source addresses

    We do not ask you to trust IP addresses: cloud egress changes, and a fixed list is the first thing an impostor copies. The signature is the identity. Customers who require an address allow-list receive a dedicated egress on request.

  • Signature

    A run's page and file requests, the identity probes and the access watch, all over plain HTTP, carry an RFC 9421 HTTP Message Signature with the web-bot-auth tag and Signature-Agent: "https://vercatus.com". These name Vercatus in the user agent and are not signed: pages rendered in a headless browser, the home-page read for a meta-tag domain proof, the fetch of a host's icon, the checks of a commerce feed and of its protocol files, the fetch of a brand logo for a report, and the crawler-traffic connection test (VercatusBot/1.0). An agent task runs under the identity its owner chose. Our Ed25519 public key is published at /.well-known/http-message-signatures-directory on this domain.

    Key idLo05gqn5rxvfysifin51QvfM4CntGnWGxcyj28uYYN0

  • Identity probes

    To measure how a customer's verified site treats other crawlers, we also send a small number of requests carrying the published user agents of Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Over plain HTTP they are signed as Vercatus, so they are distinguishable from the real crawlers and from spoofing. The daily access watch does the same. An agent task a customer sets up on their own verified site runs a headless browser under an identity the customer picks: plain Chrome, a Chrome that names Vercatus and signs its requests to the site, or an assistant's declared user agent. Only the signed identity carries a signature.

Scope:
what we request.

Before verification a new workspace gets an access preview of its first domain and a check of the domains typed in its setup wizard, both described below.

  • robots.txt, your sitemaps, and a small set of well-known files (llms.txt, /.well-known/*).
  • On a customer's verified site: the pages its topics and plays point to, and a sample from its sitemaps, and the daily access watch, which can also be run by hand from the site's page.
  • On any other site: only the public pages answer engines cite, to see which brands they name, at a slower pace.
  • Before verification, on a workspace's first domain, once the owner's email is confirmed: the access preview. It reads robots.txt first and requests nothing else if that disallows Vercatus; otherwise it reads the home page, llms.txt and the sitemaps in a few dozen requests, one after another, and requests the home page again under the user agents of Googlebot and the AI crawlers, signed as Vercatus, to show which of them your site lets in. Nothing is analysed or measured; it is repeated when the workspace asks, no more than once a day.
  • A domain typed into the setup wizard: robots.txt first, then one request for its home page, to say whether Vercatus can read it. The public access check, which has no workspace, reads a domain's robots.txt, home page (under the same identities as the preview) and llms.txt the same way.
  • The home page and icon of a host the app shows (a site, a competitor, a cited domain), for the icon beside its name, at most once a month per host. This fetch names Vercatus in its user agent, is not signed and does not consult robots.txt. To verify a domain by meta tag, the home page of the domain being proven is read for the tag, also under the Vercatus user agent and unsigned.

Never: no form submissions, no authenticated areas, no scraping of content for resale, and no use of your content to train models.

How it behaves
on your site.

Paced, honouring robots.txt where it says so, and quiet about pages that have not changed.

  • Rate

    A run's page requests, the access watch, the preview and the checks are paced at no more than about two a second per host (a gap of 600 ms between requests by default), from a single origin, and slow down after a 429, a 5xx or a timeout. The files a rendered page loads are not paced. Configurable lower on request.

  • robots.txt

    Honoured for the Vercatus user agent when a run reads a verified site and when it reads a cited page. If a customer's own site disallows us, we stop and tell them their site is blocking what they asked for. The access preview before verification, the setup wizard's domain check and the public access check read robots.txt first, and read nothing else when it disallows Vercatus. The fetch of a host's home page and icon, for the icon beside its name in the app, does not consult robots.txt.

  • Re-fetching

    A customer's page is read again when a play's acceptance check runs or a deploy reports a change. Unchanged pages, compared by content hash, are not re-analysed; a refused page waits a week.

  • Rendering

    A headless browser loads pages to measure what a JavaScript-executing crawler sees. With JavaScript on it loads what the page asks for except audio and video, so your scripts, including analytics and tag scripts, run, and the cookies a page sets are kept while pages load. The load with JavaScript off runs no scripts and loads no fonts.

Allow-listing,
at two levels.

Customers who need an address list for a self-managed WAF receive a dedicated egress on request.

  • Proves identity

    Verification

    Your edge checks the RFC 9421 signature on the request against the public key published at the directory above. Bot-management products and any WAF that implements Web Bot Auth can do this, and it is the only method that proves the request is ours.

  • Convenience only

    Matching

    A rule that skips challenges for requests carrying the Signature-Agent: "https://vercatus.com" header or our user-agent string. It is quick to set up, but a header or string alone can be copied by anyone, so treat a match rule as a convenience, not as identity.

Blocking us

One line in robots.txt.

Disallow Vercatus in robots.txt and we stop reading the site once the rule is read. If you believe someone is analysing a site you own without your authorisation, write to abuse@vercatus.com and we will suspend it while we check the domain-verification record.

Crawler reference in the docsSecurity and data handling
robots.txt
# stop Vercatus on this site
User-agent: Vercatus
Disallow: /

Want to see what we read on your site?

Create your account and verify your domain. Analysis only visits sites its owner has verified; what is read before that is listed under Scope.

Get started

Cancel any time · Data stored in the EU