Crawler traffic

Every probe Vercatus sends asks how a site treats a request that says it is GPTBot. Crawler traffic answers the question the other way round: what did GPTBot itself do, how many times, and what did it get. It is observed, not inferred, and it comes from your own edge, so the figures are the ones your CDN saw.

Connecting

On the site’s Setup tab, under Crawler traffic, Connect crawler traffic gives the site a token and a Worker to deploy. The Worker is about twenty lines: it passes every request through to your origin untouched and, for requests whose User-agent looks like a bot’s or which Cloudflare has verified as a bot, sends one small line to Vercatus after the response has gone out, so nothing a visitor sees waits on it. Browsers and ordinary clients are never forwarded.

1. In the Cloudflare dashboard, open Workers & Pages › Create application › Start with Hello World! and deploy it. Then Edit code, replace the whole file with the one from the setup guide (Copy or Download), and deploy again.
2. On the Worker’s Domains tab (Domains and routes), Add Route: example.com/* on the zone, and www.example.com/* if the www host is live. Use the host as written, not a wildcard such as *.example.com/*, which catches every subdomain. The zone has to be proxied through Cloudflare for the Worker to see its requests; the Worker’s own workers.dev address has no origin behind it and can be switched off on the same tab.
3. The first lines arrive with the next crawler visit; the Traffic and visits view fills in within the hour. It works on every Cloudflare plan, including the free one.

The setup guide walks through these steps one at a time. The last one has Test the connection: Vercatus asks the site for a page that does not exist, as a bot, and waits for the Worker to forward that very request back. A green answer within a few seconds proves the route, the proxying and the Worker together; a red one says which of them is missing. Test on the connection card does the same at any later time.

An admin can disconnect at any time: the token stops working and the stored aggregates are removed. Crawler traffic comes with every paid plan, Starter included.

When someone else holds the Cloudflare account

Often the person in the workspace is not the one with Cloudflare access; an IT department or an agency is. The setup guide has Copy instructions for IT and Download .md: a self-contained document with what the Worker does and does not do, the four steps for the current Cloudflare dashboard, the Worker file with the site’s own ingest URL, the route to add, and a one-line test that the ingest answers with 202. It can be pasted into a ticket or an email as it is. The file carries the site’s token, so send it privately; disconnecting in Vercatus revokes the token, and a new one is issued on reconnecting.

This page can be pointed to as well. The Worker below is the same file with the site’s ingest URL left as a placeholder; the real one is in the setup guide.

// Vercatus crawler traffic forwarder. Deploy as a Worker and add a route for your zone (e.g. example.com/*).
// It forwards one small line per bot request (method, path, status, User-agent, Cloudflare's bot verdict, the ray id)
// after the response is sent; browsers are never forwarded, and nothing about a visitor is.
const INGEST = "<the site's ingest URL, from the setup guide>";
const BOT = /bot|crawl|spider|slurp|fetch|agent|gpt|claude|perplexity|bytespider|ccbot|applebot|amazonbot|duckduck|yandex|baidu|petal|diffbot|cohere|anthropic|openai|meta-external/i;
export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const ua = request.headers.get("user-agent") || "";
    const cf = request.cf || {};
    const bm = cf.botManagement || {};
    if (cf.verifiedBotCategory || bm.verifiedBot || BOT.test(ua)) {
      const line = { ts: Date.now(), method: request.method, path: new URL(request.url).pathname, status: response.status, ua,
        verified: cf.verifiedBotCategory || bm.verifiedBot ? true : ("verifiedBot" in bm ? false : null), category: cf.verifiedBotCategory || null, score: bm.score ?? null, ray: request.headers.get("cf-ray") };
      ctx.waitUntil(fetch(INGEST, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(line) }).catch(() => {}));
    }
    return response;
  },
};

What each line carries, and what is kept

The Worker sends the method, the path, the response status, the User-agent, Cloudflare’s own verdict on whether the bot was who it claimed (its verified-bot category, and the bot score where the zone has Bot Management) and the request’s ray id. Vercatus names the crawler through its crawler catalogue, folds the path onto the site’s page types, and keeps one aggregate row per day, crawler, verified flag, status class and page type, with one sample path. No IP address, no full URL beyond the sample, and nothing about a visitor is stored; the lines themselves are not kept. Bots the catalogue does not name are counted as “other bots”, with the first word of their User-agent remembered for the curious. Thirteen months of aggregates are kept, so a review can look a year back.

A catalogued name that Cloudflare says is not verified is counted apart from the verified one, so a spoofed GPTBot shows beside the real one rather than as it.

What the Traffic and visits view shows

• One sentence: how many requests the crawlers made in the last 7 days against the 7 before, how many of the crawlers seen are AI crawlers, which came most, what share was refused and what share Cloudflare verified.
• Four figures: requests in 7 days, AI crawlers seen, the refused share, the verified share.
• Requests per day by crawler over 90 days, the eight busiest stacked, refused requests as a line.
• A table per crawler: operator and category, requests, the last 7 days with their change, the shares served, refused and rate-limited, the verified share, page types reached, last seen.
• Page types by crawler: which of the site’s page types each crawler reaches.

Where the observation appears

Once traffic is connected, the crawlers’ own requests sit beside every probe:

• The access matrix (the site’s Access tab, and the access watch) gains a last row, “The crawlers themselves, last 7 days”: per identity, how many requests the crawler made and what share was refused. Where the probe and the observation disagree, the observation wins the wording.
• Audit findings. An access finding about refused AI crawlers cites the crawlers’ own refusals as evidence and its confidence rises; where the crawlers themselves get through and only the probe is refused, the finding says so and drops to low. Two findings come from the traffic alone: a crawler that used to come and stopped (none in two weeks, at least twenty a week over the eight before), and a crawler served on the home page but refused on a page type.
• Recorded fixes. An access fix is confirmed from the crawler’s own status as well as the probe’s, with the requests refused before and after in the crawler’s own traffic.
• Verdicts. For a change to your own pages, a play’s verdict window starts when the engine’s own crawler first fetched the changed page after the check passed. Without crawler traffic, the check’s date is used, and the verdict says so. See Verdicts.
• Outcomes and demand. The assistant fetches (ChatGPT-User, Claude-User, Perplexity-User and the like: a person’s question made an assistant read your page) are an outcome on Results, and they calibrate AI demand across sites; see Vercatus Intelligence.
• The access watch reports, day against the week before, an AI crawler newly refused or newly served at the edge, and a crawler gone quiet (said once, until it returns). These go out through the alert channels under the access-changes switch and as the traffic.changed webhook.
• Results, the board brief, the quarterly review and the monthly brief carry the assistant fetches in the window against the starting point.

Other ways in

Every input ends as the same rows, so the Traffic and visits view and the figures do not care where the lines came from.

• Cloudflare Logpush (Enterprise zones): a Logpush job on the http_requests dataset with an HTTP destination at the site’s ingest URL (in the setup guide’s first step). Logpush sends gzipped NDJSON; the fields read are the request method, path, User-agent, the edge response status, the verified-bot category and the bot score, and the ray id. No other field is kept.
• Vercel and Netlify log drains: point the drain at the ingest URL. The request’s method, path, status and User-agent are read from the drain’s records; these sources carry no bot verification, so the verified share shows as unknown.
• Any other host (nginx, Apache, IIS, a CDN with an access log): an admin uploads the log under Setup, in the combined or common access-log format or as JSON lines, up to 200 MB, gzipped or not, as often as wanted. The file is read as it arrives and never stored; the answer says how many lines were read, how many crawler hits were kept, how many browser lines were dropped and how many lines could not be read.

Limits

Each site may send a number of lines a day that depends on the plan; past it, the rest of the day is not read and the Traffic and visits view says so.