Monitoring AI Bot Traffic in Server Logs: Capture, Verify, Count

Monitoring AI bot traffic in server logs means finding the requests AI crawlers and assistants made to your site, confirming each came from the vendor it names, and counting them by crawler, day, response status and page type. Your access log, or your CDN’s, is the only record of what the crawlers themselves did; a probe, an audit or a robots.txt check can only tell you what would happen to such a request.

This guide is published by Vercatus, which reads crawler traffic as one of its inputs. The manual method below works without it.

Capture the requests: access logs and Cloudflare

Every request carries a User-agent, and AI crawlers put a token in it: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot, Claude-SearchBot and Claude-User for Anthropic, PerplexityBot and Perplexity-User for Perplexity, and others. The crawler tokens reference lists what each one is for. In the combined log format used by nginx and Apache, the user agent is the sixth field when you split on double quotes:

awk -F'"' 'tolower($6) ~ /gptbot/' access.log

If your site sits behind Cloudflare, the requests a crawler makes may never reach your origin’s log, so capture them at the edge. Cloudflare’s Logpush sends the http_requests dataset as gzipped JSON lines on Enterprise zones. On any plan, including the free one, a small Worker on the zone can pass every request through untouched and forward one line for bot requests only. For crawler analysis you need few fields: method, path, user agent, edge status, Cloudflare’s verified-bot verdict and the ray id. Leave visitors’ IP addresses and full query strings out.

Verify that a crawler is who it says

A user agent is a claim, not an identity; anyone can send GPTBot from any machine. Verify with the method each vendor publishes:

• Published IP ranges. OpenAI publishes the address ranges of its crawlers; check the request’s address against the current list.
• Reverse and forward DNS. Google documents verifying Googlebot by resolving the address to a Google hostname and back.
• Your CDN’s verdict. Cloudflare marks verified bots on each request, which saves the lookups.

Count unverified requests that carry a crawler’s name beside the verified ones, never merged into them.

Count crawl frequency and coverage

Group the matched lines by crawler and day to get a frequency series:

awk -F'"' '{print tolower($6)}' access.log \
  | grep -oE 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|googlebot|bingbot' \
  | sort | uniq -c | sort -rn

Run it on each day’s rotated log. Crawl rates differ by site and change without notice, so the useful signal is a change against your own baseline: a crawler that used to come and has stopped, or one whose refusals are rising.

Coverage is which parts of the site each crawler reaches. Fold every logged path onto a page type (product, article, docs, pricing), then build a table of crawlers by page type, with the status each was served. A crawler served 403, a challenge page or 429 on a page type reached it without reading it; AI crawler 403 errors on Cloudflare covers how to find the rule behind a refusal.

Keep three kinds of request apart: training crawlers (GPTBot, ClaudeBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User). A fetch by the last kind means a person’s question made an assistant read your page.

A dashboard of AI crawler visits

Store one row per day, crawler, verified flag, status class and page type rather than raw lines. That keeps months of history small and free of visitor data, and four views answer most questions: requests per day by crawler with refusals as their own line; a table per crawler with requests, change against last week, shares served and refused, verified share and last seen; page types by crawler; and the user-initiated fetches on their own.

If you would rather not build it, Vercatus does this for a site on every paid plan; the crawler traffic guide has the setup. A Worker on your Cloudflare zone (any plan), Logpush, a Vercel or Netlify log drain, or an uploaded access log of up to 200 MB feeds the same rows. It names each crawler from its catalogue, counts a spoofed name apart from the verified one, keeps thirteen months of daily aggregates and no IP addresses, and shows the Traffic and visits view: requests per day by crawler over 90 days, a table per crawler and page types reached. It also says when a crawler that used to come has gone quiet, and shows the crawlers’ own requests beside its access probes, so a probe that says GPTBot is refused can be checked against what GPTBot itself was served.

Crawler visits are an input, not a result. A page fetched daily can still be cited rarely; read the counts next to what the answer engines say on your topics.

Questions people ask

Does Cloudflare let me capture GPTBot hits for analysis?

Yes. On Enterprise zones, Logpush sends every request, with its user agent and Cloudflare’s verified-bot verdict, to a destination you choose. On any plan, including the free one, a small Worker on the zone can forward one line for each bot request after the response has gone out.

How do I measure AI crawler coverage of my site?

Fold every logged path onto a page type, then count, for each crawler, which page types it reached and what status it was served. Compare that table with the pages you want cited: a search crawler that reaches your home page but never your guides points at a linking, sitemap or access problem on those templates.

Can I get a dashboard of AI crawler visits by user agent?

Store one aggregate row per day, crawler, verified flag, status class and page type, and chart requests per day by crawler with refusals as their own line. Vercatus builds that view from your Cloudflare Worker, Logpush, log drain or uploaded logs on every paid plan.

Which tool shows how often ClaudeBot and GPTBot crawl my site?

Your own access log does, grouped by user agent and day, as long as you verify the requests. Vercatus shows the same per crawler, verified and spoofed counted apart, over 90 days, and says when a crawler that used to come has stopped.

What should I look for in AI bot log analysis software?

A maintained list of AI crawler tokens, verification of each request rather than trust in the user agent, counts by status and page type, history longer than your log retention, and a way to read crawler visits next to whether AI answers cite you. Logs alone show that a page was fetched, not that it was used.