VercatusEvidence and disputed claims about Perplexity crawling
On 4 August 2025 Cloudflare reported that, on test domains which disallowed Perplexity in robots.txt and blocked PerplexityBot and Perplexity-User at the WAF, content was still retrieved by an undeclared crawler using a generic Chrome-on-macOS user agent (3–6 million requests a day, against 20–25 million for the declared Perplexity-User), rotating IPs and ASNs; Cloudflare de-listed Perplexity as a verified bot and added managed-rule heuristics to block it. In the same test ChatGPT-User fetched robots.txt and stopped when disallowed. Perplexity replied the same day that the 3–6 million requests were BrowserBase, a third-party cloud browser it uses for fewer than 45,000 requests a day, and that user-driven fetches are not crawling. Perplexity's crawler page states that Perplexity-User generally ignores robots.txt and publishes IP lists for both of its agents.
Sources
- 1Cloudflare · 2025Perplexity is using stealth, undeclared crawlers to evade website no-crawl directivesblog.cloudflare.com
- 2
- 3
CGrade C: reputable industry data or observation. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries