VercatusInitial HTML and AI crawler rendering limits
Vercel and MERJ's December 2024 measurement (nextjs.org and the Vercel network, plus two job-board sites) found that none of the major AI crawlers rendered JavaScript: OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. ChatGPT and Claude fetched JavaScript files (11.50% and 23.84% of requests) without executing them; Gemini (on Googlebot's infrastructure) and AppleBot did render. For those crawlers only content in the initial HTML response counts, so render-blocking resources, INP and layout shift do not affect what they extract, and a site can be fast for people and empty for them.
What we observe across audited sites
On 9 of the 56 sites where the page could be read both ways, at least half of a deep page's words appeared only after JavaScript ran. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1
- 2
CGrade C: reputable industry data or observation. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Performance, or search the whole base.
All Performance entries