VercatusUseful content in initial HTML for AI crawlers
Googlebot renders JavaScript with a headless Chromium, but Vercel and MERJ's December 2024 traffic analysis found that the AI crawlers it measured (OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider, Meta-ExternalAgent) do not execute JavaScript: ChatGPT and Claude crawlers fetched JavaScript files (11.50% and 23.84% of their requests) without running them. Only Gemini (via Google's infrastructure) and AppleBot rendered. Content injected client-side is therefore missing for the non-rendering crawlers; Google itself notes that not all bots can run JavaScript.
What we observe across audited sites
On 9 of the 56 sites where the page could be read both ways, at least half of a deep page's words appeared only after JavaScript ran. Measured on 18 September 2026 on 64 well-known public websites, from Vercatus's own network; a probe shows what a site served to a request with that crawler's user agent, not what the vendor's own servers receive. How it was measured.
Sources
- 1
- 2
CGrade C: reputable industry data or observation. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Technical, or search the whole base.
All Technical entries