Blog · 16 September 2026
AI crawlers do not run your JavaScript. Here is what that means for a news site.
On one national news site, 78% of the words on the gallery template and 51% on the article template appear only after JavaScript runs. The crawlers behind ChatGPT, Claude and Perplexity never see them.
A search engine’s crawler and an AI assistant’s crawler behave differently in one way that matters more than any other: Google renders JavaScript, and the AI crawlers do not.
Googlebot fetches a page, queues it for rendering, and eventually indexes what the page looks like after scripts run. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot fetch the HTML and stop. Whatever is in that first response is what they read; whatever arrives afterwards — through a client-side framework, a “load more” call, a personalisation script — does not exist for them.
What that looks like on a real site
In a run on a national daily this week, Vercatus compared the words visible in the raw HTML with the words visible after a full browser load, for every sampled page, grouped by template:
| Template | Words in HTML | Words after JavaScript | Added by JavaScript |
|---|---|---|---|
| Gallery | 48 | 205 | 78% |
| Article | — | — | 51% |
| Home | — | — | 45% |
The gallery template is the sharpest case: 48 of 205 words are in the HTML. An AI assistant asked about the event in that gallery has a caption and a headline to work with. The article template — the site’s main product — paints half its text on the client.
The site is not doing anything unusual. This is what a modern front-end framework does by default. It is also why “we allow AI crawlers in robots.txt” and “AI crawlers can read our journalism” are different claims.
Why the usual test misses it
Most audits check whether a page can be fetched by an AI crawler — a 200 versus a 403. That is the precondition, and it fails often enough to be worth checking first. But a 200 that carries a quarter of the content passes that test and fails the reader.
The measurement that catches it is the render delta: fetch the page as a browser with JavaScript disabled and as a browser with it enabled, from the same client so the site treats both identically, and compare the extracted text. Vercatus does this for every sampled page and reports the ratio per template, because the fix is per template.
What to do about it
- Server-render or pre-render the primary content. The headline, the body, the captions, the structured data. Interactivity can stay on the client; the text that carries entities, claims and answers must be in the initial HTML.
- Check the templates, not the pages. A news site has twenty templates and a million URLs. A finding that says “the article template paints 51% of its text on the client, across 122 sampled pages” is a ticket; a list of URLs is homework.
- Measure after the deploy. Re-fetch the same pages and confirm the ratio dropped. If the number did not move, the fix did not ship where you thought it did.
The knowledge behind this
Vercatus’s findings cite the hypotheses they rest on. This one rests on tech.raw-html-for-ai-crawlers (grade A — vendor documentation and independent measurement that the major AI crawlers do not execute JavaScript) and tech.rendering-budget (grade A — Google’s own documentation of the render queue). The full knowledge base is 137 graded entries, each with sources and a revalidation date, and it is described in the documentation.