Do AI Crawlers Run JavaScript? How to Test Your Website's Content

In our 18 September 2026 measurement of 64 well-known sites, 9 of the 56 where we could compare the two readings kept at least half of a deep page’s words for after JavaScript ran, so a fetcher that reads only the first response sees under half of that page. Check yours before arguing about which crawler renders what.

Do not assume an AI crawler will execute the JavaScript your browser runs. Check whether the answer-bearing content exists in the initial HTML, then test the relevant crawler’s access separately. Server-rendering or pre-rendering essential text reduces dependence on the fetcher’s rendering capabilities.

In short

• Check whether the answer is in the initial HTML before debating which crawler renders what.
• Render the headline, body, tables, prices and captions on the server; keep interaction in the browser.
• Test access and content separately, and group findings by template.
• A readable page is a precondition, not a promise of citation.

The practical question is not whether a website uses JavaScript. It is whether a fetcher can read the headline, explanation, comparison or product details without completing the same interactions as a person.

Google rendering and AI retrieval are different paths

Google documents a crawl, render and index process for JavaScript pages. Rendering can occur after the initial crawl, and blocked resources or errors can prevent content from appearing as expected. See Google’s JavaScript SEO guidance.

AI assistants can draw on several retrieval paths, including search indexes and direct page visits. Vendor documentation distinguishes search, training and user-initiated fetchers; it does not justify treating every path as an identical browser. Review the relevant OpenAI crawler or Anthropic crawler before deciding which access policy to change.

A page readable without scripts is more resilient across these paths. That does not guarantee that an answer engine will select or cite it.

An observation from a news-site run

In the run described in the original article of 16 September 2026, Vercatus compared extracted text before and after JavaScript on an unnamed national daily. The gallery example contained 48 words in the initial HTML and 205 after rendering.

Those counts demonstrate a substantial gap in available text. They are a dated observation from one site, not a benchmark for publishers or proof that a particular AI engine missed a citation. The original article’s gallery percentage did not reconcile with its displayed word counts, so this update retains the counts without repeating that percentage.

A headline and a short caption may be readable while the explanation arrives later. A robots rule allowing a crawler and a page containing a complete answer are two separate requirements.

How to test content that depends on JavaScript

Pick representative URLs from your article, category, product and other important templates. Test the same canonical URL, location, viewport and consent state in each comparison.

1. Save the initial response. Record its status, headers and HTML. Check whether the headline, body, captions and essential tables appear in the response rather than only in scripts or later requests.
2. Load a browser without JavaScript. Extract the visible primary text. This helps isolate rendering dependence under browser-like request conditions.
3. Load the same page with JavaScript enabled. Wait for the primary content to settle, then extract the same region with the same rules.
4. Compare the useful passages. Identify which definitions, instructions or evidence are missing. Word counts help summarize a gap but do not establish answer completeness.
5. Test access separately. Review verified crawler requests in server logs and bot-protection events. A same-client rendering comparison does not prove that a vendor crawler is allowed through. For the home page, the free check of whether AI crawlers can read your site requests it as Googlebot, OAI-SearchBot, PerplexityBot and the training crawlers and reads your robots.txt, in about a minute.

Consent banners, navigation, personalized widgets and duplicate hidden text can distort a word-count comparison. Use consistent extraction rules and inspect the text behind the numbers.

How to interpret the result

Result	Likely investigation	Useful next action
Essential text appears only after scripts run	Client-side content dependency	Render the primary answer into initial HTML
Both browser versions contain the answer, but a verified crawler is blocked	Access policy or bot protection	Review rules for that crawler and path
Both versions contain the answer, but it is incomplete	Content coverage	Improve the canonical answer page
A response is successful but contains a challenge or empty shell	Response-content mismatch	Inspect the body and protection events

Avoid turning “200 OK” into a verdict that machines can read the page. Conversely, a lower raw word count is not automatically a problem if the missing words are decorative navigation rather than the answer.

Fix the template, then verify the same pages

Server-render or pre-render the primary content: headline, body, captions, product details and answer tables. Keep interactive controls where they help visitors. Ensure structured data describes what the reader can actually see.

Group findings by template so the fix has a clear owner. Recheck the original URLs after deployment and inspect the missing passages again. Add representative template checks to your release process when content can regress into client-only rendering.

After confirming the content is readable, measure the outcome separately. Improved HTML availability is an input; an observed citation is an outcome. Our guide to missing ChatGPT visibility explains the other checks, and the citation tracking guide explains how to compare results.

What the sources say

• Most AI crawlers measured did not render JavaScript. In a December 2024 analysis by Vercel and MERJ, none of the major AI crawlers measured rendered JavaScript: OpenAI’s OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic’s ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. ChatGPT and Claude crawlers fetched JavaScript files (11.50% and 23.84% of their requests) without executing them; Gemini, on Googlebot’s infrastructure, and AppleBot did render (AI crawlers fetch HTML only, grade C). It is a dated measurement on a few sites, not a permanent fact about every crawler.
• Fetch tools are explicit about it. Anthropic’s web fetch tool fetches server-side and does not render JavaScript (Claude’s fetch behaviour, grade A). A price, stock state, plan table or add-to-cart control inserted client-side is therefore invisible to such fetchers, while browser agents that work from screenshots and the accessibility tree can see it (raw HTML, prices and actions, grade C).
• Google is a different path. Googlebot queues every 200-status page for rendering, and it may wait a few seconds or longer before its headless Chromium runs the scripts; links found only in the rendered HTML are discovered only after rendering. Google recommends server-side or pre-rendering and checking the rendered HTML for all expected content (rendering budget, grade A).
• Interaction-gated content may never be seen. Google Search does not scroll, click or type, so content that loads only on interaction may not be seen (lazy loading and hidden content, grade A).
• A challenge page is not your page. A JavaScript challenge shows a crawler a page that is not your content at all (JavaScript challenges, grade A); see the 403 workflow.

What to move into the initial HTML

Content	Why it matters to a non-rendering fetcher
The headline and the first answer	They are what an answer most often quotes
Body text and captions	Explanations often arrive later in client-side templates
Plan and price tables	A price inserted by script is invisible
Specifications and availability	They decide whether a product fits
Navigation to the next answer	Links found only after rendering are discovered late

A worked example, with invented values

A fictional recipe site renders the page shell on the server and loads the ingredient list and method through a script after the page opens. A fetcher that does not run scripts sees the title and a “Loading…” placeholder. The fix is to render the ingredients and steps into the HTML on the server and leave only the “scale servings” control to the browser. The same URL is then checked with scripts off to confirm that the ingredient list and the steps are present.

Common mistakes

• Judging readability by whether the page looks right in a browser.
• Reading “200 OK” as “readable”.
• Counting words without looking at what the extra words are (cookie banners and navigation inflate both versions).
• Fixing one URL and leaving the template unchanged.
• Treating a headline-and-caption render as a complete answer.

Questions people ask

Do I have to abandon JavaScript?

No. Keep interaction where it helps visitors. Make sure the answer-bearing content exists without it.

Is server-side rendering enough?

It is the most direct fix. Pre-rendering key templates also works; test what the fetcher receives.

Will this get me cited?

It makes citation possible for fetchers that cannot render. Selection is still the engine’s.

Vercatus’s plays include the page fixes that come first, such as content that appears only after JavaScript runs, and a machine check that the fix is live. The supporting hypotheses and their sources are published in the knowledge base.