The AI crawlers are not running your JavaScript

In September we checked whether the AI crawlers could reach this site. GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot each returned 200 on every page we tested.

That felt like a clean result for about a day, until the obvious objection landed. A 200 means a file was delivered. It says nothing about whether anything inside it was read — and for a site built the way most sites are now built, the interesting content is not in the file at all.

Which crawlers actually render JavaScript?#

Vercel and MERJ watched live crawler traffic on production sites and found that none of the major AI crawlers execute JavaScript. OpenAI's, Anthropic's, Perplexity's, Meta's and ByteDance's crawlers all fetch .js files — the request lands in your logs, the bandwidth is spent — and then nothing runs them.

The fetching is not incidental, either. In that sample ChatGPT's crawler spent 11.50% of its requests on JavaScript files and Claude's 23.84%, which is a meaningful share of a crawl budget going to files whose contents are never read.

There is one exception, and it matters more than the rule.

Crawler	Renders JavaScript	Why
Googlebot	Yes	Has rendered for years
Google Gemini	Yes	Rides Googlebot's infrastructure
Applebot	Yes	Renders for Siri and Spotlight
GPTBot, OAI-SearchBot	No	Fetches scripts, does not execute
ClaudeBot	No	Same
PerplexityBot	No	Same
Bytespider, Meta-ExternalAgent	No	Same

Read that column twice if your site is a single-page application. The same page can be completely visible to Google's AI surfaces and completely blank to ChatGPT and Perplexity, with no error anywhere to tell you.

One caveat on the evidence, because it cuts both ways. That study ran in December 2024, and crawler behaviour is not a law of nature — rendering is expensive, but it is the kind of expense a well-funded lab can decide to absorb. Treat the table as the current best measurement rather than a permanent fact, and re-run the check on your own pages rather than trusting a table in a blog post, including this one.

What does "doesn't render" mean for a page that works fine in my browser?#

It means the crawler sees your HTML file before the browser has done any work on it.

For a server-rendered page that is nearly everything: the headings, the paragraphs, the tables, the links. For a client-rendered page it can be almost nothing — a <div id="root">, a few script tags, and a loading state. Your browser turns that into a page in a few hundred milliseconds, which is exactly why the problem is so easy to miss. Everyone who looks at the site has a browser.

The failure is silent in both directions. Nothing in your analytics shows it, because the crawler never ran the analytics script either. Nothing in your server logs shows it, because the request succeeded. The only symptom is an absence: you are not quoted, and you cannot tell why.

How do I see what a crawler sees?#

Fetch the page without a browser and read what comes back.

curl -s https://example.com/your-page | head -100

That is approximately what a non-rendering crawler gets. If your headline, your opening paragraph and your main links are in there, you are fine. If what comes back is a shell and a script tag, that is the whole of what GPTBot has to work with.

Two smaller checks are worth running at the same time. View the page with JavaScript disabled in your browser, which catches content that loads on scroll or after an interaction. And check that the crawlers are not blocked outright — plenty of sites block them through a CDN rule nobody remembers enabling, and no amount of HTML fixes that.

Why is "we rank fine on Google" no longer reassurance?#

Because Google is the one engine where rendering is not the question.

Googlebot has rendered JavaScript for years, and Gemini inherits that. So a client-rendered site can rank perfectly well, pass every SEO audit, and still be invisible to the engines that do not render. The check that would have caught it is one most tools do not run, because for a decade there was no reason to.

This compounds with something we wrote about in our piece on GEO and SEO: the engines do not cite the same sources as each other anyway. Bing Chat and Perplexity overlap on roughly 26% of cited domains. If your content is also invisible to the non-rendering half of them, the gap is not a ranking problem. It is an absence of input.

What this changed on our own site#

Two things, and one of them we found by getting it wrong.

When we built this blog we chose plain Markdown rendered at build time rather than MDX, specifically so that no part of a post would depend on JavaScript running. A chart that only exists after hydration is a chart that cannot be quoted. Anything we publish as a figure gets rendered into the HTML as a build step.

The second came from a different direction. In September our visitor analytics quietly recorded nothing for about two weeks. The cause turned out to be that this site is served as static assets from a Cloudflare Worker, and the analytics script was supposed to be injected automatically by a step that only runs on proxied origin responses — which ours never are. The script was simply absent from the HTML, and every dashboard still looked plausible.

That is the same lesson in a different costume. What is in the delivered HTML is what exists. Everything else is a thing you assume is happening, and the way you find out is by fetching the page and reading it.

Is there something between a rewrite and doing nothing?#

Yes, and for most sites it is the sensible answer.

Prerendering sits in the middle. The page stays a client-rendered application for people, and a build step or an edge service produces a plain HTML version for anything that does not run scripts. Frameworks call it different things — static generation, server-side rendering, a prerender step — and the practical difference between them matters far less than whether any of them is switched on.

Two warnings about the middle path. Serving different HTML to crawlers than to people, based on detecting who is asking, is a technique with a long history of going wrong and a short history of being recommended; the version that is safe is producing the same HTML for everyone and letting browsers enhance it. And a prerender service is another dependency that can fail quietly — the failure mode is identical to the one this post is about, which is a page that looks fine to you and is empty to everything else.

What should you actually check?#

Four things, in the order they cost you.

Whether the crawlers can reach you at all. A CDN rule or a robots.txt line can make every other item on this list irrelevant. One request per crawler answers it.

What is in the HTML before JavaScript runs. The curl above. If the answer is a shell, that is your largest single problem and no amount of writing fixes it.

Whether your key pages are server-rendered. Most frameworks can do this; many projects simply never switched it on. The pages that matter are the ones you would want quoted — your service pages and your best writing, not your dashboard.

Whether the fix is even worth it for you. If your business does not depend on being found through an AI answer, a client-rendered app is a perfectly reasonable thing to own — a booking tool used by people who already have the link does not need to be quotable. The point is to know which situation you are in rather than to discover it next year.

Worth saying plainly: none of this is a reason to rebuild a working site. The pages that need to arrive as HTML are usually a small, identifiable set — the service pages, the writing, the thing someone would ask an assistant about. Most frameworks will render those at build time with a configuration change rather than a rewrite.

Frequently asked questions#

Does this mean I should not use React or Vue?#

No. It means the pages you want quoted should arrive as HTML. Every major framework supports server rendering or static generation; the question is whether your project uses it for the pages that matter.

If a crawler downloads my JavaScript file, doesn't it read the content inside?#

It downloads the file and counts the request. Executing it is a separate and much more expensive step, and the Vercel and MERJ research found the major AI crawlers do not take it. Content that only exists after execution is not seen.

My site ranks well on Google. Am I affected?#

Possibly, and Google will not tell you. Googlebot renders JavaScript and Gemini inherits that, so Google is the one place this problem does not show up. The engines that do not render are the ones to test separately.

How is this different from normal SEO advice about JavaScript?#

The advice is the same; the stakes changed. For years the fallback was that Googlebot would render your page eventually. For a crawler that never renders, there is no eventually.

We use Next.js or Nuxt. Are we fine by default?#

Usually yes for pages that are statically generated or server-rendered, and not for pages that fetch their content in the browser after loading. Both are normal in the same project, which is why the curl check is per-page rather than per-framework.

Is there a tool that checks this for me?#

Several, but curl answers the main question in one line and has no incentive to sell you anything. Reach for a tool when you need it across hundreds of pages rather than a handful.

Does server rendering help with Google's AI Overviews too?#

It does not change whether Google can see the page, since Googlebot renders either way. It does reduce the time between publishing and being crawlable, which matters more for news than for evergreen writing.

Does an llms.txt file solve this?#

No. Google's guidance on its AI features says you "don't need to create new machine readable files, AI text files, or markup" to appear in them. And a file listing your content does nothing about content that is missing from the page a crawler actually fetched. The HTML is the interface.

How often should I re-check?#

Once a quarter is plenty for most sites, and after any framework or hosting change. The second one is the real trigger — our own analytics broke on a delivery change, not on anything anyone wrote.

---

Published 13 October 2026. Sources: research by Vercel and MERJ on AI crawler behaviour, Google Search Central's guidance on AI features, and our own crawler-reachability checks on this domain in September 2026.