, a script tag, a few meta tags. The browser downloads the bundle, executes it, calls your API and paints everything you can see. What a visitor reads, and what an AI crawler receives A JavaScript site serves a full page to a browser that runs its code, and an almost empty shell to the AI crawlers that do not run it. A pre-rendered site serves the same complete HTML to both. Characters of body text from one real news page, before and after. An assistant recommends what it could read, so the second number is the one that decides whether you are in the answer. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User and PerplexityBot do none of that. They request the URL, read the bytes that come back, and move on. No JavaScript engine, no waiting for hydration, no API calls on your behalf. Anything absent from the raw HTML response does not exist for them. Googlebot is the exception that confuses everyone. It does render JavaScript, on a delay and on a budget, which is why a client-rendered site can rank acceptably in Google and still be a blank page in every assistant your buyer actually asks. On one bilingual hotel site we took over in June 2026, Google had indexed only 4 of roughly 30 pages before we prerendered it. What is missing from the raw HTML is usually more than people expect: - Text that renders after hydration, including your H1 and first paragraph. - Copy fetched in a useEffect from an API or a database at runtime. - Tab and accordion panels that only mount when the panel is active. - JSON-LD injected by a client-side helper instead of written at build time. - Anything gated behind a loading spinner, a cookie banner or a scroll trigger. ## Test your own site in two minutes Do not take anyone's word for this, including ours. Point curl at your own page with a crawler user agent and count what comes back. First, fetch the page as GPTBot sees it: ```bash curl -s -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://example.com/ -o crawler.html wc -c crawler.html ``` Byte size lies, so measure text, not bytes. On one site we audited, the shell was 3,559 bytes and contained no readable content at all. This counts the characters a model could actually read: ```bash python3 - <<'PY' import re h = open('crawler.html').read() h = re.sub(r'(?is)]+>', ' ', h) print(len(' '.join(t.split())), 'characters of readable text') PY ``` A real content page lands in the thousands. An empty shell lands in the low hundreds, most of it your meta description. Then check that the page states what it is: ```bash grep -Eio ' [^<]*| ]*>[^<]*' crawler.html grep -c 'application/ld+json' crawler.html ``` Next, make sure nothing at the edge is quietly refusing the bots. Bot protection and firewall rules return 403 or a JavaScript challenge that no AI fetcher can solve, and your analytics will never show it: ```bash for ua in "Mozilla/5.0" "GPTBot/1.2" "OAI-SearchBot/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0"; do printf '%-22s ' "$ua" curl -s -o /dev/null -w '%{http_code}\n' -A "$ua" https://example.com/ done ``` Every line should read 200. Finally, sweep the whole site instead of one lucky URL, and sort the worst pages to the top: ```bash curl -s https://example.com/sitemap.xml | grep -oE ' [^<]+' | cut -c6- | while read -r url; do n=$(curl -s -A "OAI-SearchBot/1.0" "$url" | sed -E 's/<[^>]+>/ /g' | tr -s ' ' | wc -c) echo "$n $url" done | sort -n | head -20 ``` ## What zero actually looks like We rebuilt a Spanish premium tableware store in the summer of 2026. When we measured the old site the way you just did, the answer was not "thin content": across the 65 public URLs, the pages served 0 characters of readable text to AI crawlers. Product names, prices, series descriptions, shipping terms, all of it arrived only after JavaScript ran. After the rebuild, the same sweep on 4 August 2026 returned 105,212 characters across the same 65 URLs. Nothing was written for robots. It is the same copy a human reads, moved from runtime into the HTML file. That is the whole trick, and it is the only measurement in this article that we ran ourselves. ## The three ways to fix it ### Static site generation, the right default At build time you render every route to a real HTML file. With Vite and React, vite-react-ssg does this without leaving your stack: you export your routes, provide getStaticPaths for dynamic segments, and the build writes dist/pricing/index.html with the text already in it. The site stays a SPA once hydrated, so navigation still feels instant. Use it when the route list is known at build time: marketing sites, service pages, blogs, documentation, catalogues. Content that changes daily is not a blocker, a scheduled deploy hook rebuilds it. We run the tableware catalogue this way, with a daily refresh. ### Server-side rendering, when the page depends on the request Next.js, Remix or a Vite SSR setup render the HTML per request. You need this when the page genuinely cannot be known in advance: live inventory across thousands of SKUs, search results, anything behind a login. The price is a running server, cache design and more ways to fail at 3am. Do not adopt SSR because it sounds more modern than SSG. Adopt it when a page's content depends on who is asking. ### Crawler prerendering at the edge, the retrofit When a large SPA cannot be restructured now, an edge middleware can serve crawlers a real HTML body while humans keep the app untouched. We use this on a corporate site with roughly 700 news articles: the middleware detects the bot, returns the H1, the article body and the JSON-LD, and everyone else gets the SPA. Two conditions make this safe. The text served to bots must be the same text a human sees, verified by an automated parity test in CI, otherwise it is cloaking. And treat it as a bridge, not a destination, because you now maintain two rendering paths. ### Which one - Route known at build time, content changes daily or less: SSG. - Content depends on the request or the user: SSR. - Large existing SPA you cannot rebuild this quarter: edge prerender, with a parity test. ## What does not fix it - An llms.txt file. Consumer assistants do not request it and Google ignores it. It is useful for coding agents, not for visibility. - JSON-LD on its own. In a 2026 Ahrefs study of 1,885 pages against 4,000 controls, added schema produced no meaningful uplift in AI Overviews, AI Mode or ChatGPT. Visible text is the primary layer, structured data mirrors it. - Writing "for AI": chunking pages into fragments, AI-only markdown copies, invented schema types. Google's 2026 guidance says these do not help, and AI-only pages are a cloaking risk. - nosnippet or max-snippet:0 on pages you want quoted. They remove the eligibility you are trying to earn. ## Verify, then keep verifying Run the sweep again after you deploy, and put a version of it in CI so a refactor cannot silently take the text back out of the HTML. Watch the server logs rather than your analytics: crawlers do not run JavaScript, so GA4 and Plausible never see them, and Vercel or Cloudflare logs filtered by user agent are the only honest record of whether the bots came back. If you would rather have someone measure it and fix it with you, Polargate runs this as part of a paid, fixed-price Discovery Sprint. Either way, run the curl commands first. The number you get is not an opinion. On this page - 01 [An AI crawler asks for your HTML and reads whatever comes back](#an-ai-crawler-asks-for-your-html-and-reads-whatever-comes-back) - 02 [Test your own site in two minutes](#test-your-own-site-in-two-minutes) - 03 [What zero actually looks like](#what-zero-actually-looks-like) - 04 [The three ways to fix it](#the-three-ways-to-fix-it) - 05 [What does not fix it](#what-does-not-fix-it) - 06 [Verify, then keep verifying](#verify-then-keep-verifying) FAQ ## Questions, answered How do I check if ChatGPT can see my website? Fetch the page with a crawler user agent and count text, not bytes. Run curl with -A "GPTBot/1.2" against your URL, save the response, strip the tags and count the characters left. A real content page returns thousands of readable characters. An empty React shell returns a few hundred, most of it your meta description. Repeat it for every URL in your sitemap, because one page proves nothing. Does Google index React single page applications? Yes, but slowly and often partially. Googlebot does render JavaScript, on a delay and within a crawl budget, so client-rendered pages get indexed late or not at all. On a bilingual hotel site Polargate took over in June 2026, Google had indexed only 4 of roughly 30 pages before we prerendered it. AI assistants are stricter still: they never render JavaScript. SSG or SSR: which one do I need for AI visibility? Static generation for almost everything. If the route list is known at build time and the content changes daily or less, SSG writes a real HTML file per page and costs nothing to run. Choose SSR only when the page depends on the request or the user: live stock across thousands of items, search results, logged-in views. Both make your text visible, SSR simply costs more to operate. Does an llms.txt file make my site visible to AI assistants? No. Consumer assistants do not request llms.txt, Google ignores it, and no study shows it lifts citations. It is useful for coding and agent tooling, not for AI search. What decides whether an assistant can quote you is whether your visible text sits in the raw HTML response. Fix the rendering first, then spend time on extras like structured data. Is serving different HTML to AI crawlers considered cloaking? Only if the content differs. Serving a prerendered body to bots while people get the app is acceptable when both receive the same text. Polargate ships this with an automated parity test in CI that compares the crawler response against the rendered page and fails the build when they diverge. Serving bots content that humans cannot see is cloaking, and it can trigger a manual action. ## More reading ai-agents ### How we use AI agents to fix production bugs, and what it costs How Polargate really delivers with AI agents: an agent triages each ticket and proposes a fix as a diff, audit agents refute each other, and a human merges. 2026-08-05 Read more [How we use AI agents to fix production bugs, and what it costs](https://polargate.ai/insights/agentic-delivery-in-production) pricing ### What a website costs in Spain, and why the ranges are so wide What a website really costs in Spain in 2026: the four market tiers, what sits inside each one, what makes a quote climb, and the eight questions to ask before you sign. 2026-09-03 Read more [What a website costs in Spain, and why the ranges are so wide](https://polargate.ai/insights/what-a-website-costs-in-spain) pricing ### What a website costs in the Netherlands in 2026 The four price bands for a business website in the Netherlands, what moves the number, the accessibility deadline, and fixed price versus hourly rate. 2026-09-03 Read more [What a website costs in the Netherlands in 2026](https://polargate.ai/insights/what-a-website-costs-in-the-netherlands) INITIATE ## Start the engine Tell us what you are building in a few short questions. A senior engineer answers in writing within 48 business hours, with a first take on scope, timeline and price. [Start your project](https://polargate.ai/start) · [Talk to us](https://polargate.ai/start#static-brief-heading)