Being found by an answer engine in 2026 is mostly a technical problem with a content tail, not a content problem with a technical tail. This is the checklist Polargate works through on every site it builds or maintains, with the evidence behind each item.
Almost no AI crawler runs JavaScript
Start here, because this one fact decides everything else. When ChatGPT, Claude or Perplexity fetch your page, they read the raw HTML your server returned. They do not execute your JavaScript. Vercel and MERJ measured more than 500 million GPTBot fetches and recorded zero JavaScript execution. If your site is a React shell that fills an empty div after hydration, those engines see an empty div.
The current picture, crawler by crawler:
- Googlebot: renders JavaScript (evergreen Chromium). Powers Search, AI Overviews and AI Mode.
- Bingbot: renders JavaScript (Edge based), but handles single-page apps worse than Google. Powers Bing, Copilot and part of ChatGPT retrieval.
- Applebot: renders JavaScript (browser based). Powers Siri, Spotlight and Apple Intelligence.
- OAI-SearchBot, which builds ChatGPT's search index: no JavaScript.
- ChatGPT-User, the live fetch fired when a person asks about your page: no JavaScript.
- GPTBot (training): no JavaScript. It downloads your .js files and never runs them.
- Claude-SearchBot and Claude-User: no JavaScript.
- ClaudeBot (training): no JavaScript. Roughly 24% of its requests are JS files it never executes.
- PerplexityBot and Perplexity-User: no JavaScript. Perplexity is the most freshness-sensitive engine of the group.
- Amazonbot, Meta-WebIndexer, Bytespider: no documented rendering.
- Agentic browsers (ChatGPT Atlas, Perplexity Comet): real Chromium, so they do render. They were about 70% of agentic traffic in April 2026, Comet at 48% and Atlas at 21%.
So the first checklist item is not a meta tag, it is static HTML for every public route. On a news site Polargate rebuilt, the crawler was previously served 14 characters ("Redirecting...") and now gets 11,421 characters of article text; the served document went from a 3,559-byte shell to a real body. On a hotel site we maintain, Search Console showed only 4 of about 30 pages indexed before pre-rendering.
Verify it the way a crawler would, not in DevTools: fetch the URL with the bot's user agent and grep the response for your H1 and body text. If it is not in view-source, it does not exist.
Let the right bots in, and check the edge as well as robots.txt
Every vendor now splits three kinds of agent: training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended), search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). OpenAI states plainly that a site disallowing OAI-SearchBot will not be shown in ChatGPT search answers.
- Allow the search and answer agents explicitly, by name, in robots.txt.
- Decide on training crawlers as a separate business question. Blocking them does not remove you from citations, but it does keep you out of model weights.
- Never block /assets/, CSS or JS: that breaks rendering for Googlebot and Applebot, the two crawlers that actually render.
- Audit the edge. A firewall preset that denies "AI bots", or a bot-protection challenge, returns 403 to fetchers that cannot solve challenges, and nothing in robots.txt will save you.
- Fix 404s. About 35% of GPTBot and ClaudeBot fetches hit missing URLs, so keep old paths redirected.
- Google-Extended and Applebot-Extended are usage tokens, not crawlers. Disallowing them has no effect on search or AI Overviews.
These fail quietly. In one audit Polargate ran on a client site, a robots.txt group-inheritance mistake was silently narrowing which agents were allowed, alongside fifteen other confirmed bugs.
Put the answer first
ChatGPT's instant mode sees the title, the H1 and roughly the first 200 characters after it. Perplexity favours the first 100 words, reads around ten pages per query and cites three or four. Across ChatGPT citations, 44.2% come from the first 30% of a page. A hero video and a tagline-only H1 spend that budget on nothing.
The structure that gets quoted
- One H1 that names the entity and the claim, followed immediately by a plain-text answer of one or two sentences.
- H2s phrased as the question a buyer would type. Question-shaped content is cited about twice as often, 78.4% of question-tied citations come from headings, and a heading closely matching the query is cited 41% of the time against 29% for a weak match.
- A self-contained answer of 40 to 70 words under each heading, in definitional language ("X is", "X refers to").
- Real HTML tables for comparisons and pricing. A table rendered from JSON at runtime, or exported as an image, contributes nothing.
- Concrete numbers with units and dates, attributed quotes and links to primary sources. The Princeton and Georgia Tech GEO study of 10,000 queries measured quotation addition at +41% visibility, statistics at +33% and source citation at +28%, with the largest gains going to lower-ranked sites.
- Natural-language slugs. Ahrefs found an 89.8% citation rate for them against 81.1% for opaque ones.
- Focused pages instead of one ultimate guide. On 815,000 query-page pairs, pages covering 26% to 50% of an engine's fan-out sub-queries in depth beat pages covering all of them shallowly.
Freshness is the signal you control
AI citations are on average 25.7% fresher than organic results, and of the "best X" listicles ChatGPT cited, 79.1% had been updated in the current year. A visible year in the title correlated with about 30% higher citation rates in a study of 22.7 million citations.
- Show a visible updated date, mirror it in dateModified, and use it as the sitemap lastmod.
- Only change it when the text actually changed. Stamping every page with the build date destroys the signal and trains crawlers to ignore your sitemap.
- Ping IndexNow on production deploys. Bing recrawls submitted URLs within minutes, and the Bing index feeds Copilot and part of ChatGPT retrieval.
- Verify the site in Bing Webmaster Tools, not only Search Console.
What does not work, despite the advice you will read
- llms.txt. Consumer crawlers do not request it, Google Search ignores it, and no study shows a citation lift. Useful for coding and agent tooling, not a visibility lever.
- Schema as the lever. Ahrefs tracked 1,885 pages that added structured data: AI Overviews moved -4.6%, AI Mode +2.4%, ChatGPT +2.2%, all inside noise. Keep schema accurate and minimal for indexing and rich results, and never mark up facts that are not visible on the page.
- nosnippet and max-snippet on pages you want quoted. They remove the eligibility you are trying to win.
- Chunking pages into fragments "for LLMs", or publishing AI-only duplicate pages. Google says neither is required, and cloaked parallel content is a policy risk.
Measure it where the traffic actually shows up
AI bots do not run JavaScript, so client-side analytics never sees them, and between 35% and 70% of AI-referred human visits arrive with no referrer at all.
- Server logs (a Vercel log drain works) filtered by user agent: who fetched what, and with which status code. Alert on 403 and 429 to any answer crawler.
- A GA4 custom channel group for AI assistants. GA4 has assigned the ai-assistant medium since May 2026, but Perplexity still lands in Referral.
- Search Console's Generative AI report, available on all sites since 31 August 2026, and Bing's AI Performance report, the only first-party dashboard that counts AI citations per URL.
- A fixed panel of 25 to 40 buyer prompts, run monthly across engines, recording mention, position and cited URL. Citations and brand mentions diverge, so track both.
None of this is a trick. It is the ordinary discipline of shipping real HTML, saying true things in a readable order, and checking that the machines can reach them.
