Single-Page Applications and AI Crawling

Why SPAs Fail with AI Crawlers

A single-page application loads once and dynamically updates content by manipulating the DOM through JavaScript. From a user's perspective, the experience is fast and fluid — pages transition instantly without full reloads. From an AI crawler's perspective, the experience is empty.

Here is what happens when GPTBot visits a typical React SPA:

1. GPTBot sends GET request to https://yoursite.com/products
2. Server returns the same index.html for every URL (SPA routing)
3. The HTML contains <div id="root"></div> and a <script> tag
4. GPTBot reads the HTML — sees no content
5. GPTBot cannot execute the JavaScript bundle
6. GPTBot records: this page has no content

Your product catalog, blog posts, FAQ pages, about page — all invisible. The JavaScript that would render them remains unexecuted.

For a broader look at JavaScript rendering issues, see our JavaScript rendering and AI guide. For the SSR vs CSR comparison, see SSR vs Client-Side Rendering.

The Five SPA Problems for AI

Problem 1: Empty initial HTML

The fundamental issue. SPA servers return the same minimal HTML shell for every URL. AI crawlers receive no content in the response body.

Problem 2: Hash-based routing

SPAs using hash routing (e.g., yoursite.com/#/products) are completely undetectable by crawlers. The hash fragment (#/products) is never sent to the server — the server only receives a request for yoursite.com/. Crawlers cannot discover or distinguish between hash-routed pages.

Problem 3: API-dependent content

SPA content is typically fetched from APIs after the page loads. AI crawlers do not execute JavaScript and therefore never trigger these API calls. The data that populates your pages exists only in API responses that crawlers never request.

Problem 4: Lazy-loaded sections

Content that appears on scroll, on click, or on tab selection is invisible to crawlers. If your pricing table only loads when users click a "Pricing" tab, AI crawlers never see your prices.

Problem 5: Client-side structured data

If your JSON-LD schema markup is injected by JavaScript (common in React Helmet or Vue Meta), AI crawlers see no structured data. Schema markup must be in the initial HTML response.

Solution 1: Migrate to SSR Framework

The most comprehensive solution. Replace your SPA's client-side-only rendering with a framework that supports server-side rendering.

React SPA to Next.js

1. Create a new Next.js project
2. Move your React components into the app/ directory
3. Convert client-side data fetching (useEffect + fetch) to Server Components or getServerSideProps
4. Ensure JSON-LD schema is rendered server-side
5. Update routing from React Router to Next.js file-based routing

Vue SPA to Nuxt

1. Create a new Nuxt project
2. Move Vue components to the Nuxt directory structure
3. Replace Vue Router with Nuxt's file-based routing
4. Convert onMounted data fetching to useAsyncData or useFetch

Angular SPA to Angular Universal

1. Add Angular Universal to your existing Angular project: ng add @angular/ssr
2. Configure server-side rendering in your Angular module
3. Test that content renders server-side

Incremental migration

You do not need to migrate everything at once. Start with your highest-value content pages (homepage, product pages, blog) and migrate them to SSR first. Lower-priority application pages can remain CSR.

Solution 2: Hybrid Architecture

The pragmatic approach for teams that cannot fully migrate from SPA architecture. Run two rendering modes on the same domain:

Marketing pages (SSR/SSG)

Your homepage, about page, product descriptions, blog, pricing, FAQ, and any content you want AI to discover and cite. These render server-side and deliver complete HTML.

Application pages (CSR)

Dashboard, settings, user profiles, admin panels, and interactive features. These remain client-side rendered because AI crawlers do not need to see them.

Implementation patterns

Subdomain split: www.yoursite.com (SSR marketing site) + app.yoursite.com (SPA application)

Path-based split: /blog/*, /products/*, /about served by Next.js/Nuxt SSR; /app/*, /dashboard/* served by your SPA

Framework hybrid: Next.js app directory with Server Components for content pages and Client Components for interactive features — all in one codebase

Solution 3: Pre-rendering

Pre-rendering generates static HTML for each SPA route at build time. The output is a folder of HTML files that any web server can host.

Tools

• react-snap — Crawls your SPA in a headless browser and saves the rendered HTML
• prerender-spa-plugin (Webpack) — Renders specified routes during build
• vite-plugin-ssr — SSR/pre-rendering for Vite-based projects

Limitations

• Only works for pages known at build time (not dynamic user-generated pages)
• Must rebuild when content changes
• Not suitable for pages with real-time data (pricing that changes hourly)

Best for

Brochure sites, documentation, blogs — any content that changes infrequently and has a known set of URLs.

Solution 4: Dynamic Rendering

Dynamic rendering detects bot user agents and serves them pre-rendered HTML while serving the normal SPA to human visitors.

How it works

1. Request arrives at your server or CDN edge
2. User agent is checked against known bot patterns
3. If bot: Request is routed to a pre-rendering service (Rendertron, Prerender.io)
4. If human: Normal SPA is served

Cautions

• Google considers this acceptable but not ideal
• Content served to bots must match content served to humans (otherwise it is cloaking)
• Adds infrastructure complexity
• May not keep up with rapid content changes
• Use only as a bridge while migrating to SSR

Routing Considerations

Replace hash routing immediately

If your SPA uses hash-based routing (/#/page), this is your highest priority fix. Switch to history mode routing (/page):

React Router:

// Before (hash): <HashRouter>
// After (history): <BrowserRouter>

Vue Router:

// Before
const router = createRouter({ history: createWebHashHistory() })
// After
const router = createRouter({ history: createWebHistory() })

Configure server-side fallback

History mode routing requires server configuration to return index.html for all routes (so client-side routing can take over):

# Nginx
location / {
    try_files $uri $uri/ /index.html;
}

This at least ensures crawlers receive the SPA shell for any URL, though the content problem remains until you add SSR.

Testing SPA AI Visibility

Disable JavaScript test

Open your SPA in Chrome, press F12, go to Settings (gear icon), and check "Disable JavaScript." Reload the page. What you see is what AI crawlers see. If the page is blank, you have a problem.

curl test

curl -s https://yoursite.com/products | wc -w

If the word count is under 50, the page has no meaningful content without JavaScript.

View Source vs Inspect

Right-click your page and choose "View Page Source" (shows raw HTML from server). Compare with "Inspect Element" (shows DOM after JavaScript execution). If View Source shows content but Inspect shows more, you have some server-rendered content. If View Source shows only a script tag, nothing is server-rendered.

For comprehensive verification methods, see our verifying AI crawler access guide.