JavaScript Bundle Size and AI Crawling

JavaScript and AI Crawler Limitations

The fundamental problem: most AI crawlers operate like simplified browsers that fetch HTML but do not execute JavaScript. When your page relies on JavaScript to render content (typical for React, Vue, or Angular single-page applications), these crawlers see the empty shell — not your content.

Which AI crawlers execute JavaScript?

| AI Crawler | JavaScript Execution | Content Access | |---|---|---| | Googlebot | Yes (via WRS, with delays) | Full content after rendering | | OAI-SearchBot | No / Very limited | HTML-only content | | PerplexityBot | No / Very limited | HTML-only content | | ClaudeBot | No | HTML-only content | | ChatGPT-User | Partial (via Bing infrastructure) | Varies by page complexity | | Bingbot | Yes (with delays) | Full content after rendering |

The pattern is clear: only Googlebot and Bingbot reliably execute JavaScript. The rest — including the crawlers for ChatGPT search, Perplexity, and Claude — primarily read the raw HTML. This makes JavaScript bundle size a two-dimensional problem for AI SEO:

1. Content access: If content requires JS to render, most AI crawlers cannot see it at all
2. Performance: Even when crawlers do not execute JS, large bundles slow page delivery and hurt Core Web Vitals

For a deep dive into JavaScript rendering challenges, see JavaScript rendering and AI.

Bundle Size Targets for AI SEO

Recommended limits

| Metric | Target | Impact | |---|---|---| | Initial JS (compressed) | Under 200KB | Fast parse, minimal main thread blocking | | Total JS (all chunks) | Under 500KB | Reasonable total footprint | | Third-party JS | Under 100KB | Controlled external impact | | Critical-path JS | Under 50KB | Fastest possible first render |

Why these numbers matter

JavaScript has a unique cost compared to other resources. A 200KB image and a 200KB JavaScript bundle have very different performance impacts:

• 200KB image: Downloaded, decoded, painted. Cost is primarily network time.
• 200KB JavaScript: Downloaded, parsed, compiled, executed. Cost is network time PLUS CPU time. On a mid-range mobile device, parsing and executing 200KB of JavaScript takes 200-400ms.

This CPU cost directly affects Core Web Vitals:

• LCP: JS execution blocks the main thread, delaying render of content elements
• INP: JS tasks create input delay, making interactions feel sluggish
• TBT (Total Blocking Time): Long JS tasks contribute directly to TBT, Lighthouse's proxy for INP

How to check your current bundle size

Use the Chrome DevTools Coverage tool (Ctrl+Shift+P > "Coverage") to see how much of your JavaScript is actually used during a page load. Most sites discover that 40-70% of their JavaScript is unused on any given page.

Tree Shaking: Remove Unused Code

Tree shaking eliminates dead code from your bundles at build time. When you import one function from a large library, tree shaking ensures only that function and its dependencies are bundled — not the entire library.

Requirements for effective tree shaking

1. Use ES modules (import/export) — CommonJS (require) cannot be statically analyzed for tree shaking
2. Set sideEffects: false in package.json for your own code
3. Use a modern bundler — Webpack 5, Rollup, Vite, and esbuild all support tree shaking

Common tree-shaking wins

// BAD: imports entire library (~70KB)
import _ from 'lodash';
const result = _.debounce(fn, 300);

// GOOD: imports only the function (~2KB)
import debounce from 'lodash/debounce';
const result = debounce(fn, 300);

// BEST: use native alternative (0KB added)
function debounce(fn, ms) {
  let timer;
  return (...args) => {
    clearTimeout(timer);
    timer = setTimeout(() => fn(...args), ms);
  };
}

Audit with bundle analyzers

Use visualization tools to identify the largest modules in your bundle:

• webpack-bundle-analyzer — Interactive treemap showing module sizes
• source-map-explorer — Analyze bundle composition from source maps
• bundlephobia.com — Check the size cost of npm packages before installing

Code Splitting Strategies

Code splitting divides your application into smaller chunks that load on demand.

Route-based splitting

Load JavaScript only for the current page. Each page/route gets its own chunk:

// Next.js: automatic route-based splitting
// pages/about.js only loads when /about is visited

// React Router with lazy loading
const About = React.lazy(() => import('./pages/About'));
const Blog = React.lazy(() => import('./pages/Blog'));

AI SEO impact: Reduces the JavaScript loaded for each page, speeding up initial load. AI crawlers fetching a specific page only trigger that page's chunk — not the entire application.

Component-based splitting

Load heavy components only when they are needed:

// Load chart library only when user scrolls to the chart section
const Chart = React.lazy(() => import('./components/Chart'));

function ArticlePage() {
  return (
    
      <p>Article content here (visible immediately)</p>
      <Suspense fallback={<div>Loading chart...</div>}>
        <Chart data={chartData} />
      </Suspense>
    
  );
}

Vendor chunk splitting

Separate third-party libraries into their own chunk that can be cached independently:

// webpack.config.js
optimization: {
  splitChunks: {
    cacheGroups: {
      vendor: {
        test: /[\\/]node_modules[\\/]/,
        name: 'vendors',
        chunks: 'all',
      },
    },
  },
}

This allows the vendor chunk to be cached in the browser (and CDN) even when your application code changes.

Third-Party Script Audit

Third-party scripts often account for 50-70% of total JavaScript on a page:

Common offenders

| Script Type | Typical Size | Alternative | |---|---|---| | Google Analytics (GA4) | 45-90KB | gtag with minimal config, or server-side tracking | | Chat widgets (Intercom, Drift) | 200-500KB | Load on interaction only | | Social media embeds | 100-300KB each | Static screenshots with links | | A/B testing (Optimizely) | 50-200KB | Server-side A/B testing | | Cookie consent banners | 50-150KB | Lightweight alternatives (Klaro, CookieConsent) |

Loading strategies for third-party scripts

<script src="/critical.js"></script>


<script src="/analytics.js" defer></script>


<script>
document.addEventListener('click', () => {
  const script = document.createElement('script');
  script.src = '/chat-widget.js';
  document.body.appendChild(script);
}, { once: true });
</script>

The "facade" pattern

For heavy embeds (YouTube, social media, chat widgets), show a static placeholder image. Load the actual embed only when the user interacts:

<div class="youtube-facade" onclick="loadYouTube(this)" data-id="VIDEO_ID">
  <img src="/thumbnails/video.webp" alt="Video title" loading="lazy">
  <button aria-label="Play video">Play</button>
</div>

This eliminates hundreds of kilobytes of JavaScript from the initial load.

SSR as the Definitive Solution

For AI SEO, server-side rendering solves the JavaScript-content access problem completely. When you use SSR, the server sends fully rendered HTML to both users and AI crawlers. No JavaScript execution is needed to see the content.

SSR options by framework

| Framework | SSR Method | AI Benefit | |---|---|---| | Next.js | getServerSideProps, App Router Server Components | Full content in HTML, automatic code splitting | | Nuxt | Universal rendering | Vue content server-rendered | | SvelteKit | Server-side by default | Minimal JS footprint | | Astro | Static-first, opt-in JS | Zero JS by default, add only where needed | | Remix | Server-first routing | Nested layouts with SSR |

The "islands" architecture

Frameworks like Astro use an "islands" architecture where the page is mostly static HTML with small "islands" of JavaScript interactivity. This gives you:

• Zero JavaScript by default — Content pages ship no JS
• Selective hydration — Only interactive components (forms, accordions, carousels) include JavaScript
• Optimal for AI crawlers — 100% of content is in the HTML response

For content-focused sites like blogs, knowledge bases, and documentation, Astro's islands approach is ideal for AI SEO.

Measuring and Monitoring Bundle Size

Build-time measurement

Add bundle size tracking to your CI/CD pipeline:

• bundlesize — Fails the build if bundles exceed configured limits
• size-limit — Tracks bundle size with performance budget enforcement
• Lighthouse CI — Monitors performance metrics including JS execution time

Runtime measurement

• Chrome DevTools Network panel — Filter by JS to see total JavaScript transferred
• Chrome DevTools Coverage — Shows percentage of JavaScript actually used
• Performance panel — Shows JS parsing and execution time in the timeline

Budgets and alerts

Set up performance budgets that alert when JavaScript exceeds limits:

{
  "budgets": [
    {
      "resourceType": "script",
      "budget": 200,
      "unit": "kb"
    },
    {
      "resourceType": "third-party",
      "budget": 100,
      "unit": "kb"
    }
  ]
}