Cache-Control Headers and AI Crawlers

Why Caching Matters for AI SEO

AI crawlers operate under strict time constraints. When GPTBot, OAI-SearchBot, or PerplexityBot requests a page, they need a response quickly. Research shows that sites with First Contentful Paint under 0.4 seconds are cited by ChatGPT three times more often than slow sites.

Your Cache-Control headers determine whether AI crawlers receive a fast cached response from your CDN edge or a slow uncached response from your origin server. The difference can be dramatic: a CDN cache hit typically responds in 20-50ms, while an origin server request may take 500ms-2000ms.

Cache-Control headers do not directly tell AI crawlers what to do — AI bots do not cache content locally the way browsers do. Instead, these headers control your CDN and reverse proxy behavior, which in turn determines the performance AI crawlers experience.

The equation is simple: better caching means faster responses, faster responses mean more successful crawls, and more successful crawls mean better AI visibility.

Cache-Control Directives Explained

public vs private

Cache-Control: public    # CDN and browsers can cache
Cache-Control: private   # Only the end user's browser can cache

For AI SEO: Always use public for content you want AI to access. private prevents CDN caching, meaning every AI crawler request hits your origin server directly.

max-age vs s-maxage

Cache-Control: max-age=3600       # Browser cache: 1 hour
Cache-Control: s-maxage=86400     # CDN/proxy cache: 24 hours

max-age controls browser caching. s-maxage controls shared cache (CDN) duration and overrides max-age for CDN purposes. Use both: a short max-age for browsers and a longer s-maxage for your CDN.

no-cache vs no-store

Cache-Control: no-cache   # Cache it, but revalidate every time
Cache-Control: no-store   # Never cache this content at all

For AI SEO: Avoid no-store on public content — it prevents all caching. no-cache is less harmful but still increases response times because every request requires origin revalidation.

stale-while-revalidate

Cache-Control: stale-while-revalidate=86400

This directive tells the CDN: "If the cache is stale, serve the stale version immediately and refresh in the background." This is the most important directive for AI SEO because it guarantees fast responses at all times. The AI crawler gets a fast response, and the cache gets updated for the next request.

must-revalidate

Cache-Control: must-revalidate

Forces the cache to check with the origin server when content expires. This adds latency at cache expiry. Prefer stale-while-revalidate over must-revalidate for AI-facing content.

Recommended Configurations by Content Type

| Content Type | Cache-Control Header | Rationale | |---|---|---| | Blog posts / articles | public, max-age=3600, s-maxage=86400, stale-while-revalidate=86400 | Long CDN cache, instant responses | | Product pages | public, max-age=1800, s-maxage=14400, stale-while-revalidate=14400 | 4-hour CDN cache for pricing updates | | Homepage | public, max-age=900, s-maxage=3600, stale-while-revalidate=7200 | Hourly CDN refresh, fast for crawlers | | API / dynamic data | private, no-cache | Never cache user-specific responses | | Static assets (CSS/JS/images) | public, max-age=31536000, immutable | Year-long cache with content hashing | | Sitemap XML | public, max-age=3600, s-maxage=14400 | Fresh enough for AI crawlers | | robots.txt | public, max-age=86400 | Daily refresh is sufficient |

The key principle: public content that AI crawlers should read gets long CDN caching with stale-while-revalidate. Private or dynamic content gets no CDN caching.

CDN Caching for AI Bots

Your CDN is the primary beneficiary of Cache-Control headers. When configured correctly, the CDN serves cached HTML to AI crawlers from edge locations closest to the crawler's IP, delivering sub-50ms response times.

How CDN caching works with Cache-Control

1. First request arrives at CDN edge — no cached version exists
2. CDN forwards request to your origin server
3. Origin responds with content and Cache-Control: s-maxage=86400
4. CDN caches the response for 86400 seconds (24 hours)
5. All subsequent requests within 24 hours are served from cache
6. With stale-while-revalidate, even after 24 hours, the next request gets the stale cache instantly while CDN refreshes in the background

CDN-specific considerations

Different CDNs interpret Cache-Control headers slightly differently:

• Cloudflare: Respects s-maxage. Also supports custom Cache Rules that can override origin headers. See our CDN configuration guide.
• Fastly: Strictly follows s-maxage and supports Surrogate-Control for CDN-specific caching separate from browser caching.
• AWS CloudFront: Respects Cache-Control headers from the origin. Minimum TTL, default TTL, and maximum TTL settings can override origin headers.
• Vercel: Respects s-maxage and stale-while-revalidate natively in its edge network.

For more on optimizing server response for AI crawlers, see our server response optimization guide.

The Vary Header Problem

The Vary HTTP header tells caches which request headers to use as cache keys. A common mistake is adding Vary: User-Agent, which creates a separate cache entry for every unique user agent string.

Why Vary: User-Agent is harmful

There are hundreds of unique user agent strings — desktop browsers, mobile browsers, bots, and AI crawlers each have different strings. Vary: User-Agent creates a separate cache entry for each, dramatically reducing cache hit rates. An AI crawler may never get a cache hit because its exact user agent has not been seen before.

Better alternatives

# Good: Vary by cookie (logged-in vs anonymous)
Vary: Cookie

# Good: Vary by Accept-Encoding (compressed vs uncompressed)
Vary: Accept-Encoding

# Bad: Vary by User-Agent (too many variants)
Vary: User-Agent

If you need to serve different content to mobile vs desktop, use client hints (Vary: Sec-CH-UA-Mobile) instead of the full user agent string.

Server Configuration Examples

Nginx

# Blog posts and articles
location /blog/ {
    add_header Cache-Control "public, max-age=3600, s-maxage=86400, stale-while-revalidate=86400" always;
}

# Homepage
location = / {
    add_header Cache-Control "public, max-age=900, s-maxage=3600, stale-while-revalidate=7200" always;
}

# Static assets
location ~* \.(css|js|woff2|webp|svg|png|jpg)$ {
    add_header Cache-Control "public, max-age=31536000, immutable" always;
}

# API endpoints (no CDN caching)
location /api/ {
    add_header Cache-Control "private, no-cache" always;
}

Apache

# Blog posts
<Directory "/var/www/html/blog">
    Header set Cache-Control "public, max-age=3600, s-maxage=86400, stale-while-revalidate=86400"
</Directory>

# Static assets
<FilesMatch "\.(css|js|woff2|webp|svg|png|jpg)$">
    Header set Cache-Control "public, max-age=31536000, immutable"
</FilesMatch>

Next.js / Vercel

// pages/blog/[slug].tsx
export async function getServerSideProps() {
  // Set cache headers
  res.setHeader(
    'Cache-Control',
    'public, s-maxage=86400, stale-while-revalidate=86400'
  );
  // ... fetch data
}

Testing Your Cache Headers

Check headers with curl

curl -I https://yoursite.com/ 2>/dev/null | grep -i cache-control

Test CDN cache hit status

Most CDNs add headers indicating cache status:

# Cloudflare
curl -I https://yoursite.com/ 2>/dev/null | grep -i "cf-cache-status"
# HIT = served from cache, MISS = fetched from origin

# Fastly
curl -I https://yoursite.com/ 2>/dev/null | grep -i "x-cache"

# CloudFront
curl -I https://yoursite.com/ 2>/dev/null | grep -i "x-cache"

Measure response time

curl -s -o /dev/null -w "Time: %{time_total}s\n" https://yoursite.com/

Target under 400ms total response time. If your site consistently exceeds this, caching improvements are your highest-priority optimization for AI SEO.