Caching Strategies for AI Bots

Caching Layers and AI Crawlers

A modern web application has multiple caching layers. Understanding which layers benefit AI crawlers helps you prioritize optimization efforts:

| Cache Layer | Benefits AI Crawlers? | Benefits Users? | Priority for AI SEO | |---|---|---|---| | CDN edge cache | Yes — serves instant responses | Yes | Highest | | Reverse proxy (Varnish, Nginx) | Yes — fast origin responses | Yes | High | | Application cache (Redis) | Yes — faster dynamic pages | Yes | Medium | | Browser cache | No — crawlers start fresh | Yes | Low for AI | | Service worker cache | No — crawlers don't install SW | Yes | Low for AI |

The critical insight: AI crawlers only benefit from server-side and CDN caching. They do not maintain a local cache between visits, and they do not execute service workers. Every optimization dollar spent on AI crawler performance should go to CDN edge caching and server-side caching first.

For the full picture of how speed affects AI crawling, see page speed for AI crawlers and server response optimization.

Cache-Control Headers for AI SEO

The Cache-Control HTTP header is your primary tool for controlling caching behavior across all layers.

Recommended header for AI SEO

Cache-Control: public, max-age=0, s-maxage=3600, stale-while-revalidate=86400

Breaking this down:

• public — Allows shared caches (CDN, proxies) to store the response
• max-age=0 — Tells browsers to always revalidate with the server (ensures users see fresh content)
• s-maxage=3600 — Tells CDN/shared caches to store the response for 1 hour (this is the key directive for AI bots)
• stale-while-revalidate=86400 — Allows CDN to serve stale content for up to 24 hours while fetching a fresh copy in the background

Why this combination works

AI crawlers hit the CDN edge, which serves the cached response instantly (the s-maxage part). If the cache has expired, the CDN serves the stale version immediately (the stale-while-revalidate part) while fetching a fresh copy from your origin server. The AI crawler never waits for your origin server to respond.

Human users get fresh content because max-age=0 causes their browsers to check with the CDN on every visit. The CDN responds with either the cached version (fast) or validates that the content has not changed (304 Not Modified, also fast).

Headers for different content types

| Content Type | Cache-Control | Rationale | |---|---|---| | HTML pages | public, max-age=0, s-maxage=3600, stale-while-revalidate=86400 | CDN-cached, always fresh for users | | CSS/JS (hashed) | public, max-age=31536000, immutable | Cache forever — filename changes on update | | Images (hashed) | public, max-age=31536000, immutable | Same as CSS/JS | | API responses | private, no-store | Dynamic, user-specific data | | robots.txt | public, max-age=86400 | Changes rarely, cache 24 hours |

CDN Edge Caching Configuration

CDN edge caching is the most impactful caching layer for AI bots. Here is how to configure it properly:

Enable HTML caching

Most CDNs only cache static assets by default. You must explicitly enable HTML page caching:

Cloudflare:

• Cache Rules: Set "Edge TTL" to "Override" with 3600 seconds for HTML responses
• Exclude admin pages, API endpoints, and authenticated routes

Vercel:

• Set s-maxage in your response headers or next.config.js
• ISR automatically handles edge caching for static pages

AWS CloudFront:

• Create a cache behavior for *.html and / with custom TTL settings
• Use origin response timeout to handle slow origin responses gracefully

Cache warming

After a deployment or cache purge, your CDN cache is empty and the first visitor to each page experiences a slow origin fetch. Cache warming pre-populates the CDN cache by requesting all important pages:

#!/bin/bash
# Warm CDN cache after deployment
while IFS= read -r url; do
  curl -s -o /dev/null -w "%{url_effective} %{http_code} %{time_total}s\n" "$url"
done < important-urls.txt

This ensures AI crawlers always hit a warm cache, never a cold origin fetch.

Geographic distribution

If your target audience and AI crawlers are in different regions, ensure your CDN has strong presence in both:

• AI crawlers: US East (Virginia), US West — where most AI data centers operate
• Your audience: Your primary market region
• Content freshness: Use tiered caching so inner CDN layers share cached content with outer layers

Server-Side Caching

When CDN cache misses, your server must respond quickly. Server-side caching reduces origin response time:

Full-page caching

Store the complete rendered HTML in memory and serve it without executing application code:

Nginx FastCGI cache:

fastcgi_cache_path /tmp/nginx-cache levels=1:2 keys_zone=CACHE:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

location ~ \.php$ {
    fastcgi_cache CACHE;
    fastcgi_cache_valid 200 60m;
    fastcgi_cache_use_stale error timeout updating;
}

Varnish VCL:

sub vcl_backend_response {
    set beresp.ttl = 1h;
    set beresp.grace = 24h; // Serve stale for 24h if backend is down
}

Object caching with Redis

For pages that cannot be fully cached (personalized content, logged-in users):

# Cache expensive database queries
import redis
cache = redis.Redis()

def get_products(category):
    cache_key = f"products:{category}"
    cached = cache.get(cache_key)
    if cached:
        return json.loads(cached)

    products = db.query("SELECT * FROM products WHERE category = ?", category)
    cache.setex(cache_key, 3600, json.dumps(products))  # Cache 1 hour
    return products

HTTP/2 server push (deprecated context)

HTTP/2 server push was once recommended for proactively sending resources to clients. As of 2026, Chrome has removed server push support and no AI crawlers use it. Focus on preload hints in HTML instead.

Browser Cache and Service Workers

While these layers do not benefit AI crawlers, they are important for overall performance and user experience:

Browser caching for static assets

Use long cache durations with content-hashed filenames:

# Static assets with content hashes
/assets/style.a1b2c3.css  → Cache-Control: public, max-age=31536000, immutable
/assets/app.d4e5f6.js     → Cache-Control: public, max-age=31536000, immutable

# Non-hashed assets
/favicon.ico              → Cache-Control: public, max-age=86400
/robots.txt               → Cache-Control: public, max-age=86400

The immutable directive tells browsers the file will never change at this URL — do not even bother revalidating. When the file changes, the filename hash changes, creating a new URL.

Service workers for offline and instant repeats

Service workers run in the browser and can intercept network requests to serve cached responses instantly:

// Simple cache-first service worker
self.addEventListener('fetch', event => {
  event.respondWith(
    caches.match(event.request).then(response => {
      return response || fetch(event.request).then(fetchResponse => {
        const cache = caches.open('v1');
        cache.put(event.request, fetchResponse.clone());
        return fetchResponse;
      });
    })
  );
});

AI SEO relevance: Service workers improve metrics like repeat-visit speed and offline availability, which contribute to better user engagement. Better engagement signals can indirectly support Google rankings.

Cache Invalidation Strategies

Caching is only useful if you can invalidate stale content when it changes:

Tag-based purging

Assign cache tags to related content groups:

Surrogate-Key: blog post-123 category-tech author-john

When a blog post is updated, purge only the post-123 tag. When a category page changes, purge the category-tech tag. This is far more precise than purging by URL.

CMS-triggered invalidation

Integrate your CMS with the CDN purge API:

WordPress + Cloudflare:

// Purge specific URL when post is updated
function purge_cloudflare_cache($post_id) {
    $url = get_permalink($post_id);
    // Call Cloudflare API to purge this URL
    wp_remote_post('https://api.cloudflare.com/client/v4/zones/ZONE_ID/purge_cache', [
        'headers' => ['Authorization' => 'Bearer API_TOKEN'],
        'body' => json_encode(['files' => [$url]])
    ]);
}
add_action('save_post', 'purge_cloudflare_cache');

Avoiding full cache purges

Never purge your entire CDN cache unless absolutely necessary. A full purge means every subsequent request hits your origin server until the cache is repopulated. During this window, AI crawlers experience slow TTFB, and your origin server may be overwhelmed by the sudden traffic spike.

Cache versioning

For major site updates, use cache versioning instead of purging. Deploy the new version with updated asset hashes and HTML content. The CDN gradually replaces old cached content as items expire naturally or are requested.

Monitoring Cache Performance

Cache hit ratio

Track your CDN cache hit ratio — the percentage of requests served from cache versus forwarded to origin:

• Target: Above 80% for HTML, above 95% for static assets
• Where to check: CDN analytics dashboard (Cloudflare Analytics, CloudFront CloudWatch)

TTFB comparison

Compare TTFB for cached vs uncached requests:

# Test CDN-cached response (second request should be faster)
curl -o /dev/null -w "TTFB: %{time_starttransfer}s\n" https://example.com/page
curl -o /dev/null -w "TTFB: %{time_starttransfer}s\n" https://example.com/page

Cache header verification

Verify your responses include correct caching headers:

curl -I https://example.com/page | grep -i "cache-control\|cf-cache-status\|age\|x-cache"

Look for:

• Cache-Control: public, s-maxage=3600, stale-while-revalidate=86400
• CF-Cache-Status: HIT (Cloudflare) or X-Cache: Hit from cloudfront (AWS)
• Age: 1234 (indicates the response has been cached for 1234 seconds)