CDN Setup for Global AI Crawler Performance

Why CDNs Matter for AI Crawlers

A Content Delivery Network places cached copies of your website on servers distributed globally. When a request arrives, the CDN serves it from the nearest edge server instead of routing it to your origin server. This eliminates most of the network latency and server processing time.

For AI crawlers, CDNs provide three critical benefits:

1. Reduced latency. AI crawlers fetch thousands of pages per session. Every millisecond saved per page multiplies across the entire crawl. A CDN can reduce per-page fetch time from 800ms to 50ms, allowing the crawler to visit 16x more pages in the same time window.

2. Origin server protection. When multiple AI crawlers (ChatGPT, Perplexity, Gemini, Claude) visit simultaneously, they can overwhelm a small origin server. CDN edge caching absorbs this traffic without any load on your server.

3. Consistent response times. Without a CDN, response times vary based on server load, geographic distance, and current traffic. CDNs provide consistent sub-100ms responses regardless of these factors.

For the full picture of how page speed affects AI crawling, see our page speed for AI crawlers guide. Making sure crawlers can actually access your pages is covered in robots.txt for AI crawlers.

AI Crawler Locations and CDN Strategy

Understanding where AI crawlers originate helps you prioritize CDN edge locations:

| AI Crawler | Primary Data Centers | Secondary Locations | |---|---|---| | OAI-SearchBot / ChatGPT-User | US East (Virginia), US West | Azure global network | | Googlebot / Google-Extended | US (multiple locations) | Global Google infrastructure | | PerplexityBot | US East (AWS) | US West | | ClaudeBot | US East, US West | AWS global regions | | Bingbot / BingPreview | US (multiple) | Azure global network |

The conclusion is clear: US East (Virginia) is the single most important edge location for AI crawlers. If your CDN only has one edge node, it should be there.

However, your human visitors likely come from different locations. A good CDN strategy serves both audiences:

• AI crawlers: Prioritize US East and US West edge nodes with aggressive HTML caching
• Human visitors: Ensure edge coverage in your target market regions
• Both: Use the same CDN — the global network handles both audiences automatically

Cloudflare Configuration for AI SEO

Cloudflare is the most popular CDN choice for AI SEO due to its free tier, 300+ global edge locations, and built-in bot management.

Step 1: Enable full-page caching

By default, Cloudflare only caches static assets. To cache HTML pages:

Using Page Rules (legacy):

• URL pattern: example.com/*
• Setting: Cache Level = Cache Everything
• Setting: Edge Cache TTL = 2 hours

Using Cache Rules (recommended): Create a cache rule that matches all HTML responses:

• Expression: (http.request.uri.path ne "/admin" and http.request.uri.path ne "/api/")
• Cache eligibility: Eligible for cache
• Edge TTL: Override = 7200 seconds (2 hours)

Step 2: Configure bot management

Navigate to Security > Bots and ensure these settings:

• Bot Fight Mode: Set to allow verified bots. AI search crawlers should be categorized as verified bots by Cloudflare.
• Custom rules: Create explicit allow rules for AI crawler User-Agents if needed:
  • OAI-SearchBot
  • ChatGPT-User
  • PerplexityBot
  • ClaudeBot
  • anthropic-ai

Step 3: Enable performance features

• Auto Minify: Enable for JavaScript, CSS, and HTML
• Brotli compression: Enable (default on Cloudflare)
• Early Hints: Enable to send 103 responses with preload hints
• HTTP/3: Enable for QUIC protocol support
• Tiered Caching: Enable to reduce origin requests by sharing cache between Cloudflare data centers

Step 4: Optimize cache hit ratio

Monitor your cache hit ratio in Cloudflare Analytics. Target above 80% for HTML pages. Common reasons for low hit ratios:

• Cookies in responses (Set-Cookie headers bypass cache)
• Vary headers that create too many cache variants
• Short cache TTLs requiring frequent revalidation
• Query strings creating unique cache keys

Vercel Edge Network Setup

Vercel's Edge Network is optimized for Next.js and Jamstack frameworks with automatic edge caching.

Static and SSG pages

Static pages and SSG-generated pages are automatically cached at the edge with no configuration needed. Response times are typically 5-20ms from edge locations.

ISR (Incremental Static Regeneration)

For content that changes periodically:

export async function getStaticProps() {
  return {
    props: { data },
    revalidate: 3600, // Revalidate every hour
  };
}

ISR serves cached content instantly while revalidating in the background. AI crawlers always get a fast response — never waiting for regeneration.

Edge functions for bot-specific logic

Use Vercel Edge Middleware to customize responses for AI bots:

// middleware.js
export function middleware(request) {
  const userAgent = request.headers.get('user-agent') || '';
  const isAIBot = /OAI-SearchBot|PerplexityBot|ClaudeBot/i.test(userAgent);

  if (isAIBot) {
    // Add headers to ensure full content delivery
    const response = NextResponse.next();
    response.headers.set('X-Robots-Tag', 'noarchive');
    return response;
  }
}

Vercel caching headers

Set appropriate cache-control headers in next.config.js or vercel.json:

{
  "headers": [
    {
      "source": "/(.*)",
      "headers": [
        {
          "key": "Cache-Control",
          "value": "public, s-maxage=3600, stale-while-revalidate=86400"
        }
      ]
    }
  ]
}

The s-maxage controls CDN edge cache duration, while stale-while-revalidate allows serving stale content while refreshing.

AWS CloudFront for AI Bots

AWS CloudFront offers the most granular control through Lambda@Edge and CloudFront Functions.

Cache behavior configuration

Create a cache behavior for HTML content:

• Path pattern: * (or specific paths like /blog/*)
• Cache policy: Create a custom policy with TTL: min=60, default=3600, max=86400
• Origin request policy: Forward only essential headers (Host, Accept-Encoding)
• Compress objects: Yes (Gzip and Brotli)

Lambda@Edge for AI bot optimization

Deploy a Lambda@Edge function to customize responses for AI crawlers:

exports.handler = async (event) => {
  const request = event.Records[0].cf.request;
  const userAgent = request.headers['user-agent']?.[0]?.value || '';

  if (/OAI-SearchBot|PerplexityBot|ClaudeBot/i.test(userAgent)) {
    // Increase cache TTL for bot requests
    request.headers['x-bot-request'] = [{ key: 'X-Bot-Request', value: 'true' }];
  }

  return request;
};

CloudFront Functions for lightweight processing

For simpler bot-specific logic, CloudFront Functions execute at the edge with sub-millisecond latency:

function handler(event) {
  var response = event.response;
  var headers = response.headers;

  // Add security headers without affecting cache
  headers['x-content-type-options'] = { value: 'nosniff' };
  headers['x-frame-options'] = { value: 'DENY' };

  return response;
}

Bot Protection Without Blocking AI

CDN bot protection can accidentally block AI crawlers. Here is how to prevent this:

Identify blocked crawlers

Check your CDN analytics for blocked or challenged requests from AI User-Agents. In Cloudflare, go to Security > Events and filter by User-Agent containing "Bot" or specific AI crawler names.

Create allow rules

Explicitly allow known AI search crawlers:

Cloudflare WAF Custom Rule:

• Expression: (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "ChatGPT-User")
• Action: Skip all remaining rules

Important: Allow AI search bots but keep blocking training bots if desired. Refer to our robots.txt guide for AI crawlers for the distinction between search and training bots.

Rate limiting considerations

If you use rate limiting, set higher thresholds for AI crawlers. A bot might request 50-100 pages in a short burst during a crawl session. Standard rate limits designed for human users (5-10 requests per second) can trigger blocks.

CAPTCHA and JavaScript challenges

Never serve CAPTCHAs to AI crawlers — they cannot solve them. JavaScript challenges are equally problematic since most AI bots have limited JS execution. Ensure your CDN's security level allows known bots to pass without challenges.

Cache Configuration Best Practices

Cache-Control header strategy

Use a layered approach:

Cache-Control: public, max-age=0, s-maxage=3600, stale-while-revalidate=86400

• public — Allows CDN caching
• max-age=0 — Browser always revalidates (fresh content for users)
• s-maxage=3600 — CDN caches for 1 hour
• stale-while-revalidate=86400 — CDN can serve stale content for 24 hours while refreshing in the background

Vary header management

The Vary header tells the CDN to create separate cache entries for different request variations. Be conservative:

Vary: Accept-Encoding

Avoid Vary: User-Agent — this creates a separate cache entry for every unique User-Agent string, effectively disabling caching. If you need to serve different content to bots, use CDN edge logic instead.

Cache key optimization

Ensure your CDN cache key does not include unnecessary query parameters. Marketing parameters like utm_source, utm_medium, and fbclid should be stripped from the cache key:

Cloudflare: Use Cache Rules to ignore specific query strings CloudFront: Use a cache policy that forwards only whitelisted query strings

Purge strategy

Set up automatic cache purging when content is updated:

• CMS integration: Purge specific URLs when content is published or edited
• Tag-based purging: Use cache tags (supported by Cloudflare and Fastly) to purge groups of related content
• Avoid full purges: Purging the entire cache forces all pages to be re-fetched from origin, temporarily increasing TTFB for all requests