CDN Configuration for AI Bot Access

Why CDNs Block AI Crawlers

Content Delivery Networks process every request to your website before it reaches your origin server. This edge layer is designed to cache content, reduce latency, and protect against attacks. The problem is that bot protection features — built to stop scrapers, DDoS bots, and credential stuffers — often catch AI crawlers in the crossfire.

When Cloudflare's Bot Fight Mode sees a non-browser user agent making rapid requests, it issues a JavaScript challenge or CAPTCHA. AI crawlers cannot solve CAPTCHAs. They receive a 503 response and move on — your content never gets indexed.

This is invisible from your side. Your robots.txt allows all the right bots. Your content is perfectly optimized. But the CDN rejects the crawler before it ever sees your robots.txt file.

The fix requires configuring your CDN to recognize and allow legitimate AI search crawlers while still protecting against malicious bots.

Cloudflare Configuration

Cloudflare is the most common CDN, powering over 20% of all websites. It has several features that can interfere with AI crawler access.

Bot Fight Mode

Bot Fight Mode (free plans) and Super Bot Fight Mode (Pro+ plans) automatically challenge requests that Cloudflare classifies as automated. AI crawlers are classified as "verified bots" by Cloudflare, which means they should pass through — but this classification is not always applied correctly, especially for newer AI bots.

Recommended action: If you have Bot Fight Mode enabled, navigate to Security > Bots and check whether "Verified Bot" traffic is set to "Allow." If AI crawlers are still being blocked, create explicit WAF exceptions.

Creating WAF exceptions for AI bots

1. Navigate to Security > WAF > Custom Rules
2. Create a new rule with these conditions:
   • Field: User Agent — Contains: GPTBot OR
   • Field: User Agent — Contains: OAI-SearchBot OR
   • Field: User Agent — Contains: ChatGPT-User OR
   • Field: User Agent — Contains: PerplexityBot OR
   • Field: User Agent — Contains: ClaudeBot OR
   • Field: User Agent — Contains: Applebot
3. Action: Skip — select all security features (WAF, rate limiting, bot fight mode)

This creates a bypass that ensures known AI search crawlers are never challenged or blocked.

Cloudflare AI audit feature

As of 2026, Cloudflare offers an AI bot management dashboard under Security > Bots > AI Crawlers. This panel shows which AI bots have accessed your site and lets you allow or block each one individually. For detailed Cloudflare-specific instructions, see our Cloudflare settings for AI crawler access guide.

Fastly Configuration

Fastly uses Varnish Configuration Language (VCL) for custom logic. Unlike Cloudflare, Fastly does not block bots by default, but custom VCL rules or Fastly's Signal Sciences WAF integration can cause issues.

VCL-based bot allowlisting

Add this to your custom VCL in vcl_recv to ensure AI bots are never blocked by downstream rules:

sub vcl_recv {
  # Identify AI search crawlers
  if (req.http.User-Agent ~ "(?i)(GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|ClaudeBot|Applebot-Extended)") {
    set req.http.X-AI-Search-Bot = "true";
  }
}

sub vcl_pass {
  # Skip bot protection for verified AI search crawlers
  if (req.http.X-AI-Search-Bot == "true") {
    # Allow request to pass to origin without challenge
    return(pass);
  }
}

Fastly Signal Sciences

If you use Fastly's Signal Sciences WAF, check your agent rules for bot-blocking patterns. Create an allowlist for AI crawler user agents or IP ranges.

Vercel Edge Configuration

Vercel's edge network generally allows AI crawlers by default. However, custom middleware or Vercel's firewall rules can block them.

Middleware considerations

If you use Next.js middleware on Vercel, check for user-agent filtering logic:

// middleware.ts — ensure AI bots bypass any blocking logic
import { NextRequest, NextResponse } from 'next/server';

const AI_BOTS = [
  'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
  'PerplexityBot', 'ClaudeBot', 'Applebot-Extended'
];

export function middleware(request: NextRequest) {
  const ua = request.headers.get('user-agent') || '';

  // Allow AI search crawlers to pass through
  if (AI_BOTS.some(bot => ua.includes(bot))) {
    return NextResponse.next();
  }

  // Your existing middleware logic for other requests
  // ...
}

Vercel Firewall

Vercel's Firewall (available on Pro and Enterprise plans) lets you create rules based on request properties. Add an allow rule for AI crawler user agents with priority higher than any blocking rules.

AWS CloudFront Configuration

AWS CloudFront does not block bots by default, but AWS WAF (when attached to a CloudFront distribution) can.

AWS WAF bot control

If you use the AWS WAF Bot Control managed rule group, AI crawlers may be classified as "automated" and blocked. Create a custom rule with higher priority than the Bot Control rules:

1. Open the AWS WAF console
2. Select your Web ACL attached to CloudFront
3. Add a custom rule before the Bot Control rule group
4. Set condition: String match on User-Agent header containing AI bot names
5. Action: Allow

CloudFront Functions

For edge-level bot management, use CloudFront Functions to add custom headers that identify AI bots, then reference those headers in your WAF rules:

function handler(event) {
  var request = event.request;
  var ua = request.headers['user-agent'] ? request.headers['user-agent'].value : '';

  if (/GPTBot|OAI-SearchBot|PerplexityBot|ClaudeBot/i.test(ua)) {
    request.headers['x-ai-bot'] = { value: 'true' };
  }

  return request;
}

Caching Strategy for AI Bots

AI crawlers need fast responses. Sites with First Contentful Paint under 0.4 seconds are cited 3x more often by ChatGPT. Your CDN cache is your best tool for delivering content quickly to AI bots.

Serve cached HTML to AI crawlers

Configure your CDN to always serve cached HTML to AI bot user agents, even if the page would normally be dynamically generated for human visitors. AI crawlers do not need personalized content, shopping cart data, or user-specific recommendations.

Cache duration recommendations

| Content Type | Recommended TTL for AI Bots | |---|---| | Blog posts / articles | 24 hours | | Product pages | 4-6 hours | | Homepage | 1-2 hours | | Pricing pages | 1 hour | | Dynamic API responses | Do not cache |

Avoid Vary: User-Agent

Adding Vary: User-Agent to your cache headers can cause CDNs to create separate cache entries for every unique user agent string, which dramatically reduces cache hit rates. Instead, use your CDN's edge logic to normalize bot user agents into a single cache key.

For more on how page speed affects AI crawling, see our guide on page speed optimization for AI crawlers.

IP Verification for AI Crawlers

Any bot can claim to be GPTBot by setting its user agent string. To distinguish real AI crawlers from spoofed requests, verify the source IP against official ranges.

Published IP ranges

• OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User): Published at openai.com/gptbot-ranges.txt
• Anthropic (ClaudeBot): Published in their bot documentation
• Perplexity (PerplexityBot): Available through their developer documentation
• Apple (Applebot-Extended): Verifiable through reverse DNS lookup to *.applebot.apple.com

Reverse DNS verification

For bots that support it, use reverse DNS to verify authenticity:

# Get the IP's hostname
host 23.98.100.50
# Verify the hostname resolves back to the same IP
host bot-23-98-100-50.search.example.com

Most CDNs can automate IP verification. Cloudflare's "Verified Bots" category handles this automatically for recognized AI crawlers.

Testing Your CDN Configuration

After configuring your CDN, verify that changes are working correctly.

Test through the CDN

Send requests through your public domain (which routes through the CDN) and compare with requests sent directly to your origin server:

# Through CDN
curl -s -o /dev/null -w "%{http_code}" \
  -A "GPTBot/1.2" https://yoursite.com/

# Direct to origin (bypass CDN)
curl -s -o /dev/null -w "%{http_code}" \
  -A "GPTBot/1.2" https://origin.yoursite.com/

If the origin returns 200 but the CDN returns 403 or 503, your CDN is the blocking layer.

Check CDN analytics

Most CDN dashboards show bot traffic statistics. Look for AI crawler user agents in your analytics and check their status codes. Cloudflare's Security Events log shows every request that was challenged or blocked, including the reason.

Monitor after changes

CDN configurations can take minutes to propagate globally. After making changes, wait 5-10 minutes, then test from multiple geographic locations if possible. Use a tool like AImetrico's scanner for comprehensive multi-bot testing.