Cloudflare Settings for AI Crawler Access

Cloudflare Features That Affect AI Crawlers

Cloudflare sits between every visitor (including AI crawlers) and your origin server. Multiple Cloudflare features can independently block or challenge AI bots:

| Feature | Risk Level | Location in Dashboard | |---|---|---| | Bot Fight Mode | High | Security > Bots | | Super Bot Fight Mode | Medium | Security > Bots (Pro+) | | WAF Managed Rules | Medium | Security > WAF | | WAF Custom Rules | Varies | Security > WAF > Custom Rules | | Security Level | Medium | Security > Settings | | Under Attack Mode | Critical | Overview (top bar) | | Rate Limiting | Medium | Security > WAF > Rate Limiting | | IP Access Rules | Low | Security > WAF > Tools |

Each feature operates independently. An AI crawler must pass through all of them to reach your content. Configuring one correctly while leaving another misconfigured still results in blocked access.

Bot Fight Mode and Super Bot Fight Mode

Bot Fight Mode is Cloudflare's automated bot detection system. It classifies incoming requests as human, verified bot, or unverified bot, then challenges or blocks unverified bots.

Free plan: Bot Fight Mode

On Cloudflare's free plan, Bot Fight Mode is a simple toggle. When enabled, it challenges requests classified as automated. Major AI crawlers like GPTBot are usually classified as "verified bots" and allowed through. However, newer or less common AI bots may not yet be in Cloudflare's verified bot database.

Recommended setting: Enable Bot Fight Mode but verify each AI crawler individually using curl tests. If any AI bot receives a challenge page, create a WAF exception rule (see next section).

Pro plan: Super Bot Fight Mode

Super Bot Fight Mode offers more granular control with separate actions for different bot categories:

• Definitely automated: Challenge or block
• Likely automated: Challenge or allow
• Verified bots: Allow

Recommended settings:

1. Set "Verified bots" to Allow
2. Set "Definitely automated" to Challenge (not Block)
3. Set "Likely automated" to Allow (or Challenge if you have WAF exceptions for AI bots)

The critical setting is ensuring "Verified bots" is set to Allow, not Challenge or Block.

Creating WAF Exception Rules

WAF custom rules are the most reliable way to ensure AI crawlers bypass all Cloudflare security features. Even if Bot Fight Mode misclassifies an AI crawler, a WAF exception rule overrides it.

Step-by-step rule creation

1. Navigate to Security > WAF > Custom Rules
2. Click Create Rule
3. Name the rule: "Allow AI Search Crawlers"
4. Set the expression using the Expression Builder:
   • When incoming requests match:
   • Field: User Agent — Operator: contains — Value: GPTBot
   • OR Field: User Agent — Operator: contains — Value: OAI-SearchBot
   • OR Field: User Agent — Operator: contains — Value: ChatGPT-User
   • OR Field: User Agent — Operator: contains — Value: PerplexityBot
   • OR Field: User Agent — Operator: contains — Value: ClaudeBot
   • OR Field: User Agent — Operator: contains — Value: Applebot-Extended
5. Then take action: Skip
6. Under Skip settings, select all: WAF, Rate Limiting, Bot Fight Mode, Security Level
7. Click Deploy

Expression editor (advanced)

If you prefer the expression editor, use this expression:

(http.user_agent contains "GPTBot") or
(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Applebot-Extended")

Rule priority

Ensure this rule has a higher priority (lower number) than any blocking rules. Cloudflare evaluates custom rules in order — the first matching rule wins. If a blocking rule matches before your allow rule, the crawler is still blocked.

The AI Audit Panel

Cloudflare introduced the AI Audit panel (also called AI bot management) to give site owners visibility and control over AI crawler traffic.

Accessing the panel

Navigate to Security > Bots > AI Crawlers (the exact location may vary as Cloudflare updates its dashboard). This panel shows:

• Which AI bots have visited your site
• Request volume per bot over time
• Current status — whether each bot is allowed or blocked
• Toggle controls — allow or block each AI bot individually

Recommended configuration

Allow all AI search bots while blocking AI training-only bots:

| Bot | Recommended Setting | Reason | |---|---|---| | GPTBot | Allow | Powers ChatGPT search | | OAI-SearchBot | Allow | ChatGPT search specifically | | ChatGPT-User | Allow | Link previews and browsing | | PerplexityBot | Allow | Perplexity search engine | | ClaudeBot | Allow | Claude AI search | | Applebot-Extended | Allow | Apple Intelligence | | CCBot | Block (optional) | Training data for Common Crawl | | GPTBot (training) | Block (optional) | Training data collection |

For more on distinguishing search bots from training bots, see our robots.txt for AI crawlers guide.

Security Level Settings

Cloudflare's Security Level determines how aggressively it challenges visitors based on their IP reputation.

Navigate to Security > Settings > Security Level. The options range from "Essentially Off" to "I'm Under Attack":

| Level | Effect on AI Crawlers | |---|---| | Essentially Off | No challenge — all crawlers pass | | Low | Challenges only the most threatening IPs — AI crawlers usually pass | | Medium (default) | May challenge some AI crawler IPs with poor reputation scores | | High | Will challenge many AI crawler requests | | I'm Under Attack | Challenges ALL non-browser requests — blocks all AI crawlers |

Recommended setting: Low or Medium. If set to High or Under Attack, create WAF exception rules for AI crawlers.

Under Attack Mode

Under Attack Mode presents a JavaScript challenge page to every single request. AI crawlers cannot execute JavaScript challenges and will receive a 503 error.

This mode should only be active during an actual DDoS attack. If you need it temporarily, ensure your WAF exception rule for AI crawlers is configured to skip the Under Attack check. After the attack subsides, disable Under Attack Mode immediately.

Caching Configuration for AI Bots

Serving cached content to AI crawlers improves response times and reduces origin server load.

Creating a Cache Rule for AI bots

1. Navigate to Caching > Cache Rules
2. Create a new rule matching AI bot user agents (same expression as the WAF rule)
3. Set Cache Level to "Cache Everything"
4. Set Edge Cache TTL to 2-24 hours depending on content freshness needs
5. Deploy the rule

This ensures AI crawlers receive cached HTML from Cloudflare's edge network instead of waiting for your origin server to generate the page.

Firewall Rules for IP Verification

For maximum security, combine user agent matching with IP verification. This ensures only genuine AI crawlers bypass your security — not attackers spoofing AI bot user agents.

Using Cloudflare's AS Number matching

AI crawler requests come from known Autonomous System (AS) numbers. In your WAF rule expression, add AS number conditions:

(http.user_agent contains "GPTBot" and ip.src.asnum eq 394161) or
(http.user_agent contains "PerplexityBot" and ip.src.asnum in {394161 13335})

Check each AI provider's documentation for their current AS numbers and IP ranges.

Using IP lists

Cloudflare's IP Lists feature (available on higher plans) lets you create named lists of IP addresses. Create a list called "AI Search Crawlers" containing the published IP ranges, then reference this list in your WAF rules.

Testing Your Configuration

After making Cloudflare changes, wait 2-5 minutes for propagation, then test:

# Test GPTBot
curl -s -o /dev/null -w "%{http_code}" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://yoursite.com/

# Verify response is not a challenge page
curl -A "GPTBot/1.2" https://yoursite.com/ | grep -i "challenge\|captcha\|verify"

If the status code is 200 and the response body contains your actual page content (not a challenge page), Cloudflare is configured correctly.

Check Security > Events in your Cloudflare dashboard to see how your new rules are being applied. Filter by AI bot user agents to verify they are matching the Skip rule.

For a comprehensive multi-bot test, see our verifying AI crawler access guide.