WAF and Bot Protection: Don't Accidentally Block AI

How WAFs Block AI Crawlers

A Web Application Firewall inspects incoming HTTP requests and blocks those that match threat patterns. The problem for AI SEO is that legitimate AI search crawlers share many characteristics with malicious bots: they use non-browser user agents, make automated requests, do not execute JavaScript, and do not carry cookies.

WAF rules designed to catch bad bots often use these exact signals. The result is a blanket block that catches GPTBot, PerplexityBot, and ClaudeBot alongside actual scrapers and attack tools.

The most common WAF blocking patterns that affect AI crawlers are:

• User agent filtering — Rules that block any user agent not matching a known browser pattern
• JavaScript challenge pages — CAPTCHA or JS challenges that automated crawlers cannot solve
• Rate limiting — Thresholds that AI crawlers hit when making sequential page requests
• Behavioral analysis — Rules that flag non-human browsing patterns (no mouse movement, no cookies, sequential page access)
• Geographic blocking — Blocking traffic from data center IP ranges where AI crawlers operate

Your robots.txt configuration is irrelevant if your WAF blocks the request before the crawler can even read robots.txt.

WordPress Security Plugins

WordPress sites are particularly susceptible to AI crawler blocking because of the popularity of security plugins that include aggressive bot protection.

Wordfence

Wordfence is the most-installed WordPress security plugin. Its firewall includes bot-blocking features that can interfere with AI crawlers.

How to configure Wordfence for AI access:

1. Go to Wordfence > Firewall > Blocking
2. Check the "Blocked IPs" list for any AI crawler IP ranges
3. Add AI crawler IP ranges to the Allowlisted IP Addresses in Wordfence > Firewall > All Firewall Options
4. Under Rate Limiting, set "If a crawler's requests exceed" to a higher threshold or add exceptions for AI bot user agents
5. Under Brute Force Protection, ensure "Block IPs that send POST requests with blank User-Agent and Referer" does not interfere with GET requests from AI bots

Sucuri

Sucuri's cloud-based WAF proxies all traffic through its network. AI crawlers must pass through Sucuri before reaching your WordPress site.

1. Log in to the Sucuri dashboard
2. Navigate to Firewall > Access Control > Whitelist
3. Add the IP addresses of major AI crawlers (OpenAI, Anthropic, Perplexity)
4. Check Firewall > Blocked Attacks for any entries matching AI bot user agents

iThemes Security (Solid Security)

iThemes Security's bot protection can flag AI crawlers as threats.

1. Go to Security > Settings > Lockouts
2. Check if AI crawler IPs appear in the banned hosts list
3. Add AI crawler IP ranges to the Authorized Hosts list
4. Review the "Banned Users" settings for user-agent-based blocking rules

Cloud-Based WAFs

Cloud WAFs operate at the network edge, processing requests before they reach your server. Because they sit upstream of your web server, they override any server-level or application-level access rules.

Cloudflare WAF

Cloudflare's WAF is the most common cloud WAF. For detailed configuration, see our dedicated Cloudflare settings for AI crawlers guide and CDN configuration guide.

AWS WAF

AWS WAF attached to CloudFront distributions or Application Load Balancers can block AI crawlers through managed rule groups, especially the "Bot Control" rule set. Create custom rules with higher priority that allow AI bot user agents before the Bot Control rules execute.

Akamai

Akamai's Bot Manager classifies bots into categories. Ensure AI search crawlers are classified as "good bots" in your Bot Manager configuration. Check Akamai's bot directory for pre-classified AI crawler entries.

Server-Level Firewalls

Beyond application-level WAFs, your server itself may have firewall rules that block AI crawlers.

iptables / nftables (Linux)

Server-level firewalls like iptables or nftables block traffic by IP address or port. They do not inspect user agents, so they cannot selectively block or allow AI bots by name. However, if you have blocked entire IP ranges (such as data center ranges), AI crawler IPs may be caught.

Check for broad IP blocks:

iptables -L -n | grep -i DROP

fail2ban

fail2ban monitors log files and automatically bans IPs that show malicious behavior. If an AI crawler triggers a pattern (such as too many 404 requests from scanning non-existent paths), fail2ban may ban the crawler's IP.

Check fail2ban's jail status:

fail2ban-client status
fail2ban-client status nginx-botsearch

If AI crawler IPs appear in the banned list, add them to fail2ban's ignoreip configuration.

ModSecurity

ModSecurity is an open-source WAF that runs as an Apache or Nginx module. Its Core Rule Set (CRS) includes bot detection rules. Check for rules that block based on user agent patterns and add exceptions for AI crawlers:

# Apache ModSecurity exception for AI crawlers
SecRule REQUEST_HEADERS:User-Agent "@contains GPTBot" "id:9999001,phase:1,allow,nolog"
SecRule REQUEST_HEADERS:User-Agent "@contains PerplexityBot" "id:9999002,phase:1,allow,nolog"
SecRule REQUEST_HEADERS:User-Agent "@contains ClaudeBot" "id:9999003,phase:1,allow,nolog"
SecRule REQUEST_HEADERS:User-Agent "@contains OAI-SearchBot" "id:9999004,phase:1,allow,nolog"

Rate Limiting and AI Crawlers

Rate limiting is a subtle form of blocking that many site owners overlook. AI crawlers fetch multiple pages in sequence — your homepage, key content pages, sitemaps, robots.txt. If your rate limiter allows only 10 requests per minute per IP, the crawler may hit the threshold after just a few pages.

Symptoms of rate limiting

• HTTP 429 Too Many Requests responses in your logs for AI bot user agents
• AI crawlers accessing your homepage successfully but failing on subsequent pages
• Inconsistent crawler behavior — sometimes access works, sometimes it does not

Solutions

1. Increase thresholds for AI bots — Set higher rate limits for verified AI crawler user agents (100-200 requests per minute is reasonable)
2. Use crawl-delay in robots.txt — Tell crawlers to slow down rather than hitting your rate limit: Crawl-delay: 2
3. Whitelist AI crawler IPs — Exempt verified AI crawler IP ranges from rate limiting entirely

Creating Effective Allowlist Rules

The best approach is a layered allowlist that verifies both user agent and IP address.

Step 1: User agent matching

Match requests where the user agent contains known AI search crawler identifiers:

• GPTBot
• OAI-SearchBot
• ChatGPT-User
• PerplexityBot
• ClaudeBot
• Applebot-Extended

Step 2: IP verification

Cross-reference the user agent claim with the request's source IP. Only requests from verified IP ranges should bypass security rules. This prevents attackers from spoofing AI bot user agents to bypass your WAF.

Step 3: Apply to all security layers

Your allowlist must be configured at every layer that can block traffic: CDN, cloud WAF, server firewall, application-level plugin. Missing even one layer means the crawler is still blocked.

For a complete list of AI crawler user agents and IP ranges, see the AI crawler bots list.

Auditing Your WAF Configuration

Run a WAF audit specifically for AI crawler access using this checklist:

1. Review WAF logs — Search for blocked or challenged requests from AI bot user agents in the past 30 days
2. Test each AI bot — Use curl to simulate each AI crawler and verify the response. See our verification guide
3. Check rate limit thresholds — Ensure AI crawlers can make at least 60-100 requests per minute without being throttled
4. Review security plugin settings — Audit every WordPress security plugin, CDN setting, and server-level firewall rule
5. Test after changes — After modifying any security rule, re-test all AI crawlers within 24 hours
6. Document your exceptions — Keep a record of all allowlist rules for AI crawlers so they survive configuration updates