Configuring robots.txt for AI Crawlers: The Complete Guide

What Is robots.txt and Why It Matters for AI

A robots.txt file is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that tells web crawlers which parts of your site they are allowed or not allowed to access. It follows the Robots Exclusion Protocol, a standard that has been in use since 1994. Every major search engine and AI platform respects it.

Here is the critical point for AI SEO: if your robots.txt blocks AI crawlers, your website is completely invisible to AI-generated answers. It does not matter how well-written your content is, how perfect your schema markup is, or how strong your domain authority is. If the crawler cannot fetch your page, you do not exist in AI search.

This is not a theoretical problem. Our analysis shows that roughly 1 in 5 business websites accidentally blocks at least one major AI search bot. Many block all of them. The cause is usually a legacy robots.txt written years ago when "bots" meant scrapers and spam crawlers, not ChatGPT delivering your next customer.

The landscape has changed. In 2024-2026, a new generation of AI crawlers appeared -- each with its own user-agent string, its own purpose, and its own rules. Some crawlers fetch pages to power real-time AI search (these you almost certainly want to allow). Others collect data to train AI models (these you may want to block). Your robots.txt needs to distinguish between them.

This guide will walk you through every AI crawler that matters, show you the exact robots.txt configuration we recommend, and help you avoid the mistakes that make websites vanish from AI results.

The Complete List of AI Crawler Bots (2026)

Not all AI bots are created equal. They fall into three distinct categories, and understanding these categories is the foundation of a correct robots.txt configuration. For a deep dive into every bot, see our complete AI crawler bots list for 2026.

Category 1: Search Bots -- MUST Allow

These bots crawl your pages to power AI search features. When a user asks ChatGPT a question and it cites your website in the answer, it was one of these bots that fetched your page. Blocking them means zero visibility in AI-generated answers.

| Bot Name | Operator | Purpose | |---|---|---| | OAI-SearchBot | OpenAI | Powers ChatGPT Search and SearchGPT results | | PerplexityBot | Perplexity | Powers Perplexity search answers and citations | | ClaudeBot | Anthropic | Powers Claude's web search and citation features | | Bingbot | Microsoft | Powers Bing, Copilot, and feeds data to ChatGPT's Bing integration | | Googlebot | Google | Powers Google Search, AI Overviews, and Gemini's grounded answers | | Applebot | Apple | Powers Siri, Apple Intelligence, and Safari Suggestions |

Rule: Allow all of these. Blocking any of them directly removes you from that platform's AI answers.

Category 2: User-Triggered Fetchers -- MUST Allow

These bots activate only when a user explicitly asks the AI to visit a specific URL or browse the web. They are not autonomous crawlers -- they act on behalf of a human user in real time. For a detailed breakdown of how these differ from search bots, read our guide on search bots vs training bots.

| Bot Name | Operator | Purpose | |---|---|---| | ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks to browse a URL | | Claude-User | Anthropic | Fetches a page when a Claude user provides a link | | Perplexity-User | Perplexity | Fetches a page during a Perplexity user's research session |

Rule: Allow all of these. They represent a real person actively seeking your content right now. Blocking them is like refusing to answer the phone when a customer calls. For more on how OpenAI's bot family works together, see GPTBot, OAI-SearchBot, and ChatGPT-User explained.

Category 3: Training Bots -- Optional Block

These bots collect data to train or fine-tune AI models. They do not directly affect whether your site appears in AI search results today. Blocking them is a business decision: you may choose to opt out of contributing to AI training while still maintaining full AI search visibility.

| Bot Name | Operator | Purpose | |---|---|---| | GPTBot | OpenAI | Collects training data for OpenAI models | | Google-Extended | Google | Collects training data for Gemini and related models | | CCBot | Common Crawl | Builds the Common Crawl dataset used by many AI labs | | Bytespider | ByteDance | Collects data for ByteDance AI models (TikTok, Doubao) | | meta-externalagent | Meta | Collects data for Meta AI (Llama models, Meta AI assistant) |

Rule: Your choice. Blocking these bots will not affect your visibility in AI search results.

robots.txt Syntax and Directives

Before jumping into configurations, you need to understand the syntax. A robots.txt file uses four main directives:

User-agent

Specifies which bot the following rules apply to. Use * to target all bots, or a specific name to target one.

User-agent: OAI-SearchBot

Disallow

Blocks the specified bot from accessing a path. An empty Disallow: means "allow everything."

Disallow: /admin/
Disallow: /private/

Allow

Explicitly allows access to a path, even if a broader Disallow rule would otherwise block it. This directive is essential for fine-grained control.

Allow: /blog/
Allow: /products/

Sitemap

Points crawlers to your XML sitemap. Place this at the bottom of the file, outside any User-agent block.

Sitemap: https://example.com/sitemap.xml

Key syntax rules

1. Each User-agent line starts a new block. All Allow and Disallow lines below it apply to that bot until the next User-agent line.
2. More specific paths take priority. Allow: /blog/ overrides Disallow: / for the /blog/ directory.
3. The file must be at the root. https://example.com/robots.txt is correct. https://example.com/pages/robots.txt will be ignored.
4. Case sensitivity matters for paths, not for directives. Disallow: /Admin/ and Disallow: /admin/ are different rules. But disallow: and Disallow: both work.
5. Wildcard support varies. The * in paths (e.g., Disallow: /*.pdf$) is supported by Googlebot and most major bots, but not all AI crawlers handle wildcards identically. When in doubt, use explicit paths.
6. Comments use #. Anything after # on a line is ignored. Use comments liberally to document your rules.

Here is a minimal example showing the basic structure:

# This is a comment
User-agent: ExampleBot
Disallow: /private/
Allow: /private/public-page/

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Recommended robots.txt Configuration

Below are three configurations ranked from best to worst. We strongly recommend the first one.

Configuration 1: Recommended (Allow search bots, block training bots)

This is the configuration we recommend for most websites. It allows all AI search bots and user-triggered fetchers to access your content, while blocking bots that only collect training data. This gives you maximum AI visibility with minimum data contribution to model training.

# ============================================================
# robots.txt -- Optimized for AI Search Visibility
# Last updated: 2026-03-22
# ============================================================

# ----------------------------------------
# CATEGORY 1: AI Search Bots -- ALLOW
# These power AI search results and citations
# ----------------------------------------

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Applebot
Allow: /

# ----------------------------------------
# CATEGORY 2: User-Triggered Fetchers -- ALLOW
# These fetch pages when a real user requests it
# ----------------------------------------

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# ----------------------------------------
# CATEGORY 3: Training Bots -- BLOCK (optional)
# These collect data for AI model training only
# ----------------------------------------

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: meta-externalagent
Disallow: /

# ----------------------------------------
# Traditional Search Engines -- ALLOW
# ----------------------------------------

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# ----------------------------------------
# Default: Allow all other bots
# ----------------------------------------

User-agent: *
Allow: /

# ----------------------------------------
# Private paths -- block for everyone
# ----------------------------------------

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /staging/

# ----------------------------------------
# Sitemap
# ----------------------------------------

Sitemap: https://example.com/sitemap.xml

Why this works: Each AI search bot is explicitly allowed, so there is no ambiguity. Training bots are blocked by name, preventing data collection for model training while preserving full search visibility. The User-agent: * catch-all at the bottom ensures that any new bots you haven't heard of yet can still access your site by default.

Configuration 2: Minimal (Allow everything)

If you are comfortable with AI training bots accessing your content (or simply want the simplest possible setup), this works:

# robots.txt -- Allow all bots
User-agent: *
Allow: /

Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Trade-off: Maximum simplicity, but your content will be used for AI training as well as search.

Configuration 3: Overly Restrictive (DO NOT USE)

This is what we see on far too many websites. It is usually a legacy configuration that was never updated for AI crawlers:

# BAD EXAMPLE -- DO NOT USE
# This blocks ALL AI bots and makes your site invisible to AI search

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /

What goes wrong: This configuration treats every AI bot the same. It blocks search bots (killing your AI visibility) and user-triggered fetchers (refusing real users) alongside training bots. The final User-agent: * / Disallow: / rule is especially destructive -- it blocks every bot not explicitly mentioned, including future AI search bots that haven't been released yet.

Common Mistakes That Kill AI Visibility

Based on thousands of website audits, these are the robots.txt mistakes we see most often:

Mistake 1: Blocking all bots with a blanket Disallow

# BAD: This blocks EVERYTHING
User-agent: *
Disallow: /

This single rule makes your entire website invisible to every AI search bot, every user-triggered fetcher, and every search engine. It is the nuclear option and almost never what you actually want. If you intend to block specific bots, list them by name. The * wildcard in User-agent should almost always be paired with Allow: /, not Disallow: /.

Mistake 2: Not distinguishing search bots from training bots

Many websites block GPTBot and assume they have "blocked ChatGPT." In reality, GPTBot is OpenAI's training crawler. The bots that actually power ChatGPT's search feature are OAI-SearchBot and ChatGPT-User. Blocking GPTBot while allowing OAI-SearchBot gives you the best of both worlds: no training data contribution, full search visibility. See our detailed explainer on GPTBot, OAI-SearchBot, and ChatGPT-User for the full breakdown.

Mistake 3: Using wildcards incorrectly

# BAD: Wildcard path syntax not universally supported
User-agent: OAI-SearchBot
Disallow: /blog/*.html$
Allow: /blog/

While Googlebot handles path wildcards like * and $ reliably, not all AI crawlers interpret them correctly. Some may treat *.html$ literally rather than as a pattern. Stick to explicit paths when configuring rules for AI bots. If you need complex path-based rules, test them thoroughly using the methods described in the testing section below.

Mistake 4: Forgetting that order matters (sort of)

In robots.txt, the most specific matching rule wins -- but when two rules are equally specific, different bots may interpret the conflict differently. Avoid creating contradictory rules:

# BAD: Contradictory rules for the same bot
User-agent: ClaudeBot
Disallow: /blog/
Allow: /blog/

Which rule wins? It depends on the crawler's implementation. Some bots use the last matching rule, others use the most specific, and some use the first match. Avoid ambiguity entirely by having clear, non-contradictory rules for each bot.

Mistake 5: Editing robots.txt and forgetting about CDN caching

You update your robots.txt locally, deploy it, and assume AI crawlers will see the new version immediately. But if your CDN (Cloudflare, Vercel, Fastly) caches the old robots.txt, crawlers will keep using the cached version for hours or even days. After any robots.txt change, purge the CDN cache for /robots.txt specifically. See our CDN configuration guide for AI bots for platform-specific instructions.

Mistake 6: No robots.txt file at all

If your site has no robots.txt file (returning a 404 at example.com/robots.txt), most crawlers will interpret this as "allow everything." That sounds fine -- but it means you have zero control. Any training bot can freely access your content, and you have no way to selectively block them. Always create an explicit robots.txt, even if its only rule is Allow: /.

How to Test Your robots.txt

Making changes to robots.txt without testing is like editing production code without a staging environment. Here are four methods to verify your configuration:

Method 1: Direct browser check

Open https://yourdomain.com/robots.txt in your browser. Confirm:

• The file loads (HTTP 200, not 404 or 500)
• The content is plain text, not HTML
• Your intended rules are present
• No stale cached version is showing

Method 2: Google Search Console robots.txt Tester

Google Search Console includes a robots.txt testing tool under Crawl > robots.txt Tester. While it is designed for Googlebot, you can manually change the user-agent dropdown to test rules for other bots. Enter a URL path and see whether it would be blocked or allowed.

Method 3: Fetch the page as a specific bot

Use curl with a custom User-Agent header to simulate how a specific AI crawler would see your site:

# Test as OAI-SearchBot
curl -A "OAI-SearchBot" -I https://yourdomain.com/

# Test as ClaudeBot
curl -A "ClaudeBot" -I https://yourdomain.com/

# Test as PerplexityBot
curl -A "PerplexityBot" -I https://yourdomain.com/

Look for HTTP 200 (allowed) vs HTTP 403 or 503 (blocked by WAF/CDN). Note that this tests server-level access, not robots.txt compliance -- bots voluntarily obey robots.txt, so curl will always get through. But if your WAF blocks the request at the server level, curl will reveal it.

Method 4: Use AImetrico's crawler access check

The fastest method: run a free scan at AImetrico. The scanner checks your robots.txt for all major AI bots, tests actual HTTP access from known crawler IP ranges, and flags any CDN/WAF blocks. For more testing approaches, read our guide on verifying AI crawler access to your website.

CDN and WAF Considerations

Your robots.txt can be perfect, and AI crawlers can still be blocked. The problem often sits between your server and the internet: your CDN (Content Delivery Network) or WAF (Web Application Firewall).

Cloudflare

Cloudflare's Bot Fight Mode and Super Bot Fight Mode are the most common culprits. These features automatically challenge or block bot traffic -- including legitimate AI crawlers. To fix this:

1. Go to Security > Bots in your Cloudflare dashboard
2. Check whether Bot Fight Mode is enabled
3. If it is, create WAF Custom Rules that allow traffic from verified AI crawler IP ranges
4. OpenAI publishes its IP ranges at platform.openai.com/docs/bots
5. Anthropic and Perplexity similarly publish their crawler IP documentation

Vercel

Vercel's built-in firewall can interfere with AI crawlers, especially on sites using Edge Functions. Check your vercel.json for any headers or rewrites rules that might return non-200 responses to bot user-agents. Vercel also caches robots.txt aggressively -- after changes, trigger a redeployment or clear the edge cache.

AWS CloudFront / WAF

If you use AWS CloudFront with AWS WAF, check your Web ACL rules. Default managed rule groups like AWSManagedRulesAmazonIpReputationList and AWSManagedRulesBotControlRuleSet may block AI crawler IPs. Create exceptions for known crawler IP ranges.

General CDN checklist

Regardless of which CDN you use, run through this checklist after updating your robots.txt:

1. Purge the CDN cache for /robots.txt specifically
2. Test with curl from an external server (not your local machine, which may bypass the CDN)
3. Check your CDN's analytics/logs for blocked requests from AI crawler user-agents
4. Verify that your CDN does not inject a CAPTCHA or JavaScript challenge for bot traffic

For step-by-step CDN configuration guides for each provider, see our dedicated article on configuring CDNs for AI bot access.