Search Bots vs Training Bots: Which AI Crawlers to Allow and Block

Why the Distinction Matters

Not all AI crawlers do the same thing. This is the single most misunderstood concept in AI SEO today, and getting it wrong can either make your site invisible to AI search or give away your content for model training without getting anything in return.

When a company decides to "block AI bots," they usually add a blanket rule to their robots.txt that stops every crawler with "GPT," "Bot," or "AI" in its name. The intention is to prevent their content from being used to train language models. The actual result is that they also block the bots that would have driven traffic to their site through AI-generated search results.

The distinction comes down to purpose. Some bots crawl your site so that an AI can answer a user's question right now and link back to you. Other bots crawl your site to absorb your content into a model's training data, where it becomes part of the model's general knowledge — with no attribution or traffic sent your way.

Understanding these categories is the foundation of every robots.txt decision you will make for AI crawlers. For a complete list of all known AI crawlers and their user-agent strings, see our AI Crawler Bots List for 2026.

Category 1: Search Bots (Always Allow)

Search bots are AI crawlers that retrieve your content in real time to include it in AI-generated search results. When a user asks ChatGPT a question and the model searches the web, it is a search bot that visits your page, reads the content, and potentially cites you in the response — with a link back to your site.

These bots are the AI equivalent of Googlebot. Blocking them is like deindexing your site from Google. You would never do that intentionally.

The key search bots to allow

| Bot Name | Operator | Purpose | User-Agent String | |---|---|---|---| | OAI-SearchBot | OpenAI | Powers ChatGPT web search and AI Mode results | OAI-SearchBot | | PerplexityBot | Perplexity AI | Powers Perplexity search answers with citations | PerplexityBot | | Applebot | Apple | Powers Apple Intelligence, Siri, and Safari AI features | Applebot | | Amazonbot | Amazon | Powers Alexa answers and Amazon AI features | Amazonbot |

Why search bots must be allowed

Search bots have a direct, measurable impact on your traffic and revenue:

• They drive referral traffic. When ChatGPT cites your page, users click through. AI referral traffic converts at 4.4x the rate of organic search.
• They operate in real time. The bot visits your page at the moment a user asks a question. If your page is blocked, the bot moves to a competitor's page.
• They respect robots.txt. If you block OAI-SearchBot, ChatGPT will not crawl your pages for search results — and you disappear from ChatGPT answers entirely.

For a deeper look at how OpenAI's bots differ from each other, read our guide on GPTBot, OAI-SearchBot, and ChatGPT-User explained.

Category 2: User-Triggered Fetchers (Always Allow)

User-triggered fetchers are bots that activate only when a real person explicitly asks an AI assistant to visit a specific URL. They are not autonomous crawlers — they act on behalf of individual users.

The key user-triggered fetchers

| Bot Name | Operator | When It Activates | User-Agent String | |---|---|---|---| | ChatGPT-User | OpenAI | When a ChatGPT user says "read this page" or uses browse mode | ChatGPT-User | | Claude-User | Anthropic | When a Claude user shares a URL for analysis | Claude-User | | Cohere-User | Cohere | When a user requests web access through Cohere | cohere-ai |

How fetchers differ from search bots

The distinction is subtle but important. A search bot crawls the web proactively to find relevant content. A user-triggered fetcher only activates when someone asks the AI to visit a specific page. Think of it like the difference between Googlebot indexing the web and you clicking a link in a browser — except the "browser" is an AI assistant.

Why fetchers should be allowed

Blocking user-triggered fetchers creates a frustrating experience for real people:

• A potential customer pastes your URL into ChatGPT and asks "Is this company legit?" — if ChatGPT-User is blocked, the model cannot read your page and the user gets no useful answer.
• A researcher shares your article in Claude and asks for a summary — if Claude-User is blocked, the analysis fails.
• These bots carry no training risk. The content is retrieved once, shown to one user, and not stored for model training.

There is no strategic reason to block user-triggered fetchers. They represent direct engagement from people who are already interested in your content.

Category 3: Training Bots (Your Decision)

Training bots crawl your website to collect content that will be used to train or fine-tune AI models. The content you create today becomes part of a model's internal knowledge in a future version. There is no real-time citation, no link back to your site, and no direct traffic benefit.

The key training bots

| Bot Name | Operator | What It Trains | User-Agent String | |---|---|---|---| | GPTBot | OpenAI | Future GPT models (GPT-5, GPT-6, etc.) | GPTBot | | ClaudeBot | Anthropic | Future Claude models | ClaudeBot | | Google-Extended | Google | Gemini models and AI features | Google-Extended | | CCBot | Common Crawl | Open training datasets used by many AI labs | CCBot | | Bytespider | ByteDance | TikTok AI and internal models | Bytespider | | FacebookBot | Meta | Meta AI and LLaMA models | FacebookBot |

How training differs from search

This is critical to understand: training bots and search bots are completely separate systems, even when they come from the same company. OpenAI operates both GPTBot (training) and OAI-SearchBot (search). Blocking GPTBot does NOT affect OAI-SearchBot. They are different user-agent strings, different crawling infrastructure, and different purposes.

The same principle applies to Anthropic (ClaudeBot for training vs Claude-User for fetching) and Google (Google-Extended for Gemini training vs Googlebot for search). For full details on each bot's role, see our AI Crawler Bots List for 2026.

The Decision Framework

Here is a simple three-step framework for deciding which AI crawlers to allow or block. Apply it in order:

Step 1: Allow all search bots

This is non-negotiable. Search bots are the mechanism through which AI platforms discover and cite your content. Blocking them eliminates your AI visibility entirely.

Action: Ensure OAI-SearchBot, PerplexityBot, Applebot, and Amazonbot are explicitly allowed (or at minimum, not blocked) in your robots.txt.

Step 2: Allow all user-triggered fetchers

User-triggered fetchers represent real people interacting with your content. There is no training risk and no strategic reason to block them.

Action: Ensure ChatGPT-User and Claude-User are allowed. If your CDN or WAF blocks unknown bots by default, add these to your allowlist. See our CDN configuration for AI bots guide for platform-specific instructions.

Step 3: Decide on training bots based on your business model

This is the only step that requires a genuine decision. Consider the following:

| If you are... | Recommended approach | |---|---| | A publisher whose revenue depends on page views and ads | Block training bots — your content has more value when users visit your site than when it is absorbed into a model | | A SaaS company that wants brand recognition | Allow training bots — you want AI models to know about your product in their general knowledge | | A local business or service provider | Allow training bots — the more AI models know about your business, the more likely they mention you | | An e-commerce store | Block training bots — product descriptions and reviews are proprietary competitive advantages | | A content creator or author | Block training bots — your writing is your product; protect it |

There is no universally correct answer for Step 3. The correct answer depends on whether your content is more valuable as a destination (block training) or as widespread knowledge (allow training).

Pros and Cons of Blocking Training Bots

Arguments for blocking training bots

1. Content protection. Your articles, guides, and research remain your competitive advantage. AI models cannot reproduce your content from memory if they were never trained on it.
2. Leverage for licensing. Major publishers like the New York Times and Associated Press have negotiated paid licensing deals with AI companies. Blocking training bots gives you negotiating position.
3. Bandwidth savings. Training bots can be aggressive crawlers. GPTBot and CCBot in particular can generate significant server load during crawling sessions.
4. Philosophical principle. Some site owners believe that using content for training without compensation is unfair, regardless of the practical impact.

Arguments for allowing training bots

1. Brand familiarity. If future GPT-5 or Claude-4 models were trained on content that mentions your brand, they are more likely to recommend you in conversations — even without real-time search.
2. Indirect visibility boost. Models that "know" your brand from training data may be more likely to select your pages as search results, though this is speculative and unproven.
3. Ecosystem participation. Allowing training contributes to the broader AI ecosystem, which some companies view as a net positive for their industry.
4. Simplicity. Allowing everything means fewer robots.txt rules to maintain and fewer chances of accidentally blocking the wrong bot.

The bottom line

For most businesses focused on AI visibility: allow search bots and fetchers unconditionally, then make a deliberate decision about training bots. Do not let a blanket rule decide for you by accident.

Recommended robots.txt Configuration

Here is a robots.txt configuration that implements the framework above — allowing search bots and fetchers while blocking training bots. Adjust the training bot section based on your Step 3 decision.

# ==============================================
# AI SEARCH BOTS — ALLOW (drives traffic)
# ==============================================
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Amazonbot
Allow: /

# ==============================================
# USER-TRIGGERED FETCHERS — ALLOW (real users)
# ==============================================
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# ==============================================
# TRAINING BOTS — BLOCK (optional, adjust per your needs)
# ==============================================
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: FacebookBot
Disallow: /

# ==============================================
# DEFAULT — allow all other bots
# ==============================================
User-agent: *
Allow: /

Important notes about this configuration:

• Explicit Allow: / rules for search bots ensure they are never caught by broader blocking rules elsewhere in the file.
• If you decide to allow training bots too, simply change their Disallow: / to Allow: / or remove those blocks entirely.
• robots.txt rules apply to the entire domain. If you want to block training bots from specific directories only (e.g., /blog/ but not /products/), use path-specific rules instead.
• After updating your robots.txt, verify the changes are working. Our guide on verifying AI crawler access walks through the process step by step.

For a complete robots.txt guide covering syntax, crawl-delay, sitemap directives, and edge cases, see Configuring robots.txt for AI Crawlers.

Why Sites Accidentally Block Everything

In our analysis of thousands of websites, we see the same patterns repeatedly. Here are the most common ways sites unintentionally block AI search bots along with training bots:

Pattern 1: The blanket wildcard block

# WRONG: Blocks ALL AI bots including search
User-agent: GPTBot
Disallow: /

User-agent: *Bot
Disallow: /

The wildcard *Bot catches OAI-SearchBot, PerplexityBot, and every other bot with "Bot" in the name. The site owner intended to block GPTBot for training but ended up blocking all AI search.

Pattern 2: CDN or WAF default rules

Many CDN providers (Cloudflare, Akamai, Fastly) include bot management features that block or challenge unrecognized user agents by default. If OAI-SearchBot is not on your CDN's allowlist, it may receive a 403 response or a JavaScript challenge it cannot solve — even if your robots.txt says Allow: /.

Your robots.txt and your CDN/WAF rules must be aligned. See our CDN configuration for AI bots guide for platform-specific instructions.

Pattern 3: Copy-pasted robots.txt from 2024

Many robots.txt files circulating online were written before OpenAI split its bots into GPTBot and OAI-SearchBot (which happened in late 2024). These older configs block "GPTBot" without realizing that OAI-SearchBot now handles search separately. The result: training is blocked (intended) and search is allowed (fine) — but only if the site owner understands the distinction.

Worse, some 2024-era configs list ChatGPT-User under training bots and block it, which breaks user-triggered browsing.

Pattern 4: The "block everything, allow nothing" approach

Some security-focused teams add every AI bot name they can find to a blocklist without distinguishing categories. This is the digital equivalent of locking your store's front door to prevent shoplifting — it stops theft, but it also stops customers.

How to check your current configuration

The fastest way to check is to scan your site with AImetrico. The scan tests each major AI bot's access individually and tells you exactly which are allowed and which are blocked. You can also check manually by reviewing your robots.txt file, CDN rules, and server access logs — our verifying AI crawler access guide covers the manual process.