Complete List of AI Crawler Bots (2026): Every Bot You Need to Know

How AI Crawler Bots Work

AI crawler bots are automated programs that visit your website to collect content for AI systems. They work similarly to Googlebot — they follow links, request pages, and parse HTML — but they serve different purposes depending on which company operates them and what product they feed into.

Every AI bot identifies itself with a User-Agent string in its HTTP request headers. This string is what you target in your robots.txt file to allow or block specific bots. Getting this right is the single most impactful technical decision in AI SEO: block the wrong bot and you disappear from AI answers entirely; allow the wrong bot and you may be giving away training data you intended to protect.

The challenge is that the AI bot landscape has grown rapidly. In 2023, there were roughly 3-4 AI-specific bots to manage. By mid-2026, there are over 15 distinct User-Agent strings from major AI companies, each with different purposes and different implications for your visibility. This reference catalogs all of them.

The Three Categories of AI Bots

Not all AI bots are the same. Understanding the distinction between these three categories is essential for making informed robots.txt decisions. For a deeper comparison, see our dedicated guide on search bots vs training bots.

| Category | Purpose | Recommendation | Impact if Blocked | |---|---|---|---| | Search Bots | Fetch content in real time for AI-generated answers | MUST ALLOW | Your site disappears from AI search results | | User-Triggered Fetchers | Fetch a specific URL when a user shares it in chat | MUST ALLOW | Users cannot ask AI to summarize/analyze your pages | | Training/Indexing Bots | Crawl your site for model training or general indexing | OPTIONAL | No impact on AI search visibility; prevents training use |

The critical rule: never block search bots or user-triggered fetchers unless you intentionally want to be invisible in AI answers. Training bots are a business decision — there are legitimate reasons to block them, and doing so does not hurt your AI search presence.

Category 1: Search Bots (MUST ALLOW)

Search bots crawl your site in real time (or near-real time) to retrieve content that appears in AI-generated answers. They are the AI equivalent of Googlebot — if you block them, your content cannot appear in results. These bots drive referral traffic back to your site through citations and source links.

| Bot Name | User-Agent String | Company | Purpose | Robots.txt Directive | Recommendation | Notes | |---|---|---|---|---|---|---| | OAI-SearchBot | OAI-SearchBot | OpenAI | Powers ChatGPT's web search feature (ChatGPT Search, formerly "Browse with Bing"). Fetches pages in real time when ChatGPT performs a search to answer a user query. | User-agent: OAI-SearchBot
Allow: / | MUST ALLOW | Introduced in late 2024. This is the bot you need for ChatGPT Search visibility. Separate from GPTBot (training) and ChatGPT-User (link fetching). See our breakdown of GPTBot vs OAI-SearchBot vs ChatGPT-User. | | PerplexityBot | PerplexityBot | Perplexity AI | Crawls and indexes pages for Perplexity's AI search engine. Fetches content in real time to generate cited answers. | User-agent: PerplexityBot
Allow: / | MUST ALLOW | Perplexity is the fastest-growing dedicated AI search engine. Its answers always include source citations with clickable links, making it a strong referral traffic source. Respects robots.txt and crawl-delay directives. | | Applebot-Extended | Applebot-Extended | Apple | Powers Apple Intelligence features including Safari Highlights, Siri AI responses, and Spotlight suggestions across all Apple devices. | User-agent: Applebot-Extended
Allow: / | MUST ALLOW | Distinct from the standard Applebot (which handles Apple Search and Siri web results). Applebot-Extended specifically feeds Apple's generative AI features. With Safari holding ~20% global browser share, blocking this bot means losing visibility across the entire Apple ecosystem. | | Bingbot | bingbot | Microsoft | Primary crawler for Bing search index. Powers Microsoft Copilot AI answers, Bing Chat, and Bing search results. | User-agent: bingbot
Allow: / | MUST ALLOW | Bingbot has been around for years, but its importance has increased dramatically. It now feeds into Microsoft Copilot's AI-generated answers in addition to traditional Bing results. Most sites already allow Bingbot, but verify this if you use aggressive bot-blocking rules. |

Why these bots matter

Blocking any search bot listed above is equivalent to adding noindex for that AI platform. Your pages will not appear in AI-generated answers on that platform, and you will receive zero referral traffic from it.

OpenAI's bot architecture is the most complex. They operate three separate User-Agents, each with a distinct purpose — a common source of confusion. Our detailed guide on GPTBot, OAI-SearchBot, and ChatGPT-User explains exactly how they interact and which combinations to allow.

Category 2: User-Triggered Fetchers (MUST ALLOW)

User-triggered fetchers are not traditional crawlers. They do not proactively discover or index your site. Instead, they activate only when a real human user shares a URL inside an AI chat session (e.g., "summarize this article" or "what does this page say about pricing?"). They fetch a single, specific URL on behalf of that user.

| Bot Name | User-Agent String | Company | Purpose | Robots.txt Directive | Recommendation | Notes | |---|---|---|---|---|---|---| | ChatGPT-User | ChatGPT-User | OpenAI | Fetches a URL when a ChatGPT user pastes a link into the conversation. The model then reads and responds based on the page content. | User-agent: ChatGPT-User
Allow: / | MUST ALLOW | This is not a crawler — it only visits pages explicitly shared by users. Blocking it means ChatGPT users cannot ask the model to read, summarize, or analyze your content. This damages user experience and reduces engagement with your brand. | | Perplexity-User | Perplexity-User | Perplexity AI | Fetches a URL when a Perplexity user shares a link in their search session for the AI to analyze. | User-agent: Perplexity-User
Allow: / | MUST ALLOW | Similar to ChatGPT-User but for the Perplexity platform. Introduced in 2025. Separate from PerplexityBot (which crawls proactively for search indexing). | | Claude-User | Claude-User | Anthropic | Fetches a URL when a Claude user shares a link in conversation. Powers Claude's ability to read and discuss web content shared by users. | User-agent: Claude-User
Allow: / | MUST ALLOW | Anthropic's user-triggered fetcher. Only activates when a human explicitly shares a URL. Claude is integrated into Apple Safari via Apple Intelligence, significantly expanding its reach. |

Why you should never block user-triggered fetchers

These bots represent a human actively engaging with your content through an AI interface. Blocking them is like blocking a user's browser — you are preventing someone who wants to read your page from doing so. Unlike training bots, user-triggered fetchers:

• Only visit URLs explicitly shared by a human
• Do not crawl or discover new pages
• Do not store content for training
• Drive direct engagement and potential conversions

There is no legitimate reason to block these bots unless you are intentionally preventing AI users from accessing your content entirely.

Category 3: Training/Indexing Bots (OPTIONAL -- Block or Allow)

Training and indexing bots crawl your website to collect data used for AI model training, fine-tuning, or building general-purpose web indexes. Blocking these bots does not remove you from AI search results (as long as you allow the search bots in Category 1). However, allowing them means your content may be used to train future AI models.

This is a business decision. Some organizations allow training bots because broader model training can improve how AI understands their brand. Others block them to protect proprietary content. Both positions are valid.

| Bot Name | User-Agent String | Company | Purpose | Robots.txt Directive | Recommendation | Notes | |---|---|---|---|---|---|---| | GPTBot | GPTBot | OpenAI | Crawls the web to build OpenAI's search index and collect training data for GPT models. Proactively discovers and visits pages across your site. | User-agent: GPTBot
Disallow: / (or Allow: /) | Optional | The original OpenAI crawler, introduced in August 2023. Serves dual purposes: indexing for search AND collecting training data. If you block GPTBot but allow OAI-SearchBot, ChatGPT Search can still find your content through other index sources. See GPTBot vs OAI-SearchBot for details. | | ClaudeBot | ClaudeBot | Anthropic | Crawls websites to collect training data for Claude models. Does not power Claude's real-time search features. | User-agent: ClaudeBot
Disallow: / (or Allow: /) | Optional | Distinct from Claude-User (which is user-triggered and should always be allowed). ClaudeBot is purely a training data collector. Anthropic has stated it respects robots.txt directives. Blocking ClaudeBot does not affect whether Claude cites your content in conversations. | | Google-Extended | Google-Extended | Google | Controls whether your site's content is used to train Google's Gemini models and other generative AI products. Does not affect Google Search indexing or ranking. | User-agent: Google-Extended
Disallow: / (or Allow: /) | Optional | This is Google's dedicated AI training control. Blocking Google-Extended does NOT affect your Google Search rankings, your appearance in Google AI Mode, or Gemini's ability to cite you in search results. It only prevents your content from being used as training data for future Gemini model versions. | | CCBot | CCBot | Common Crawl | Crawls the web to build the Common Crawl open dataset, which is used by many AI companies (including OpenAI, Anthropic, Google, and others) as foundational training data. | User-agent: CCBot
Disallow: / (or Allow: /) | Optional | Common Crawl is a non-profit that provides free web archives. Its dataset is one of the primary training data sources for most major LLMs. Blocking CCBot reduces your content's presence in future model training across multiple AI companies simultaneously. | | Bytespider | Bytespider | ByteDance | Crawls the web for ByteDance's AI products, including TikTok's search features and internal LLM development. | User-agent: Bytespider
Disallow: / (or Allow: /) | Optional (commonly blocked) | One of the most aggressive crawlers by volume. Many site operators block Bytespider due to high crawl rates that can impact server performance. If you allow it, consider setting a strict Crawl-delay directive. | | meta-externalagent | meta-externalagent | Meta (Facebook) | Crawls content for Meta's AI products, including Meta AI assistant and AI features within Instagram, WhatsApp, and Facebook. | User-agent: meta-externalagent
Disallow: / (or Allow: /) | Optional | Introduced in 2024. Separate from FacebookExternalHit (which handles link previews). This bot specifically serves Meta's generative AI systems. Meta has not published a detailed transparency page for this bot's behavior. | | Amazonbot | Amazonbot | Amazon | Crawls web content for Amazon's AI services, including Alexa AI responses, Amazon Q, and internal product features. | User-agent: Amazonbot
Disallow: / (or Allow: /) | Optional | Respects robots.txt. Amazon provides a verification page for Amazonbot IPs. Relevant if your business has any intersection with voice search (Alexa) or Amazon's enterprise AI (Amazon Q). | | cohere-ai | cohere-ai | Cohere | Crawls web content for Cohere's enterprise AI models and RAG (Retrieval-Augmented Generation) products. | User-agent: cohere-ai
Disallow: / (or Allow: /) | Optional | Cohere primarily serves enterprise B2B clients. Their crawler is less aggressive than consumer-facing bots. Relevant mainly if your content targets enterprise/B2B audiences that may use Cohere-powered tools. | | FacebookExternalHit | FacebookExternalHit | Meta (Facebook) | Fetches pages to generate link previews (Open Graph) when URLs are shared on Facebook, Instagram, Messenger, and WhatsApp. | User-agent: FacebookExternalHit
Allow: / | Recommended allow | While not strictly an AI training bot, it is often grouped with AI crawlers in robots.txt discussions. Blocking it prevents link previews on all Meta platforms, which significantly impacts social media sharing. Most sites should allow it. |

Decision framework for training bots

Use this framework to decide which training bots to allow or block:

Allow training bots if:

• You publish public-facing content meant to be widely known (blogs, documentation, public research)
• You want AI models to have accurate, up-to-date knowledge about your brand and products
• You benefit from broader AI awareness of your industry terminology and offerings

Block training bots if:

• You publish proprietary content, original research, or premium/paywalled material
• You want to control how your intellectual property is used
• You are concerned about AI models reproducing your content without attribution

For most commercial websites, a balanced approach works best: allow search bots and user-triggered fetchers (Categories 1 and 2), and block most training bots except where you see a strategic benefit. For the full robots.txt implementation, see our step-by-step robots.txt for AI crawlers guide.

Quick Reference: Complete robots.txt Directives

Copy-paste ready robots.txt block with the recommended configuration for maximum AI search visibility while blocking training crawlers:

# ============================================
# AI SEARCH BOTS — MUST ALLOW
# ============================================
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: bingbot
Allow: /

# ============================================
# USER-TRIGGERED FETCHERS — MUST ALLOW
# ============================================
User-agent: ChatGPT-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

# ============================================
# TRAINING / INDEXING BOTS — OPTIONAL BLOCK
# ============================================
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: cohere-ai
Disallow: /

Important: robots.txt directives are processed in order of specificity, not position. Always test your configuration after changes. Our guide on verifying AI crawler access walks through the testing process.

For sites using CDNs (Cloudflare, Fastly, Akamai), additional configuration may be needed to ensure AI bots are not blocked at the edge layer before they even reach your robots.txt. See CDN configuration for AI bots.

How to Identify AI Bot Traffic in Server Logs

Your server access logs contain every request from AI bots. Analyzing them reveals which bots are crawling your site, how often, and which pages they visit. Here is how to find AI bot traffic in common log formats.

Apache/Nginx access log patterns

AI bots identify themselves in the User-Agent field of each request. Look for these strings in your logs:

# Search for all AI bot requests in an Apache/Nginx access log
grep -E "(OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|Perplexity-User|ClaudeBot|Claude-User|Google-Extended|Applebot-Extended|CCBot|Bytespider|meta-externalagent|Amazonbot|cohere-ai)" /var/log/nginx/access.log

Key metrics to extract

| Metric | What It Tells You | How to Extract | |---|---|---| | Requests per bot per day | Crawl frequency by platform | Count log entries grouped by User-Agent and date | | Pages most frequently crawled | Which content AI prioritizes | Group by requested URL, filter by AI User-Agents | | Response codes | Whether bots can access your content | Filter for 403, 429, 503 responses to AI User-Agents | | Crawl time patterns | When bots are most active | Aggregate requests by hour | | New User-Agent strings | Emerging bots not yet in your robots.txt | Filter for unknown agents hitting multiple pages |

What to look for

• 403 or 429 responses to search bots — Your server or CDN is blocking bots you intended to allow. This is common with aggressive rate-limiting or WAF rules. See CDN configuration for AI bots.
• High crawl volume from Bytespider — This bot is notoriously aggressive. If it is consuming significant server resources, add a Crawl-delay or block it.
• Unknown User-Agent strings with AI-related keywords — New bots appear regularly. Log any unfamiliar agents and verify them with reverse DNS.
• Zero requests from search bots — If OAI-SearchBot or PerplexityBot never appear in your logs, your robots.txt may be blocking them, or your site has not been discovered yet.

How to Verify Bot Authenticity (Reverse DNS)

Anyone can set their User-Agent string to "GPTBot" or "OAI-SearchBot." Before whitelisting IPs or making access decisions based on User-Agent alone, verify that the bot is genuinely from the claimed company using reverse DNS lookup.

Step-by-step verification

Step 1: Get the bot's IP address from your server logs.

Step 2: Run a reverse DNS lookup:

host 23.98.142.176
# Expected result for OpenAI: 176.142.98.23.in-addr.arpa domain name pointer oai-search-bot.openai.com.

Step 3: Run a forward DNS lookup on the resolved hostname:

host oai-search-bot.openai.com
# Should resolve back to 23.98.142.176

Step 4: Confirm the domain matches the expected company domain.

Known verified domains by company

| Company | Bot Names | Verified Reverse DNS Domain | |---|---|---| | OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User | *.openai.com | | Anthropic | ClaudeBot, Claude-User | *.anthropic.com | | Perplexity | PerplexityBot, Perplexity-User | *.perplexity.ai | | Google | Google-Extended, Googlebot | *.googlebot.com or *.google.com | | Microsoft | Bingbot | *.search.msn.com | | Apple | Applebot-Extended, Applebot | *.applebot.apple.com | | ByteDance | Bytespider | *.bytedance.com | | Meta | meta-externalagent, FacebookExternalHit | *.facebook.com or *.meta.com | | Amazon | Amazonbot | *.amazonbot.amazon |

If the reverse DNS does not resolve to the expected domain, or the forward lookup does not match, the request is likely from a scraper spoofing the User-Agent. For a complete walkthrough with edge cases, see our verifying AI crawler access guide.

New Bots to Watch in 2026

The AI bot landscape continues to expand. These are the bots and developments to monitor in 2026:

Confirmed new entrants

• Apple Intelligence expansion — Apple is expanding Applebot-Extended's scope beyond Safari. Expect it to power AI features in Mail, Notes, and other first-party apps. Its crawl volume is increasing quarter-over-quarter.
• Grok crawler (xAI) — xAI (Elon Musk's AI company) has begun deploying a web crawler for Grok's search features. User-Agent string is not yet standardized; monitor your logs for crawlers resolving to *.x.ai or *.xai.com domains.
• Mistral AI crawler — Mistral, the European AI company, is building out web search capabilities for Le Chat. Watch for a User-Agent containing MistralBot or similar.

Trends to prepare for

1. Bot fragmentation — AI companies are splitting single bots into multiple specialized User-Agents (as OpenAI did with GPTBot, OAI-SearchBot, and ChatGPT-User). Expect Anthropic and Perplexity to follow this pattern.
2. Real-time verification standards — An industry initiative is developing standardized bot verification methods beyond reverse DNS, including signed request headers and public key verification.
3. Regulatory pressure in the EU — The EU AI Act may require AI companies to identify their crawlers more transparently. This could lead to new mandatory User-Agent fields or metadata.
4. Increased crawl frequency — As AI search tools become primary interfaces, search bots will crawl more frequently. Sites with fast response times and clean HTML will be favored over slow, JavaScript-heavy pages.

How to stay current

Review your robots.txt and server logs quarterly. New bots typically appear with little advance notice — OpenAI announced OAI-SearchBot only days before it began crawling at scale. AImetrico's monitoring tracks new AI bot activity across thousands of sites and can alert you when an unrecognized crawler appears.