Definition
Robots.txt is a plain text file placed in a website's root directory (e.g., https://example.com/robots.txt) that instructs search engine and AI crawlers which pages or sections of the site they are allowed or disallowed to access. It follows the Robots Exclusion Protocol (REP), a voluntary standard used since 1994. In the AI era, robots.txt has become the primary mechanism for controlling which AI bots can access your content.
Why It Matters
Robots.txt is one of the most critical files on your website for both traditional SEO and AI visibility. A misconfigured robots.txt can make your entire site invisible to search engines, AI assistants, or both.
- Controls AI crawler access. Robots.txt is the #1 reason websites are invisible to AI. Many sites accidentally block ALL bots — including AI search bots that would otherwise cite their content in ChatGPT, Gemini, and Perplexity responses.
- Separates search from training. Modern AI companies use different bots for search (retrieving content to answer user queries) and training (collecting data to improve models). Robots.txt allows you to permit search bots while blocking training bots — giving you visibility without sacrificing control over your data.
- Protects sensitive areas. Robots.txt prevents crawlers from accessing admin panels, staging environments, internal search results, and other pages that should not appear in search results or AI answers.
- Manages crawl budget. By blocking crawlers from low-value pages (filters, duplicates, pagination), robots.txt directs crawl resources toward your most important content.
- Industry standard. All major search engines and AI companies respect robots.txt. Google, Bing, OpenAI, Anthropic, and Perplexity have publicly committed to honoring these directives.
The most common AI SEO problem we see is a robots.txt file that blocks AI search bots. For a step-by-step configuration guide, see robots.txt for AI crawlers.
How It Works
Robots.txt works through a simple syntax of rules that apply to specific user agents (crawlers). When a crawler visits your site, it checks robots.txt first before accessing any other page.
Basic syntax
A robots.txt file consists of one or more rule groups, each targeting a specific user agent:
User-agent: Googlebot
Allow: /
Disallow: /admin/
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
- User-agent — Identifies which crawler the rules apply to. Use
*to apply rules to all crawlers. - Allow — Explicitly permits access to a URL path.
- Disallow — Blocks access to a URL path.
- Sitemap — Points crawlers to your XML sitemap location.
Key AI crawler user agents
Understanding which bots do what is essential for proper configuration:
- OAI-SearchBot — OpenAI's search bot (for ChatGPT search results). You want this ALLOWED.
- ChatGPT-User — ChatGPT browsing when a user asks it to visit a URL. You want this ALLOWED.
- GPTBot — OpenAI's training data crawler. Block if you do not want content used for training.
- PerplexityBot — Perplexity's search crawler. You want this ALLOWED.
- Google-Extended — Google's AI training crawler (for Gemini training). Block if desired.
- Googlebot — Google's main search crawler. Always allow for traditional SEO.
- ClaudeBot — Anthropic's crawler. Allow for Claude visibility.
- Bytespider — TikTok/ByteDance crawler. Block or allow based on preference.
For a complete and current list, see our AI crawler bots list for 2026.
Common mistakes
- Blocking everything with a wildcard. A
User-agent: * / Disallow: /rule blocks ALL crawlers, including search engines and AI bots. This makes your site invisible everywhere. - Not distinguishing search from training bots. Blocking
GPTBot(training) also blocks your content from ChatGPT's training data, but it does NOT affect ChatGPT's ability to search your site — that usesOAI-SearchBot. Many site owners confuse these. - Forgetting the file location. Robots.txt must be at the root of your domain:
https://example.com/robots.txt. Placing it in a subdirectory has no effect. - Using robots.txt for security. Robots.txt is publicly readable — anyone can view it. It should never be used to hide sensitive content. Use authentication, server-side access controls, or
noindexmeta tags instead. - No robots.txt at all. While technically this allows all bots, it is a missed opportunity to manage AI crawler access strategically.
Robots.txt vs. meta robots vs. X-Robots-Tag
These three mechanisms serve related but distinct purposes:
- Robots.txt — Controls whether a crawler can access a page at all (site-level, pre-crawl).
- Meta robots tag — Controls whether a crawler should index a page or follow its links (page-level, post-crawl). Example:
<meta name="robots" content="noindex, nofollow">. - X-Robots-Tag — Same function as meta robots but delivered via HTTP header. Useful for non-HTML files.
For comprehensive AI crawler management, you typically need all three working together.
Frequently Asked Questions
Does robots.txt block AI crawlers like ChatGPT?
It depends on your configuration. AI crawlers like OAI-SearchBot (ChatGPT search), PerplexityBot, and ChatGPT-User respect robots.txt directives. If your file blocks these user agents, your content will be invisible to those AI platforms. Many websites accidentally block all AI bots with broad Disallow rules. You should explicitly allow search bots while selectively blocking training bots.
Can robots.txt prevent my content from being used for AI training?
Partially. Blocking training-specific bots like GPTBot and Google-Extended prevents those crawlers from accessing your content for model training. However, robots.txt is a voluntary protocol — it works because major AI companies have committed to respecting it, but it is not enforceable. For stronger protection, combine robots.txt with terms of service, licensing headers, and server-side access controls.
What happens if I don't have a robots.txt file?
If no robots.txt file exists, all crawlers assume full permission to access every page on your site. For most sites this is harmless, but it means you have no mechanism to control AI bot access. Creating a properly configured robots.txt is a best practice for managing which bots can reach which content — especially important in the AI era.
Is your robots.txt blocking AI?
The #1 cause of AI invisibility is a misconfigured robots.txt. Check yours in 60 seconds.
Free scan -- No signup required