X-Robots-Tag: Advanced Crawler Control

Key Takeaways

X-Robots-Tag is an HTTP response header that gives crawler directives (noindex, nofollow, nosnippet) at the server level — it works for all file types, not just HTML
You can target specific AI crawlers by name: X-Robots-Tag: GPTBot: noindex affects only GPTBot while leaving other bots unaffected
X-Robots-Tag and robots.txt serve different purposes: robots.txt controls access (can the bot fetch the page?), while X-Robots-Tag controls indexing (what does the bot do with the content?)
Configure X-Robots-Tag in your web server config (Nginx, Apache) or through CDN edge rules — not in your HTML
For AI SEO, ensure you are not accidentally sending noindex to AI search crawlers — check your headers with curl -I

Are your HTTP headers blocking AI? Run a free header scan to check for noindex directives that could prevent AI citation.

What Is X-Robots-Tag?
X-Robots-Tag vs Meta Robots vs Robots.txt
Supported Directives
Targeting Specific AI Crawlers
Server Configuration Examples
Common AI SEO Use Cases
Testing and Verification
Common Mistakes
FAQ

What Is X-Robots-Tag?

The X-Robots-Tag is an HTTP response header that tells search engine crawlers and AI bots how to handle the content they have just fetched. It provides the same directives as the HTML meta robots tag — noindex, nofollow, nosnippet, and others — but operates at the HTTP header level instead of within the HTML document.

This distinction matters because X-Robots-Tag works for every type of file your server delivers. While meta robots tags can only be placed inside HTML pages, X-Robots-Tag applies to PDFs, images, JavaScript files, JSON responses, XML feeds, and any other content type. For sites serving structured data files, API responses, or downloadable documents, X-Robots-Tag is the only way to control crawler behavior for those resources.

When a crawler fetches a URL, it reads the HTTP response headers before processing the body. If it encounters an X-Robots-Tag header with a noindex directive, it will not add that content to its index — regardless of whether the content is an HTML page, a PDF, or an image.

X-Robots-Tag vs Meta Robots vs Robots.txt

These three mechanisms control different aspects of the crawler lifecycle. Understanding when to use each one is essential for AI SEO:

| Feature | Robots.txt | Meta Robots | X-Robots-Tag | |---|---|---|---| | Controls | Access (can bot fetch?) | Indexing (what to do with HTML?) | Indexing (what to do with any file?) | | File types | All URLs | HTML only | All file types | | Location | /robots.txt file | HTML <head> section | HTTP response header | | Configured in | Single text file | Page template/CMS | Server config or CDN | | Bot-specific rules | Yes (User-agent) | Limited (via name attribute) | Yes (bot-name prefix) | | Evaluated when | Before fetching | After fetching HTML | After fetching any resource |

For AI SEO, the key rule is: Use robots.txt to control which bots can access your content. Use X-Robots-Tag (or meta robots for HTML) to control what bots do with the content after accessing it.

For comprehensive HTTP header guidance, see our HTTP headers for AI indexing guide.

Supported Directives

X-Robots-Tag supports the following directives relevant to AI SEO:

noindex

X-Robots-Tag: noindex

Tells crawlers not to index this content. The content will not appear in search results or be used in AI-generated answers. This is the most impactful directive for AI visibility.

nofollow

X-Robots-Tag: nofollow

Tells crawlers not to follow links on this page. Does not prevent indexing of the page itself. Less relevant for AI SEO since AI crawlers are primarily interested in content, not link graphs.

nosnippet

X-Robots-Tag: nosnippet

Prevents search engines from showing text snippets from this page. For AI models, this may limit their ability to quote your content in responses, though not all AI platforms respect this directive.

noarchive

X-Robots-Tag: noarchive

Prevents crawlers from storing a cached copy. This can affect AI models that rely on cached versions of your content.

Combining directives

X-Robots-Tag: noindex, nofollow

Multiple directives can be combined in a single header, separated by commas.

Targeting Specific AI Crawlers

The most powerful feature of X-Robots-Tag is the ability to target specific crawlers. Prefix the directive with the bot name:

X-Robots-Tag: GPTBot: noindex
X-Robots-Tag: googlebot: nosnippet
X-Robots-Tag: PerplexityBot: noindex

Each line targets only the named crawler. All other crawlers ignore directives not addressed to them.

Practical AI SEO scenarios

Allow all AI search bots but block one:

X-Robots-Tag: CCBot: noindex

This blocks only the Common Crawl bot (used for training data) from indexing while allowing GPTBot, PerplexityBot, and others to index freely.

Block AI training while allowing AI search:

You cannot distinguish training from search indexing using X-Robots-Tag alone — this is better handled in robots.txt. X-Robots-Tag controls what happens after the content is fetched, not who can fetch it.

Server Configuration Examples

Nginx

Add X-Robots-Tag headers in your Nginx server block or location blocks:

# Allow indexing for all pages (default — no header needed)

# Block indexing for specific paths
location /internal/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# Block specific AI crawler from indexing PDFs
location ~* \.pdf$ {
    add_header X-Robots-Tag "CCBot: noindex" always;
}

# Block indexing for staging environments
server {
    server_name staging.yoursite.com;
    add_header X-Robots-Tag "noindex" always;
}

The always parameter ensures the header is sent even for error responses.

Apache

Use .htaccess or your Apache configuration:

# Block indexing for a directory
<Directory "/var/www/html/internal">
    Header set X-Robots-Tag "noindex, nofollow"
</Directory>

# Block indexing for PDFs by specific bot
<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "CCBot: noindex"
</FilesMatch>

# Block indexing for staging
<VirtualHost *:443>
    ServerName staging.yoursite.com
    Header set X-Robots-Tag "noindex"
</VirtualHost>

CDN edge rules

Cloudflare Transform Rules and similar CDN features can add X-Robots-Tag headers based on request properties. This is useful when you do not have direct access to your web server configuration.

Common AI SEO Use Cases

Protecting staging and development environments

Staging sites often get accidentally indexed by AI crawlers, leading to duplicate or incomplete content in AI responses:

X-Robots-Tag: noindex

Apply this to all responses on staging and dev environments.

Controlling PDF and document indexing

AI crawlers can and do index PDFs. If you have internal documents, draft reports, or outdated PDFs that should not appear in AI responses:

X-Robots-Tag: noindex

Apply this to specific document paths or file types.

Preventing snippet extraction

If you want AI crawlers to know your page exists but not quote its content:

X-Robots-Tag: nosnippet

This is rarely recommended for AI SEO — you generally want AI to quote your content — but it has applications for legally sensitive or paywalled material.

No restrictions (recommended for most content)

For the majority of your public content, do not send any X-Robots-Tag headers. The absence of restrictive headers is the best signal that AI crawlers are welcome to index and cite your content.

Testing and Verification

Check headers with curl

curl -I https://yoursite.com/

Look for any X-Robots-Tag header in the response. If you see X-Robots-Tag: noindex, that page is blocked from AI indexing.

Test specific pages

Check multiple page types — your homepage, blog posts, product pages, and any paths where server configuration might add restrictive headers:

# Check homepage
curl -I https://yoursite.com/ 2>/dev/null | grep -i x-robots

# Check a blog post
curl -I https://yoursite.com/blog/example-post 2>/dev/null | grep -i x-robots

# Check a PDF
curl -I https://yoursite.com/docs/example.pdf 2>/dev/null | grep -i x-robots

No output means no X-Robots-Tag is set — which is the desired state for content you want AI to index.

Check as a specific AI crawler

Some server configurations send different headers to different user agents:

curl -I -A "GPTBot/1.2" https://yoursite.com/ 2>/dev/null | grep -i x-robots

Common Mistakes

Blanket noindex on all pages — Some server configurations add X-Robots-Tag: noindex globally (often leftover from development). This makes your entire site invisible to AI. Always check production headers.
Forgetting the always keyword in Nginx — Without always, Nginx only sends the header for successful (2xx) responses, not for redirects or errors. AI crawlers need to see the header on all responses.
Conflicting meta robots and X-Robots-Tag — If your HTML meta robots says index but X-Robots-Tag says noindex, the most restrictive directive wins. Crawlers follow the noindex.
Not testing after server changes — Server configuration updates, CMS updates, or CDN changes can introduce or remove X-Robots-Tag headers. Always verify after changes.
Using X-Robots-Tag when robots.txt is more appropriate — If you want to prevent a bot from accessing content entirely (saving bandwidth and preventing any data collection), use robots.txt Disallow instead of X-Robots-Tag noindex.

Frequently Asked Questions

What is the X-Robots-Tag HTTP header?

The X-Robots-Tag is an HTTP response header that provides crawler directives at the server level. Unlike meta robots which only works in HTML, X-Robots-Tag works for any file type. It supports directives like noindex, nofollow, and nosnippet.

Do AI crawlers respect X-Robots-Tag?

Major AI search crawlers from OpenAI, Google, and Bing respect X-Robots-Tag directives. Perplexity and Anthropic also follow standard indexing directives. Combine X-Robots-Tag with robots.txt for comprehensive control.

What is the difference between X-Robots-Tag and meta robots?

X-Robots-Tag is an HTTP header set at the server level and works for all file types. Meta robots is an HTML tag in the page's head section and only works in HTML. Both support the same directives. See our meta robots guide for details on the HTML approach.

How do I set X-Robots-Tag for specific AI crawlers only?

Use the bot-specific syntax: X-Robots-Tag: GPTBot: noindex. This applies the directive only to GPTBot. Configure this in your web server configuration (Nginx, Apache) or CDN edge rules.

Can X-Robots-Tag prevent AI training on my content?

X-Robots-Tag: noindex prevents indexing, which limits use in AI responses. For explicit training control, combine it with robots.txt Disallow rules targeting training-specific bot user agents.

Does X-Robots-Tag override robots.txt?

No. Robots.txt controls access (fetching). X-Robots-Tag controls indexing (what to do after fetching). If robots.txt blocks a URL, the crawler never sees the X-Robots-Tag header.

Are your headers AI-friendly?

AImetrico checks X-Robots-Tag, meta robots, robots.txt, and 20+ other signals in one free scan.

Scan My Site Free

Trusted by 2,400+ websites