HTTP Headers That Affect AI Indexing: X-Robots-Tag, Cache-Control, and More

Why HTTP Headers Matter for AI Indexing

Every time an AI crawler requests a page from your server, the server responds with two things: the page content and a set of HTTP headers. These headers are metadata instructions that tell the crawler how to handle the content -- whether to index it, how long to cache it, what language it is in, and what format to expect.

Most AI SEO discussions focus on robots.txt and on-page content. Headers are often overlooked, but they operate at a layer that takes priority over everything else. A perfectly optimized page with excellent Schema markup and citation-friendly content will never appear in AI responses if your server sends an X-Robots-Tag: noindex header for that URL.

The headers covered in this guide are relevant to all major AI crawlers, including OAI-SearchBot (ChatGPT), Googlebot (Gemini), PerplexityBot, ClaudeBot, and Bingbot (Copilot). For a complete list of AI crawler user agents and their behaviors, see our AI crawler bots list for 2026.

X-Robots-Tag: Controlling AI Indexing Per Page

The X-Robots-Tag HTTP header is the most direct way to control whether AI crawlers index your content. It works exactly like the <meta name="robots"> HTML tag, but at the HTTP header level -- which means it applies to all file types, not just HTML pages.

Basic syntax

X-Robots-Tag: noindex
X-Robots-Tag: nofollow
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: noarchive

Bot-specific directives

This is where X-Robots-Tag becomes especially powerful for AI SEO. You can target individual crawlers:

X-Robots-Tag: OAI-SearchBot: noindex
X-Robots-Tag: GPTBot: noindex, nofollow
X-Robots-Tag: PerplexityBot: all
X-Robots-Tag: ClaudeBot: noindex

This lets you allow ChatGPT's search bot to index your content while blocking the GPT training bot, for example. You can send multiple X-Robots-Tag headers in a single response -- each targeting a different bot.

Key directives

| Directive | Effect on AI Crawlers | |---|---| | all | Allow indexing and link following (default) | | noindex | Do not index this page -- it will not appear in AI responses | | nofollow | Do not follow links on this page to discover other content | | noarchive | Do not cache or store a copy of this page | | nosnippet | Do not use any content from this page in AI-generated snippets | | max-snippet:0 | Equivalent to nosnippet -- prevents content extraction |

Why X-Robots-Tag matters more than meta robots for AI

The HTML meta robots tag only works on HTML pages. But AI crawlers also fetch your PDFs, images, JSON-LD files, XML sitemaps, and API endpoints. The X-Robots-Tag header applies to all of these. If you have PDF whitepapers that you want AI to index, or product JSON feeds you want to keep private, X-Robots-Tag is your only option at the HTTP level.

Cache-Control, Last-Modified, and ETag: Content Freshness

AI crawlers use caching headers to decide how often to re-fetch your content. If your caching configuration tells bots your page is valid for 30 days, they may not check for updates during that entire period -- meaning your content corrections, price updates, or new information could take weeks to appear in AI responses.

Cache-Control

Cache-Control: public, max-age=3600
Cache-Control: no-cache
Cache-Control: no-store
Cache-Control: public, max-age=86400, stale-while-revalidate=3600

Recommended settings for AI-indexed pages:

• Frequently updated content (news, prices, inventory): Cache-Control: public, max-age=300 (5 minutes) or no-cache with ETag validation
• Standard content (blog posts, guides): Cache-Control: public, max-age=3600 (1 hour)
• Static content (documentation, evergreen guides): Cache-Control: public, max-age=86400 (24 hours)
• Private content (dashboards, user data): Cache-Control: private, no-store

Avoid max-age=31536000 (1 year) on any content you want AI to re-index. This is fine for static assets like CSS and images, but it will freeze your page content in AI systems.

Last-Modified

Last-Modified: Sat, 22 Mar 2026 10:00:00 GMT

The Last-Modified header tells AI crawlers when the page was last changed. Crawlers use this to make conditional requests with If-Modified-Since -- if the page has not changed, the server returns a 304 Not Modified response, saving bandwidth and crawl budget for both sides.

Always set Last-Modified to the actual date your content was last substantively updated. Some CMS platforms set it to the current server time on every request, which defeats its purpose entirely and wastes AI crawler resources.

ETag

ETag: "a1b2c3d4e5f6"

An ETag (Entity Tag) is a unique fingerprint for a specific version of your content. When a crawler re-fetches a page, it sends the previous ETag in an If-None-Match header. If the content has not changed, the server responds with 304 Not Modified.

ETags are more reliable than Last-Modified because they detect any content change, not just timestamp differences. For pages that change frequently, use a strong ETag based on a content hash rather than a weak ETag based on file size and modification time.

Last-Modified and ETag work together. When both are present, well-behaved crawlers (including all major AI bots) use ETag as the primary validator and Last-Modified as a fallback.

Content-Type and Content-Language: Helping AI Parse Correctly

Content-Type

Content-Type: text/html; charset=utf-8
Content-Type: application/json; charset=utf-8
Content-Type: application/pdf

The Content-Type header tells AI crawlers what format your content is in and what character encoding it uses. This seems basic, but misconfigured Content-Type headers cause real problems:

• Serving HTML with Content-Type: text/plain causes crawlers to treat your page as raw text, ignoring all semantic HTML structure and Schema markup
• Missing charset=utf-8 can cause character encoding issues, especially for non-English content with diacritical marks or special characters
• Serving JSON-LD endpoints with text/html instead of application/json may prevent AI crawlers from parsing your structured data correctly

Always include the charset parameter. The correct Content-Type for HTML pages is text/html; charset=utf-8.

Content-Language

Content-Language: en
Content-Language: en-US
Content-Language: de, en

The Content-Language header declares the language of your page content. This is separate from the HTML lang attribute -- both should be set, and both should match.

For AI indexing, Content-Language serves two purposes:

1. Query matching -- AI models use language signals to match your content with queries in the same language. A page with Content-Language: en will be prioritized for English-language queries over an unlabeled page with identical content.
2. Multi-language disambiguation -- If your site serves the same URL with different language content based on user location (which you should generally avoid), Content-Language tells crawlers which version they received.

How to Check Your HTTP Headers

Before configuring headers, you need to know what your server is currently sending. Here are three methods, from simplest to most thorough.

Method 1: curl (command line)

The fastest way to inspect headers. Open a terminal and run:

curl -I https://yoursite.com

This sends a HEAD request and displays only the response headers:

HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=3600
last-modified: Sat, 22 Mar 2026 10:00:00 GMT
etag: "a1b2c3d4"
x-robots-tag: all
content-language: en

To see headers as a specific AI crawler would:

curl -I -H "User-Agent: OAI-SearchBot" https://yoursite.com
curl -I -H "User-Agent: PerplexityBot" https://yoursite.com
curl -I -H "User-Agent: ClaudeBot" https://yoursite.com

This is critical. Some servers and CDN configurations serve different headers based on the User-Agent string. You may be sending X-Robots-Tag: noindex to AI bots without realizing it.

Method 2: Browser DevTools

1. Open your site in Chrome or Firefox
2. Press F12 to open DevTools
3. Go to the Network tab
4. Reload the page
5. Click the first request (your page URL)
6. Check the Response Headers section

This shows you exactly what headers the browser received. Note that browser requests use a different User-Agent than AI crawlers, so the headers may differ from what bots see.

Method 3: Online tools

For a quick check without a terminal, use tools like securityheaders.com or httpstatus.io. Enter your URL and they will display all response headers along with security recommendations. However, these tools typically use their own User-Agent, so they may not reflect what AI crawlers actually receive.

Server Configuration Examples

Here are ready-to-use configurations for the four most common server environments. Each example sets the recommended headers for AI-indexed content.

Apache (.htaccess)

# Allow AI indexing, set caching and language
<IfModule mod_headers.c>
    # Default: allow all bots to index
    Header set X-Robots-Tag "all"

    # Block GPTBot (training) while allowing OAI-SearchBot (search)
    <If "%{HTTP_USER_AGENT} =~ /GPTBot/">
        Header set X-Robots-Tag "noindex, nofollow"
    </If>

    # Cache-Control for standard content pages
    Header set Cache-Control "public, max-age=3600"

    # Content-Language
    Header set Content-Language "en"
</IfModule>

# Ensure correct Content-Type with charset
AddDefaultCharset UTF-8

Nginx

server {
    # Default headers for all HTML content
    location / {
        add_header X-Robots-Tag "all" always;
        add_header Cache-Control "public, max-age=3600" always;
        add_header Content-Language "en" always;

        # Block training bots at header level
        if ($http_user_agent ~* "GPTBot") {
            add_header X-Robots-Tag "noindex, nofollow" always;
        }
    }

    # Shorter cache for frequently updated pages
    location /blog/ {
        add_header Cache-Control "public, max-age=300" always;
        add_header X-Robots-Tag "all" always;
    }

    # Private areas - no AI indexing
    location /dashboard/ {
        add_header X-Robots-Tag "noindex, nofollow" always;
        add_header Cache-Control "private, no-store" always;
    }
}

Vercel (vercel.json)

{
  "headers": [
    {
      "source": "/(.*)",
      "headers": [
        { "key": "X-Robots-Tag", "value": "all" },
        { "key": "Cache-Control", "value": "public, max-age=3600" },
        { "key": "Content-Language", "value": "en" }
      ]
    },
    {
      "source": "/api/(.*)",
      "headers": [
        { "key": "X-Robots-Tag", "value": "noindex" },
        { "key": "Cache-Control", "value": "private, no-store" }
      ]
    }
  ]
}

Cloudflare (Transform Rules)

In the Cloudflare dashboard, navigate to Rules > Transform Rules > Modify Response Header:

1. Rule 1 -- Default AI headers: Match URI Path contains / then Set X-Robots-Tag to all
2. Rule 2 -- Block training bots: Match User Agent contains GPTBot then Set X-Robots-Tag to noindex, nofollow
3. Rule 3 -- Private routes: Match URI Path starts with /dashboard then Set X-Robots-Tag to noindex, nofollow

Cloudflare Transform Rules execute after your origin server headers, so they can override origin settings. Be aware that Cloudflare's Bot Fight Mode and Super Bot Fight Mode can independently block AI crawlers regardless of your header configuration. Check your Cloudflare Security settings and review our CDN configuration for AI bots guide for detailed setup instructions.

Common Header Mistakes That Block AI

Based on analysis of thousands of websites, these are the most frequent header misconfigurations that silently prevent AI indexing:

1. Blanket X-Robots-Tag: noindex on all non-HTML resources

Some security-focused server configurations add X-Robots-Tag: noindex to all responses except HTML pages. This blocks AI crawlers from indexing your PDFs, images, and structured data files. AI models like Perplexity and ChatGPT regularly cite PDFs and reference images -- blocking these resources limits your visibility.

2. Aggressive Cache-Control preventing content updates

Setting Cache-Control: public, max-age=2592000 (30 days) on content pages means AI crawlers may not re-fetch your updated content for a month. If you fix an error, update pricing, or add new information, AI models will continue serving the stale version.

3. Missing Content-Type charset

Serving Content-Type: text/html without charset=utf-8 can cause encoding issues. AI crawlers may misinterpret special characters, breaking your content in AI responses. This is especially common on older Apache servers with default configurations.

4. CDN stripping Last-Modified and ETag headers

Some CDN configurations strip Last-Modified and ETag headers from responses to reduce header size. Without these headers, AI crawlers cannot make conditional requests -- they must download the full page every time, which wastes your crawl budget and may cause the crawler to reduce its crawl frequency for your site.

5. User-Agent-based blocking via headers

Firewall or WAF rules that return 403 Forbidden or X-Robots-Tag: noindex for unrecognized User-Agent strings will block new AI crawlers as they emerge. Rather than allowlisting specific bots, consider blocklisting only the bots you explicitly want to exclude.

6. Conflicting headers and meta tags

If your HTML contains <meta name="robots" content="all"> but your server sends X-Robots-Tag: noindex, the more restrictive directive wins. Crawlers follow the strictest instruction from any source. Audit both your HTML meta tags and HTTP headers to ensure they align.

Headers vs robots.txt: When to Use Which

HTTP headers and robots.txt both control how AI crawlers interact with your site, but they operate at different levels and serve different purposes. Using the wrong mechanism for a task leads to either gaps in control or unnecessarily blocked content.

| Scenario | Use robots.txt | Use HTTP Headers | |---|---|---| | Block an entire directory from crawling | Yes | No -- headers require the page to be fetched first | | Block indexing of a specific page type (PDFs, images) | No -- robots.txt blocks crawling, not indexing | Yes -- X-Robots-Tag: noindex | | Allow crawling but prevent caching | No | Yes -- Cache-Control: no-store | | Block a specific AI bot from all pages | Yes -- most efficient | Possible but requires per-request logic | | Control indexing per file type | Limited | Yes -- apply X-Robots-Tag by file extension | | Set content freshness signals | No | Yes -- Cache-Control, Last-Modified, ETag | | Works on non-HTML resources | Yes (blocks access) | Yes (controls indexing behavior) |

The general rule: Use robots.txt for broad crawl access control (which bots can fetch which URLs). Use HTTP headers for fine-grained indexing and caching behavior (what bots do with content after fetching it).

For a complete AI SEO technical setup, you need both. Your robots.txt configuration handles the "who can enter" question. Your HTTP headers handle the "what can they do once inside" question. Together with structured data and content optimization, they form the technical foundation described in the AI SEO checklist for 2026.