Key Takeaways
- The meta robots tag controls what AI crawlers do with your HTML pages after fetching them — noindex prevents indexing, nofollow prevents link following
- You can target specific AI bots with
<meta name="GPTBot" content="noindex">to apply directives to one crawler without affecting others - Do not add
index, followexplicitly — these are defaults. Only add meta robots when you want to restrict behavior - The
max-snippetdirective can limit how much text AI models quote from your pages, but support varies across AI platforms - A CMS accidentally adding noindex is a common cause of AI invisibility — always audit your meta robots tags after CMS or plugin updates
Hidden noindex tags killing your AI visibility? Run a free scan to check for meta robots issues across your entire site.
Table of Contents
What Is the Meta Robots Tag?
The meta robots tag is an HTML element placed in the <head> section of a web page that tells crawlers how to handle that specific page. It looks like this:
<meta name="robots" content="noindex, nofollow">
When a crawler fetches an HTML page, it reads the meta robots tag and follows the directives. The name attribute specifies which crawlers the directive applies to (robots means all crawlers), and the content attribute contains one or more comma-separated directives.
Unlike robots.txt which controls whether a crawler can access a page, meta robots controls what the crawler does after accessing it. A crawler must fetch the page to read the meta robots tag — which means the server still processes the request and sends the response.
For non-HTML files (PDFs, images, APIs), the equivalent is the X-Robots-Tag HTTP header, which provides the same functionality through response headers.
Available Directives
Core directives
| Directive | Effect | AI SEO Impact |
|---|---|---|
| index | Allow indexing (default) | Page can appear in AI responses |
| noindex | Prevent indexing | Page will NOT appear in AI responses |
| follow | Follow links (default) | AI can discover linked pages |
| nofollow | Don't follow links | AI won't discover pages linked from here |
| none | Shorthand for noindex, nofollow | Page invisible + links not followed |
| all | Shorthand for index, follow | Same as not having a meta robots tag |
Extended directives
| Directive | Effect | AI SEO Impact |
|---|---|---|
| nosnippet | No text snippets in results | May limit AI quoting ability |
| max-snippet:N | Limit snippet to N characters | Controls quote length in AI responses |
| noarchive | No cached copy stored | Affects AI models using cached content |
| noimageindex | Don't index images on this page | Images won't appear in AI image responses |
Syntax examples
<meta name="robots" content="noindex">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="max-snippet:160">
<meta name="robots" content="noindex, nofollow, noarchive">
How AI Crawlers Handle Meta Robots
Not all AI crawlers handle meta robots identically. Here is what we know about each major platform:
| AI Crawler | Respects noindex | Respects nofollow | Respects nosnippet | Respects max-snippet | |---|---|---|---|---| | Googlebot (AI Mode) | Yes | Yes | Yes | Yes | | GPTBot | Yes | Yes | Unclear | Unclear | | OAI-SearchBot | Yes | Yes | Unclear | Unclear | | PerplexityBot | Yes | Yes | Unclear | Unclear | | ClaudeBot | Yes | Yes | Unclear | Unclear | | Bingbot (Copilot) | Yes | Yes | Yes | Yes |
The core directives (noindex, nofollow) are universally respected by all major AI crawlers. The extended directives (nosnippet, max-snippet) have inconsistent support outside of Google and Bing.
The takeaway: Use noindex confidently to prevent AI indexing. Use nosnippet and max-snippet as additional signals, but do not rely on them as your only control mechanism.
Bot-Specific Meta Tags
You can target individual AI crawlers by changing the name attribute from robots (all crawlers) to a specific bot name:
<meta name="GPTBot" content="noindex">
<meta name="googlebot" content="nosnippet">
<meta name="PerplexityBot" content="noindex">
Combining universal and bot-specific tags
You can use multiple meta tags to create layered rules:
<meta name="robots" content="index, follow">
<meta name="GPTBot" content="noindex">
In this example, all crawlers except GPTBot will index the page. GPTBot reads its own specific directive and obeys noindex.
Practical scenario: Block training, allow search
While you cannot distinguish training from search indexing using meta robots alone, you can block specific bots known to be used for training:
<meta name="CCBot" content="noindex">
For access-level control over training bots, use robots.txt instead.
CMS and Plugin Considerations
Content management systems and SEO plugins are the most common source of accidental noindex tags.
WordPress + Yoast SEO
Yoast adds a per-page "Advanced" tab where you can set noindex. Additionally, WordPress has a global "Discourage search engines from indexing this site" setting under Settings > Reading. When enabled, it adds <meta name="robots" content="noindex, nofollow"> to every page.
Check: Settings > Reading > "Discourage search engines from indexing this site" must be unchecked on production.
WordPress + Rank Math
Rank Math adds similar per-page controls. Check the "Robots Meta" section in the Rank Math meta box for each important page.
Shopify
Shopify adds noindex to pages with the meta-robots-noindex Liquid tag and to paginated collection URLs by default. Check your theme's layout/theme.liquid for blanket noindex rules.
Custom CMS platforms
Any CMS that generates HTML head content can introduce noindex tags. After every CMS update, plugin installation, or theme change, verify your meta robots output.
The max-snippet Directive and AI
The max-snippet directive limits how many characters of your content can appear as a text snippet:
<meta name="robots" content="max-snippet:200">
For traditional search, this controls the length of the description snippet in SERPs. For AI search, it could theoretically limit how much of your content an AI model quotes in its response.
Current reality: Google's AI Mode respects max-snippet. Other AI platforms do not consistently document support. If you want AI to quote your content freely (which is generally the goal in AI SEO), do not add max-snippet restrictions. If you need to limit quoting for legal or paywall reasons, max-snippet provides partial protection.
Auditing Your Meta Robots Tags
Manual check with View Source
Open any page, view the HTML source, and search for <meta name="robots" or <meta name="GPTBot". Any restrictive directives (noindex, nofollow, nosnippet) will prevent AI visibility for that page.
Automated check with curl
curl -s https://yoursite.com/ | grep -i "meta.*robots"
Run this against your most important pages: homepage, key product pages, blog posts, and landing pages.
Site-wide audit
For a comprehensive audit, crawl your site with a tool like Screaming Frog and filter for pages with noindex meta robots tags. Compare the list against your intended indexing policy — any public content page with noindex is a potential AI visibility problem.
Best Practices for AI SEO
-
Do not add meta robots to public content — The default (index, follow) is what you want. Only add meta robots when you need to restrict something.
-
Use noindex sparingly — Only for pages that genuinely should not appear in AI responses: internal admin pages, duplicate content, staging previews, thank-you pages.
-
Check after every CMS update — Plugin updates, theme changes, and CMS migrations are the top causes of accidental noindex.
-
Prefer robots.txt for access control — If you want to block a bot entirely, use robots.txt. Use meta robots only when you want to allow access but prevent indexing.
-
Test with AI-specific user agents — Some CMS configurations serve different meta robots tags to different user agents. Test with GPTBot and PerplexityBot user agents specifically.
Frequently Asked Questions
Does the meta robots noindex tag affect AI crawlers?
Yes. Major AI crawlers including GPTBot, OAI-SearchBot, and Googlebot respect noindex. A page with noindex will not appear in AI-generated responses.
Can I use different meta robots directives for different AI bots?
Yes. Use <meta name="GPTBot" content="noindex"> to target specific bots. Multiple meta tags can coexist for different crawlers.
What is the max-snippet directive and does it affect AI?
max-snippet limits text snippet length. Google AI Mode respects it; other AI platforms have inconsistent support. Do not use it unless you specifically need to limit quoting.
Should I add meta robots index, follow to my pages?
No. These are defaults. Only add meta robots when restricting behavior. Adding index, follow explicitly provides no benefit.
Does nofollow in meta robots affect AI link following?
Nofollow prevents crawlers from following links on the page, limiting their ability to discover linked content. It does not prevent the page itself from being indexed.
Find hidden noindex tags in seconds
AImetrico scans your meta robots tags, X-Robots-Tag headers, and robots.txt in one free audit.
Trusted by 2,400+ websites