Definition
Noindex is a directive used in a meta robots tag or X-Robots-Tag HTTP header that tells search engines not to include a specific page in their search index. A noindexed page will not appear in search results, even though crawlers can still access and read its content. It is one of the most important tools for controlling which pages from your site appear in search engines and, indirectly, in AI-generated answers.
Why It Matters
Every website has pages that should not appear in search results. Internal search result pages, login screens, thank-you pages, thin tag archives — these dilute your site's quality in the eyes of search engines and waste crawl budget.
Noindex matters because:
- Prevents index bloat. Large sites can have thousands of low-value pages (pagination, filters, parameter variations) that clutter search indexes. Noindexing these pages keeps your index clean and focused on high-value content.
- Protects site quality signals. Search engines evaluate your site's overall quality. Having hundreds of thin or duplicate pages indexed can lower your site's perceived quality, dragging down the ranking potential of your good pages.
- Saves crawl budget. While noindex pages are still crawled (the crawler needs to see the tag), they do not consume indexing resources. Combined with strategic robots.txt rules, noindex helps direct crawler attention to pages that matter.
- Controls AI source material. Pages that are noindexed in Google are less likely to be surfaced as sources for AI-generated answers, since many AI retrieval systems draw from search engine indexes. This gives you indirect control over what AI models can cite.
- Prevents sensitive pages from appearing in results. Staging environments, internal tools, and private pages should never show up in search — noindex ensures they do not.
For a broader understanding of technical optimization, see our technical SEO overview.
How It Works
Noindex can be implemented in two ways, each appropriate for different scenarios.
Method 1: Meta robots tag (HTML)
Add the following tag to the <head> section of any HTML page you want de-indexed:
<meta name="robots" content="noindex">
You can combine noindex with other directives:
noindex, follow— Do not index this page, but follow its links to discover other pages.noindex, nofollow— Do not index this page and do not follow any of its links.noindex, nosnippet— Do not index and do not show a snippet if the page somehow appears in results.
You can also target specific crawlers:
<meta name="googlebot" content="noindex">
<meta name="bingbot" content="noindex">
Method 2: X-Robots-Tag (HTTP header)
For non-HTML content (PDFs, images) or when you cannot modify the HTML, use the X-Robots-Tag HTTP header:
X-Robots-Tag: noindex
This is configured at the server level (Apache, Nginx) or through your application's response headers.
When to use noindex
Noindex is appropriate for:
- Internal search results pages — These create near-infinite URL variations with thin content.
- Thank-you and confirmation pages — Post-form-submission pages that offer no search value.
- Tag and category archives with thin content — Archive pages that merely list post titles without substantial content.
- Paginated pages — Pages 2, 3, 4+ of paginated content (though canonical tags are often a better solution).
- Staging and test environments — Development sites that should never appear in production search results.
- Duplicate content — When canonical tags are insufficient or not practical.
- Login and admin pages — Pages that serve no purpose in search results.
When NOT to use noindex
Do not apply noindex to:
- Pages you want to rank in Google or Bing
- Pages you want AI models to cite in their answers
- Pages that are already blocked in robots.txt (the crawler cannot see the noindex tag if it cannot access the page)
- Your homepage, key landing pages, or product pages
Noindex vs. robots.txt vs. canonical
These three tools serve different purposes and are not interchangeable:
- Noindex — Allows crawling but prevents indexing. The page is read but not included in search results.
- Robots.txt Disallow — Prevents crawling entirely. The page is never read. Learn more in our guide on robots.txt for AI crawlers.
- Canonical — Tells search engines which version of a duplicate page is preferred. The canonical page is indexed; duplicates may be ignored.
A common mistake is blocking a page in robots.txt and expecting it to be de-indexed. If crawlers cannot access the page, they cannot see the noindex tag — and the page may remain in the index indefinitely, showing a "URL is known but not crawled" status.
Frequently Asked Questions
What is the difference between noindex and robots.txt Disallow?
Robots.txt Disallow prevents crawlers from accessing a page entirely. Noindex allows crawlers to read the page but instructs them not to include it in search results. A critical nuance: if you block a page with robots.txt, crawlers cannot see the noindex tag on that page, so the URL may remain in the index with limited information. For reliable de-indexing, use noindex and ensure the page is accessible to crawlers.
Does noindex prevent AI models from using my content?
Not entirely. Noindex prevents pages from appearing in traditional search indexes, but AI crawlers performing real-time web retrieval may still access noindexed pages if robots.txt allows it. The noindex directive is primarily respected by search engine indexers. To fully prevent AI access, combine noindex with robots.txt Disallow rules targeting AI-specific user agents.
When should I use noindex?
Apply noindex to pages that have no business appearing in search results: thank-you pages, internal search results, thin tag archives, paginated pages beyond page 1, staging environments, duplicate content, and login pages. Never noindex pages you want to rank or have cited by AI. Review your indexed pages regularly in Google Search Console to catch accidental noindex tags on important content.
Are important pages accidentally noindexed?
Misplaced noindex tags can make your best content invisible. Scan your site in 60 seconds.
Free scan -- No signup required