The llms.txt Specification Explained: A New Standard for AI

What Is llms.txt?

llms.txt is a Markdown file placed at the root of your website (yourdomain.com/llms.txt) that provides large language models with a structured, human-readable summary of your site. It tells AI who you are, what your site is about, and where to find your most important pages and resources.

Think of it as a welcome guide written specifically for AI. When a model like ChatGPT, Claude, or Gemini encounters your website, it can read your llms.txt file to quickly understand your organization, your products, and the structure of your content — without having to crawl and interpret every single page.

This matters because AI models face a fundamental challenge: the web is enormous, and most of it is wrapped in complex HTML, JavaScript, navigation menus, cookie banners, and advertisements. Even with sophisticated crawling, AI models can struggle to determine what a website is actually about. llms.txt cuts through that noise by providing a clean, structured Markdown document that says: "Here is who we are and here is what matters."

If you are new to the concept of optimizing your website for AI systems, start with our overview of what AI SEO is and why it matters. The llms.txt file is one piece of a broader technical setup that also includes robots.txt configuration for AI crawlers, semantic HTML structure, and structured data markup.

Who Created It and Who Uses It

The llms.txt specification was proposed by Jeremy Howard (co-founder of fast.ai and a well-known figure in the AI research community) in late 2024. It emerged from a practical observation: AI models need a lightweight, standardized way to understand websites, and nothing in the existing web infrastructure (robots.txt, sitemap.xml, meta tags) was designed for this purpose.

The specification is a community-driven standard, meaning it is not governed by a formal standards body like the W3C or IETF. Instead, it has been adopted organically by organizations that see its value. By early 2026, adoption includes:

• Anthropic (makers of Claude) — publishes llms.txt on their documentation sites
• Vercel — adopted llms.txt across their developer documentation
• Cursor — the AI code editor uses llms.txt for its own site and encourages its use
• Cloudflare — both publishes and supports llms.txt
• Mintlify, ReadMe, and other documentation platforms — offer built-in llms.txt generation
• Hundreds of SaaS companies, developer tools, and content sites — adoption has accelerated significantly through 2025 and into 2026

The growing adoption signals something important: the AI ecosystem is converging on llms.txt as a practical solution for website-to-AI communication, even without formal standardization.

llms.txt vs llms-full.txt

The specification defines two files, each serving a different purpose:

llms.txt (the summary)

This is the primary file. It lives at yourdomain.com/llms.txt and provides a concise overview of your site. It should be short enough that an AI model can read it in a single pass — typically a few hundred lines at most.

The purpose of llms.txt is to answer three questions quickly:

1. What is this website/organization?
2. What are the most important pages?
3. Where can the AI find more detailed information?

llms-full.txt (the extended version)

This is an optional companion file at yourdomain.com/llms-full.txt. It contains much more detail — potentially the full text content of your key pages, complete documentation, detailed product specifications, or comprehensive guides.

While llms.txt is a table of contents with brief descriptions, llms-full.txt is more like the entire book. It is particularly useful for:

• Developer documentation and API references
• Product catalogs with detailed specifications
• Knowledge bases with extensive article content
• Technical reference material

When to use which

| Scenario | Recommendation | |---|---| | Small business website (5-20 pages) | llms.txt only — your summary covers everything | | SaaS product with documentation | Both — llms.txt for overview, llms-full.txt for docs | | E-commerce with large catalog | Both — llms.txt for categories/top products, llms-full.txt for full catalog | | Blog or content site | llms.txt only — link to your best articles with descriptions | | Developer tools / APIs | Both — strongly recommended, as AI coding assistants actively use these files |

The Complete Format and Specification

The llms.txt format is intentionally simple. It is a standard Markdown file with a specific structure. Here is the complete format:

# Site Name

> A brief, one-paragraph description of the site or organization.
> This blockquote tells AI what your site is about at the highest level.

## Docs

- [Page Title](https://yourdomain.com/page-url): Brief description of what this page covers
- [Another Page](https://yourdomain.com/another-page): Description of this page

## Optional

- [Less Critical Page](https://yourdomain.com/optional-page): This content is helpful but not essential

Let's break down each element:

The H1 heading (required)

The file starts with a single # heading containing your site or organization name. There should be exactly one H1 in the file.

The blockquote (required)

Immediately after the H1, a blockquote (>) provides a concise description of your organization or site. Keep this to 1-3 sentences. This is the single most important piece of text in the file — it is the first thing AI reads about you.

The "Docs" section (recommended)

An ## heading labeled "Docs" (or similar) followed by a list of your most important pages. Each item is a Markdown link with an optional description after a colon. These are the pages you most want AI to know about and potentially reference.

The "Optional" section (optional)

An ## heading labeled "Optional" followed by a list of pages that provide additional context but are not critical. AI models may or may not read these depending on their context window and the query at hand.

Additional sections

You can add other ## sections as needed — for example, "Blog," "API Reference," "Products," or "About." The specification is flexible about section names.

Formatting rules

• Use standard Markdown syntax
• Each link should be on its own line, preceded by a - list marker
• Descriptions after links are separated by a colon and space (: )
• Keep the file as concise as possible — AI models have limited context windows
• Use absolute URLs, not relative paths
• The file should be served with a text/markdown or text/plain Content-Type header

What to Include in Your llms.txt

Choosing the right content for your llms.txt requires thinking about what AI models need to know to accurately represent your business. Here is what to prioritize:

Must include

• Company/organization description — Who you are, what you do, who your audience is. Be specific. "SaaS analytics platform for mid-market e-commerce" is far better than "innovative solutions provider."
• Core product or service pages — The pages that define what you offer
• Pricing page — AI frequently gets asked about pricing; give it the authoritative source
• About page — Establishes entity identity and authority
• Contact or support page — Helps AI direct users to the right place

Should include

• Top-performing content — Your best blog posts, guides, or resources
• Documentation — Especially for software products or technical services
• FAQ page — Already structured in a Q&A format that AI models love
• Case studies or testimonials — Evidence of authority and results

Consider including

• Industry-specific landing pages — If you serve multiple verticals
• Comparison pages — "Product A vs Product B" content that AI models frequently cite
• Team or author pages — Strengthens E-E-A-T signals that AI uses to assess authority

What NOT to include

• Login pages, account dashboards, or internal tools
• Pages with thin content that you would not want AI citing
• Duplicate pages or near-identical variations
• Pages you have blocked in robots.txt — there is no point listing pages that crawlers cannot access

Step-by-Step Creation Guide

Follow these steps to create and deploy your llms.txt file:

Step 1: Identify your key pages

Open your website analytics (GA4, Plausible, or similar) and list the pages that:

• Receive the most traffic
• Drive the most conversions
• Best represent your products or services
• Contain your highest-quality content

Aim for 10-30 links in your llms.txt. More than 50 starts to defeat the purpose of being concise.

Step 2: Write your description

Draft the blockquote that opens your file. This should be 1-3 sentences answering: "If an AI could only know one thing about my website, what would it be?" Be factual, specific, and avoid marketing fluff. AI models prioritize clarity over persuasion.

Step 3: Organize into sections

Group your pages logically. Common section structures:

• For a SaaS company: Docs, Product, Blog, API Reference, Optional
• For an agency: Services, Case Studies, About, Blog, Optional
• For e-commerce: Products, Categories, Guides, Support, Optional
• For a content site: Featured Articles, Categories, About, Optional

Step 4: Write link descriptions

For each link, write a brief description (one sentence) explaining what the page covers. These descriptions help AI decide which pages are relevant to a given query.

Step 5: Create the file

Create a plain text file named llms.txt using any text editor. Write valid Markdown following the format described above. Here is a minimal working example:

# Acme Analytics

> Acme Analytics is a web analytics platform for e-commerce businesses.
> We help online stores understand customer behavior, optimize conversion
> funnels, and increase revenue through data-driven decisions.

## Docs

- [Product Overview](https://acme-analytics.com/product): Complete overview of Acme Analytics features and capabilities
- [Pricing](https://acme-analytics.com/pricing): Plans start at $49/month for up to 100k monthly pageviews
- [Getting Started Guide](https://acme-analytics.com/docs/getting-started): Step-by-step setup instructions for new users
- [API Documentation](https://acme-analytics.com/docs/api): REST API reference for developers integrating Acme Analytics
- [E-commerce Tracking](https://acme-analytics.com/features/ecommerce): How we track purchases, cart abandonment, and product performance

## Blog

- [2026 E-commerce Benchmarks](https://acme-analytics.com/blog/ecommerce-benchmarks-2026): Conversion rates, AOV, and traffic benchmarks across 12 industries
- [GA4 Migration Guide](https://acme-analytics.com/blog/ga4-migration): How to switch from Google Analytics to Acme Analytics

## Optional

- [About Us](https://acme-analytics.com/about): Founded in 2022, backed by Y Combinator, serving 3,000+ stores
- [Changelog](https://acme-analytics.com/changelog): Recent product updates and new features
- [Contact Support](https://acme-analytics.com/support): Email and live chat support available Mon-Fri

Step 6: Deploy the file

Upload llms.txt to your website root so it is accessible at yourdomain.com/llms.txt. The deployment method depends on your platform:

• Static hosting (Netlify, Vercel, GitHub Pages): Add the file to your project root or public directory
• WordPress: Upload to the root directory via FTP/SFTP, or use a plugin that supports custom root files
• Webflow, Squarespace, Wix: Check if the platform allows custom files at the root; some require workarounds via hosting redirects
• Custom server (Nginx, Apache): Place the file in your document root

Step 7: Configure the Content-Type header

Ensure your server returns the file with a text/markdown or text/plain Content-Type header. Most servers handle .txt files correctly by default, but verify this by checking the response headers.

Step 8: Optionally create llms-full.txt

If you have extensive documentation or content, create an llms-full.txt following the same format but with much more detail. Reference it from your llms.txt so AI models know it exists.

Real-World Examples

Here are simplified versions of how real organizations structure their llms.txt files to give you a sense of different approaches:

Developer tools example (simplified from Anthropic's approach)

# Anthropic

> Anthropic is an AI safety company that builds reliable, interpretable AI
> systems. Our primary product is Claude, an AI assistant.

## Docs

- [Claude API Documentation](https://docs.anthropic.com): Complete API reference for building with Claude
- [Model Overview](https://docs.anthropic.com/models): Available Claude models, capabilities, and pricing
- [Prompt Engineering Guide](https://docs.anthropic.com/prompting): Best practices for getting great results from Claude

## Optional

- [Research Papers](https://anthropic.com/research): Published AI safety and alignment research
- [Company Blog](https://anthropic.com/blog): Product announcements and technical insights

SaaS platform example (simplified from Vercel's approach)

# Vercel

> Vercel is a cloud platform for frontend developers, enabling instant
> deployments and serverless functions. We are the creators of Next.js.

## Docs

- [Documentation](https://vercel.com/docs): Complete platform documentation
- [Next.js Docs](https://nextjs.org/docs): Framework documentation for Next.js
- [Pricing](https://vercel.com/pricing): Hobby (free), Pro ($20/mo), and Enterprise plans
- [Templates](https://vercel.com/templates): Starter templates for common use cases

## Blog

- [Engineering Blog](https://vercel.com/blog): Technical posts about web development and infrastructure

Notice the patterns: every example leads with a clear, factual description. The Docs section contains the pages the organization most wants AI to surface. Descriptions are informative, not promotional.

How AI Models Use llms.txt

Understanding how AI models interact with llms.txt helps you optimize the file for maximum impact.

When an AI model encounters your website (either through a user query or during a web search), it may check for the existence of a /llms.txt file — similar to how traditional search engines check for /robots.txt and /sitemap.xml. Here is what happens:

1. Discovery — The AI model or its crawling infrastructure requests yourdomain.com/llms.txt
2. Parsing — The model reads the Markdown content, extracting your site description, page list, and link descriptions
3. Context building — The information from llms.txt helps the model understand your site's purpose and content structure before diving into individual pages
4. Source selection — When a user asks a question relevant to your domain, the model may use llms.txt to identify which specific pages on your site are most relevant to the query
5. Citation — If the model uses content from your site in a response, the contextual understanding from llms.txt improves the accuracy of how your site is described and cited

The key benefit is reducing ambiguity. Without llms.txt, an AI model has to infer what your site is about by crawling and interpreting complex HTML pages. With llms.txt, you are providing an authoritative, structured summary that eliminates guesswork.

This is especially valuable when AI models use Retrieval-Augmented Generation (RAG), where the model retrieves and synthesizes information from multiple sources. Your llms.txt helps ensure that when your site is retrieved, the AI understands the context correctly.

It is worth noting that llms.txt works alongside your other technical setup. The file complements the list of AI crawler bots you allow in robots.txt, the page structure you build with semantic HTML, and the structured data you provide through JSON-LD schema markup. Together, these create a complete picture for AI.

Adoption Status in 2026

As of early 2026, llms.txt adoption is in a rapid growth phase. Here is where things stand:

Current adoption

• Thousands of websites have published llms.txt files, up from a handful in late 2024
• Major tech companies (Anthropic, Vercel, Cursor, Cloudflare) have adopted the standard
• Documentation platforms (Mintlify, ReadMe, GitBook) have built automatic llms.txt generation into their products
• AI coding assistants (Cursor, Windsurf, GitHub Copilot) actively check for llms.txt when indexing codebases and documentation sites
• CMS plugins for WordPress and other platforms now support llms.txt generation

What has changed since 2024

• The specification has matured with clearer formatting guidelines
• Tool support has expanded — you can now generate, validate, and test llms.txt files with dedicated tools
• AI models have become better at finding and using the file
• The distinction between llms.txt and llms-full.txt has become well-established

Where adoption is heading

The trajectory is clear: llms.txt is following the same path as robots.txt did in the 1990s and sitemap.xml did in the 2000s. It started as an informal convention, gained adoption among early movers, and is now approaching the point where not having one puts you at a disadvantage.

Given that AI referral traffic converts 4.4x better than organic search and is growing 326% year-over-year, the ROI of spending 30 minutes to create an llms.txt file is difficult to argue against.

How to Verify It Works

After deploying your llms.txt file, verify it with these checks:

1. Accessibility check

Open yourdomain.com/llms.txt in a browser. You should see your Markdown content displayed as plain text. If you get a 404 error, the file is not in the right location or your server is not configured to serve it.

2. HTTP header check

Use your browser's developer tools (Network tab) or a command-line tool to verify the response headers:

curl -I https://yourdomain.com/llms.txt

Look for:

• Status: 200 OK
• Content-Type: text/markdown or text/plain (both work)
• No redirect chains — the file should be served directly

3. Markdown validation

Paste your llms.txt content into a Markdown renderer (such as a GitHub Gist or any Markdown preview tool) and confirm that:

• The H1 heading renders correctly
• The blockquote is properly formatted
• All links are clickable and point to the correct URLs
• Section headings use ## (H2) consistently

4. Link verification

Click every link in your llms.txt to confirm they all resolve to live, accessible pages. Broken links in your llms.txt are worse than no llms.txt at all — they signal to AI that your site information is unreliable.

5. AI testing

The most meaningful test: ask AI models about your site and observe whether they reference the information from your llms.txt accurately. Try queries like:

• "What does [your company] do?"
• "What products does [your company] offer?"
• "Tell me about [your company's] pricing"

Compare the AI responses to what you wrote in your llms.txt. If the responses align, the file is working. If not, give it time — AI models do not update their knowledge instantly, and crawl frequency varies by platform.

6. Use an automated scanner

Tools like AImetrico can check for the existence and validity of your llms.txt as part of a broader AI readiness audit, alongside your robots.txt configuration and XML sitemap.