What Is llms.txt? Definition and How It Works

llms.txt is a proposed standard text file placed at the root of a website (e.g., https://example.com/llms.txt) that provides AI language models with a structured description of the site. Written in Markdown, it tells AI what the site is about, lists its most important pages, and offers guidance on how the content should be interpreted. Think of it as a welcome guide for AI -- where robots.txt controls access, llms.txt provides context.

Why It Matters

AI language models face a fundamental challenge when they encounter a new website: they must figure out what the site is about, which pages are most important, and how to interpret the content -- all from raw HTML. This is inefficient and error-prone.

llms.txt solves this by giving AI a concise, structured overview before it processes individual pages. Instead of an AI model guessing that your 500-page e-commerce site sells outdoor gear, your llms.txt explicitly states it, lists your key product categories, and points to your most authoritative content.

This matters for AI SEO because clarity of context improves citation quality. When an AI model understands your site's purpose and authority areas upfront, it is more likely to select your pages as sources for relevant queries. A well-crafted llms.txt can highlight your expertise, point AI toward your best content, and reduce the chance of your site being mischaracterized.

The standard is still evolving, but early adoption signals to AI systems that your site is optimized for machine readability. As more AI platforms begin checking for llms.txt, sites that already have one will have a head start. For detailed implementation instructions, see our llms.txt specification guide.

How It Works

An llms.txt file follows a simple Markdown structure with defined sections.

Basic structure:

# Site Name

> Brief description of what this site is about.

## Key Pages

- [About Us](https://example.com/about): Company background and team
- [Products](https://example.com/products): Full product catalog
- [Blog](https://example.com/blog): Industry insights and guides

## Optional

- [Terms of Service](https://example.com/terms)
- [Privacy Policy](https://example.com/privacy)

The file begins with the site name as a heading, followed by a blockquote summary. Key pages are listed with URLs and brief descriptions. An optional section can include supplementary pages.

Extended version (llms-full.txt): Some implementations support a longer file at /llms-full.txt that provides more detailed descriptions of each section, content guidelines, and specific instructions for how AI should handle the site's content.

The key principle is brevity with purpose. AI models process llms.txt before diving into individual pages, so the file should be concise enough to fit within a model's context window while informative enough to guide content selection.

Unlike robots.txt, llms.txt is not about permissions -- it does not block or allow access to anything. It is purely informational. For access control, you still need a properly configured robots.txt. See our guide on robots.txt for AI crawlers for that side of the equation.

Practical Implications

• Complements, does not replace robots.txt. Use robots.txt to control which AI crawlers can access your site. Use llms.txt to describe your site to those that do access it. Both files serve distinct roles.
• Prioritize your best content. The pages you list in llms.txt are the ones AI models will examine first. Choose pages that demonstrate your expertise and contain your most authoritative, citable content.
• Keep it current. Update llms.txt whenever you add major new sections, products, or content areas. An outdated llms.txt that points to deleted pages or ignores new product lines works against you.
• Use plain, descriptive language. AI models parse the file literally. Write clear descriptions that an AI can use to categorize your site accurately. Avoid marketing language and buzzwords.
• Monitor AI adoption. Check periodically whether AI platforms are reading your llms.txt by reviewing server logs for requests to /llms.txt. Increasing requests indicate growing platform support.