What Is llms.txt? How to Build a Machine-Readable AI Knowledge Directory

What Is llms.txt?

llms.txt is a community-driven specification that standardizes how websites expose structured, token-efficient Markdown content to LLMs. Instead of forcing AI crawlers to parse complex CSS, JavaScript bundles, and navigation chrome, an llms.txt file serves as an index of your most critical documentation in plain Markdown.

File Structure & Syntax Grammar

An llms.txt file must begin with a single H1 header stating the project or organization name, followed immediately by a blockquote summary of its purpose:

# Acme Corporation

> Acme provides enterprise distributed database infrastructure with sub-millisecond replication.

## Documentation
- [Architecture Overview](https://example.com/docs/architecture.md): Core distributed consensus engine
- [API Reference](https://example.com/docs/api.md): REST and gRPC endpoint declarations
- [Deployment Guide](https://example.com/docs/deploy.md): Kubernetes operators and bare-metal setup

llms.txt vs. llms-full.txt

The specification defines two files:

• /llms.txt: The curated directory index of clean Markdown links with short descriptions.
• /llms-full.txt: The complete, concatenated text of all referenced documentation, enabling single-request context window injection.

Discovery via rel="describedby"

Merely placing the file at the root is not enough. You should link to it semantically using the rel="describedby" relationship, both in HTML and HTTP headers:

<!-- In HTML head -->
<link rel="describedby" href="/llms.txt" type="text/markdown">

# In HTTP Response Headers
Link: </llms.txt>; rel="describedby"; type="text/markdown"

Common Syntax Mistakes

1. Missing Blockquote: Omitting the blockquote summary immediately beneath the H1 title breaks standard v2 parsers.
2. Broken Links: Linking to URLs that return HTTP 404, 500, or require login cookies.
3. Unformatted Lists: Using arbitrary paragraphs instead of standardized markdown bullet lists.