What llms.txt is and why sites add one

llms.txt is a plain markdown file served at the root of a website (for example /llms.txt) that gives AI models and agents a short, structured map of the site: what it is, and which pages matter most. It exists because large language models read pages through a context window, not a browser, and most sites are full of navigation, ads, and script noise that eats that budget without adding information. A well-formed llms.txt gives a model the equivalent of a table of contents: an H1 title, a one-line summary, and grouped links to the pages worth reading, in plain markdown a model can parse without rendering HTML or running JavaScript.

The format, proposed at llmstxt.org, is deliberately small. The only required part is the H1 title. A blockquote summary right after it gives the key context a model needs before reading further. Everything else is optional markdown sections, with H2-delimited sections holding lists of links in the form [title](url): notes. Sites commonly group core pages under a first section like "Docs" and put secondary or background material under a section named "Optional", since some agents will skip that section entirely when they are short on context.

Adding an llms.txt does not guarantee any particular model will fetch or use it today; support varies and is still evolving. What it does is give any crawler or agent that does look for it a clean, low-noise entry point to your site, instead of forcing it to guess from a sitemap or scrape raw HTML.