What is llms.txt?

What is llms.txt and who proposed it?

llms.txt is a plain-text file placed at a site's root that lists its most important pages with one-line descriptions, in Markdown format. It was proposed by Jeremy Howard in September 2024 as a way to help large language models discover and understand a site's key content.

It is a community convention, not a standard adopted by any major platform. Adoption is uneven: some smaller crawlers and agent frameworks look for it, and the major search and AI vendors have not adopted it as a ranking or retrieval input.

What does the file actually contain?

A standard llms.txt has an H1 title, a blockquote summary, and a Markdown list of links. The spec recommends including the site's purpose and an ISO-formatted last-updated timestamp, because agents cache aggressively and a stale file is worse than none.

What it does not do is replace your sitemap, replace structured data, or carry content. It is an index, not a delivery mechanism — a crawler that fetches it still has to fetch the pages it points to.

A working llms.txt example

Here is a complete, correctly formatted file for a small consultancy site. Note the structure: purpose, then a curated list of key pages, each with a one-line description.

# Acme Consulting

> Independent management consultancy specialising in manufacturing operations. We help mid-sized factories reduce unplanned downtime and raise throughput.

Last updated: 2026-09-30

## Services

- [Downtime reduction](https://acme.example/services/downtime): Diagnostic and intervention program for unplanned line stoppages
- [Throughput improvement](https://acme.example/services/throughput): Line balancing, changeover reduction and constraint analysis
- [Maintenance planning](https://acme.example/services/maintenance): Preventive schedules based on failure data rather than calendar intervals

## Key documents

- [Our maintenance methodology](https://acme.example/methodology): How we build preventive schedules from historical failure data

## Contact

- [Contact page](https://acme.example/contact): Email form, office address and telephone
- Email: hello@acme.example

Two details worth copying. The summary blockquote states what the site is for, not what the company is — an agent retrieving this needs the purpose, not the mission statement. And the descriptions say what each page actually contains, so the agent can judge whether fetching it is worth the request.

Do I actually need one?

Probably not, on current evidence. Here is the honest cost-benefit.

The decision therefore turns on whether 20 minutes is worth the small chance it helps with an ecosystem you care about. For most sites the answer is yes, because the downside is negligible and the upside is not zero. What matters is not building a strategy around it.

Which AI crawlers should I allow?

There are several distinct AI crawlers serving different purposes, and treating them as one category is the most common robots.txt mistake. The key distinction is between crawlers that fetch live pages to answer a user's question, and crawlers that gather content for model training.

The recommendation is to allow retrieval crawlers, and to make a deliberate, written decision on training crawlers. The case for allowing training is straightforward: you are not paywalled, and inclusion in future models produces citations. The case against is that your content may be proprietary, or the category valuable enough to compete with. Whichever you choose, record the reason — a decision with a stated rationale survives the next quarterly argument; one without gets relitigated every few months.

What should robots.txt actually do for AI crawlers?

Two rules. First, allow retrieval crawlers, because blocking them removes you from answers generated in response to real user questions. Second, treat robots.txt as a crawling preference, never as access control.

A correctly configured file allows everything, names the AI crawlers explicitly so the policy is documented rather than implied, and points to the sitemap.

User-agent: *
Allow: /

# AI crawlers with retrieval purpose — allow
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /

# Training crawlers — decision documented above
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /

Sitemap: https://example.com/sitemap.xml

If you do decide to block some training crawlers, do it selectively and write down why. Blanket-blocking every AI agent on a blanket instruction is a common reflex with no clear upside, and it forfeits citations in exchange for a feeling of control that robots.txt cannot actually deliver.