What Is llms.txt?
llms.txt is a proposed plain-text file at the root of a domain that gives large language models a curated Markdown index of the pages a site considers most useful.
llms.txt is a proposed plain-text file at the root of a domain that gives large language models a curated Markdown index of the pages a site considers most useful. It is not a crawler directive. robots.txt controls what a bot may fetch; llms.txt only suggests what is worth reading, and no model is obliged to look.
Why llms.txt is worth understanding
Nothing breaks without it. That is the honest starting point, and it separates llms.txt from every other file that lives in your site root. A missing robots.txt has immediate, observable consequences. A broken sitemap slows discovery on large sites. A missing llms.txt produces no symptom anyone has been able to measure.
The value sits in the work underneath it. Writing one forces you to name the twenty pages on your site that genuinely answer a question, describe each in a single line, and admit that the rest are padding. Most teams cannot do that on demand. The exercise improves your information architecture whether or not a model ever requests the file.
The risk is what gets attached to it commercially. Implementation is being quoted as though the absence of the file were costing you visibility in AI answers. Before you pay for that, get a straight read on where you actually stand in those answers, then decide what deserves budget.
What the proposal actually specifies
The format is deliberately small. It is Markdown, it is human-readable, and it can be written by hand in twenty minutes.
- One file, one location. A single Markdown document served at the domain root as
/llms.txt. Subdirectory versions are not part of the proposal. - An H1 with the site or project name. This is the only element the proposal treats as required.
- A short summary block. A blockquote explaining what the site is, who it serves, and any context a reader needs before the links make sense.
- H2 sections containing link lists. Each entry is a Markdown link, optionally followed by a colon and one line describing what the page contains and when it is relevant.
- An optional section that can be skipped. The proposal reserves a section for secondary links a model may drop when working context is tight.
- A companion full-text file. Some sites also publish an expanded document containing the page content itself rather than links to it.
The reasoning behind the format is sound even where the adoption is not. Language models work inside a limited context window. A rendered HTML page carries navigation, cookie banners, scripts, and boilerplate, and all of it competes with the sentence you wanted read. A flat Markdown index strips that away. Cheaper parsing is a real benefit, and it holds regardless of who ends up requesting the file.
What the format cannot do is confer authority. It is a table of contents, not a ranking input, and it does not decide whether a model cites you. Here is how it sits against the two root files that do have established support.
| File | What it is for | Standing |
|---|---|---|
| robots.txt | Telling crawlers which paths they may request | Long-established convention, honoured by the major search crawlers |
| sitemap.xml | Listing URLs you want discovered and considered for indexing | Documented and supported by major search engines |
| llms.txt | Offering models a curated reading list of your best pages | A community proposal with no confirmed major-provider support |
The blunt version
No major model provider has confirmed that it reads llms.txt. It is a proposal, not a standard, and anyone selling it to you as required is guessing. That is not a controversial position; it is what the absence of provider documentation means.
The commercial appeal is easy to explain. The file takes half a morning to produce, it looks technical on an invoice, and its effect cannot be disproved, because you never see the version of events where you did not publish it. Any visibility gained afterwards gets attributed to the file. Any visibility lost gets attributed to something else. Deliverables that cannot fail are good business for the supplier and expensive for you.
So use this rule. Publish one when you can write it yourself in an afternoon and it doubles as an internal audit of which pages are worth defending. Refuse to pay for it as a line item, and refuse outright if the pitch includes a promised uplift, a citation-rate figure, or a named chatbot said to be reading it. Ask for the provider’s own documentation. A vendor blog post is not documentation. The durable work is unglamorous and it is the same work that earns citations in AI answers today: clear pages, direct answers near the top, and claims a model can lift without reformatting them.
Example
Say a payroll software company with 900 URLs decides to publish one. The team lists its twelve product pages, its integration directory, and a pricing page, then stalls on the blog, because 200 posts exist and nobody can defend more than fifteen of them. That stall is the finding. The file gets published with 30 links and one line each, which takes an afternoon and costs nothing beyond that time. Six weeks later there is no measurable change in how often the brand appears in AI answers, and no way to attribute anything if there were. What did improve was the internal argument about which 185 blog posts should be merged, rewritten, or deleted. That decision was worth the afternoon. The file itself was a by-product.
FAQ
Does llms.txt affect Google rankings?
No. Google documents the files it uses for crawling and discovery, and llms.txt is not among them. Publishing one will not help or hurt your organic positions. If a proposal asks for budget on the basis that it influences ranking, that claim has no documented support behind it.
Is it worth publishing one anyway?
If you write it in-house, yes. The cost is an afternoon and the file is harmless. Treat it as a forcing function for deciding which pages actually answer something. If the cost is a five-figure engagement built around it, the answer changes, because you are paying for an unverified assumption.
Is llms.txt the same as robots.txt?
No, and they are not substitutes. robots.txt sets access rules and is honoured by the established crawlers, including the ones you may want to block. llms.txt grants nothing and blocks nothing. It only suggests what to read. Blocking an AI crawler is a robots.txt decision, never an llms.txt one.
Related terms
- Retrieval-Augmented Generation — the retrieval process that decides which passages a model actually sees.
- Above the Fold — the human version of the same problem: what gets seen before attention runs out.
- LLM citation — the outcome llms.txt is sold as improving, measured properly.
Write llms.txt yourself in an afternoon, or do not write it at all. It is a reasonable idea with no confirmed audience, and pricing it as a deliverable says more about the supplier than the file.