We gave llms.txt a weight of zero, and we still recommend it

Updated: 2026-08-27

Quick answer: llms.txt is a proposed convention for telling language models what your site contains and how to read it. No major AI provider has committed to honouring it. In our own scoring engine it carries a weight of exactly zero, and we still tell people to publish one. Those two positions are compatible, and the reason they are compatible is worth more than the file itself.

What is the file?

A markdown file at the root of your domain, /llms.txt, listing what a language model should know about your site: what it is, what the main sections are, and where the canonical version of each important document lives.

It is conceptually close to robots.txt, which tells crawlers what they may fetch, and to sitemap.xml, which tells them what exists. llms.txt tries to tell them what matters and in what order.

The idea is reasonable. Sites are full of navigation, boilerplate and duplication, and a machine reading yours for the first time has to work out which of a thousand URLs carries the substance. A short curated file could save it that work.

llms.txt

A proposed file at the site root listing what a language model should read.

Proposed convention

An agreement that has been published but that nobody has committed to honour.

What is the problem with the idea?

Nobody has agreed to read it.

robots.txt works because crawler operators chose to honour it, consistently, for decades. sitemap.xml works because search engines built ingestion for it. Both are conventions that became functional through adoption, not through publication.

llms.txt has the publication. The adoption is the open question. No major model provider has publicly committed to fetching it as part of their pipeline, and as of August 2026 there is no published measurement showing that having one changes how often a site is cited.

That is a plain statement of the current position, not a prediction. It may change. It has not yet.

Why does our engine score it zero?

We track 75 variables. Four of them are marked with a horizon of future, meaning they describe things that plausibly matter later and demonstrably do not matter now: llms.txt itself, a markdown twin of each page, content negotiation for markdown, and MCP endpoint exposure.

Each of those four carries a weight of zero in the score. They are reported in a separate section of the audit rather than folded into the number.

But the weight they would carry if adoption arrives is preserved in the model, in a field we call weightIfAdopted. The reasoning is recorded, the value is set, and switching them on is a configuration change rather than a redesign.

This is the whole design decision in one sentence: a score should reflect what is measurable today, and a model should remember what it is watching for.

We asked the models themselves whether they read llms.txt. Not one called it required or even taken into account. That is why its weight is zero. We keep the file anyway: it is an extra layer of context where the main point of every page can be stated in one line.

Why do we recommend publishing one anyway?

Three reasons, in descending order of how much I believe them.

The cost is close to zero. A useful llms.txt takes twenty minutes to write and never needs maintenance beyond adding a line when a major section appears. There is no meaningful downside, no performance cost, and no risk of penalty.

Writing it is diagnostic. The exercise forces a question most site owners have never answered: if a machine could read only ten pages of this site, which ten? People discover, doing this, that their most important page is buried, that two pages cover the same ground, or that the thing they actually do is not clearly stated anywhere. That discovery is worth the twenty minutes on its own, independent of whether any model ever fetches the file.

Asymmetric payoff. If adoption never arrives, you lost twenty minutes. If it arrives, you are already positioned and the cost of being early was nothing. That is not a strong argument on its own, but combined with a cost of zero it does not need to be.

What I am not claiming: that it will improve your citations. There is no measurement supporting that and I am not going to invent one.

How do you write one that is actually useful?

Because the diagnostic value is the real value, the file is worth writing properly rather than generating.

# Company Name

> One sentence stating what this organisation does and for whom.

## Core pages

- [What we do](https://example.com/services): the four services offered,
  with scope and price ranges for each.
- [Pricing](https://example.com/pricing): three tiers, what is included in
  each, and the criteria for choosing between them.

## Reference

- [Documentation](https://example.com/docs): setup, API reference,
  troubleshooting.

## Notes

- Canonical language is English; Spanish and Russian versions exist at
  /es/ and /ru/ and are translations, not separate content.

Rules that make the difference between a useful file and a decoration:

Descriptions, not labels. "Pricing" tells a machine nothing it could not guess from the URL. "Three tiers, what is included, and the criteria for choosing" tells it what the page answers.

Ten to twenty entries, not two hundred. The file's only advantage over a sitemap is curation. A complete listing throws that away and becomes a worse sitemap.

State your ambiguities. If you have three language versions, say which is canonical. If two pages look similar, say how they differ. This is where the file does work no other format does.

Keep it honest. A file describing a site that does not exist as described is worse than no file, for the same reason mismatched schema is worse than no schema.

What is the larger point about emerging standards?

The interesting question is not whether to publish this particular file. It is how to treat any proposed standard before adoption resolves.

The two common failures are symmetrical. One is to implement everything proposed, which produces a site cluttered with formats nobody reads and an owner who believes work has been done. The other is to dismiss everything unadopted, which means being consistently late to the ones that stick.

The position that survives both: implement it if the cost is genuinely near zero, score it at zero until there is evidence, and write down what evidence would change your mind.

That last part is the one people skip. We wrote down weightIfAdopted values for all four future variables specifically so that "adoption arrived" becomes an observable event with a prepared response, rather than a thing we notice eighteen months late.

What would change our position?

Stated in advance, so it is falsifiable:

A provider commits publicly. If a major model provider documents that their pipeline fetches /llms.txt, the weight goes from zero to its prepared value that week.

A measurement appears. A controlled study showing citation differences between matched sites with and without the file would move it, even without a provider statement.

Our own data shows it. We accumulate scans and jury results. If sites with the file consistently outperform matched sites without it across enough samples, that is evidence, though our sample is nowhere near large enough yet to say anything.

Absent any of those, it stays at zero, and we keep recommending it on the grounds that it costs nothing and teaches you something about your own site.

What are the other three files in the same category?

llms.txt is the best known of four things our engine watches and scores at zero. The others are worth naming, because they share the same shape and the same reasoning.

A markdown twin of each page. Serving /some-page.md alongside /some-page, containing the same content without navigation, styling or boilerplate. The argument is that a model reading markdown wastes nothing on layout. The counterargument is that extraction from HTML already works and nobody has committed to preferring the markdown version.

Content negotiation for markdown. The same idea without the second URL: an agent sending Accept: text/markdown receives the clean version at the same address. More elegant, more work to implement, equally unadopted.

MCP endpoint exposure. Publishing a Model Context Protocol endpoint so an agent can query your site's data structurally instead of scraping pages. This one is the most interesting technically and the furthest from mattering for an ordinary business site, because it presumes agents that want structured access to you specifically.

All four sit in the same bucket: plausible, cheap to prepare for, and currently unrewarded. All four have a recorded weightIfAdopted value. None of them affects a score today.

We list them in a separate section of every report, labelled as future bets, precisely so that nobody reads them as work that has been left undone.

Frequently asked questions

Is llms.txt an official standard? No. It is a proposed convention with a specification and community adoption. It has no endorsement from major model providers.

Does it replace robots.txt? No. robots.txt controls access and is honoured. llms.txt describes content and is aspirational. Keep both, and make sure robots.txt is not blocking the crawlers you want.

Will publishing it hurt me if it is not adopted? No. It is an unreferenced file at a URL nobody fetches. The failure mode is that nothing happens.

Should I list every page? No. Ten to twenty entries with real descriptions. A complete listing is a sitemap and you already have one of those.

Does it help with Google? There is no evidence that it does, and no reason from the mechanism to expect it to. Classical search has its own ingestion path.

Why score it zero rather than a small positive number? Because a small positive number is a claim that it helps a little, and we have no basis for that claim. Zero with a documented trigger for changing it is the honest position.

The short version

llms.txt is a good idea that nobody has agreed to honour. We give it a weight of zero because a score should describe what is measurable, and we recommend publishing it because it costs twenty minutes and the act of writing it tells you which of your pages actually matter.

If you take one thing from this: the useful discipline is not deciding whether a proposed standard will win. It is separating what you score from what you do, and writing down in advance what would change your mind.

Sources: llms.txt specification v1.1.x; GeoScan v1.2 variable definitions and horizon: future classification; research digest compiled from 165 sources on AI search, August 2026. Details in sources/research-notes.md.