llms.txt and the shards

Overview

llms.txt is a convention: a single markdown file at the root of a site that tells a model what the site contains and where to find it, in a form that fits in a prompt. opsinjs publishes one, plus a full corpus dump, plus four topic shards.

The reason for six files rather than one is arithmetic. This corpus is large enough that pasting all of it into a prompt wastes most of a context window on pages irrelevant to the task, and a model that has read everything about typography and nothing about clinical status will still get the status wrong. Shards let a caller spend context where it changes the answer.

The six files

Every URL in every one of these files is absolute. A relative URL in an index is useless to a client that fetched the file from somewhere else and no longer knows where it came from. That is the normal case for a model.

The four topic shards do not cover the whole tree, so "use the shards" misleads for the half they miss. Between them they carry eight of the sixteen sections: Components, Screens, Health, Accessibility, Content & language, Foundations, Theming & tokens and Reference. Start here, Recipes, Patterns, Handbook, Agents, Registry & distribution, Project and the introduction have no shard, and neither do the package pages, which sit outside the sixteen groups. /llms.txt names the sections with no shard rather than leaving you to work it out, and for a page in one of them the markdown twin is the route: one request, exactly the page you asked for.

What is in the index

/llms.txt is generated from the page tree, so it cannot drift from the site. Each entry carries the page title, its one-sentence description, and its search aliases, because an alias is what a model was actually thinking of. Sections match the sidebar's sixteen groups.

A component page's status is included too, and it is the most load-bearing field in the file. Sixty component ids are built, and a built id has source in the registry item and an API that may change in any release. Every one of the sixty has been audited against WCAG 2.2 AA, but that audit was run by the authors of the components, not by an independent reviewer, and no component has had a clinical review; see ADR 0025 for the record. shipped means the source installs and passed that author-run audit, and it does not mean either an independent or a clinical review has happened. An index that did not carry the field at all would be an invitation to import a specification. Each entry is one line, and the tail in brackets carries the status where there is one, the kind and the aliases:

## Components

- [Card](https://opsinjs.pensievelabs.org/components/card): A bounded group of related content with a border and consistent padding is the ordinary box that most of a screen is made of. (status: shipped · kind: component · also known as: panel, container, box, tile group, content box)
- [Select](https://opsinjs.pensievelabs.org/components/select): Choosing one option from a list. (status: shipped · kind: component · also known as: dropdown, picker)

The size budget

Two limits, both applied by the generator, and both now reported by pnpm run check:llms when it is pointed at a running server.

/llms-full.txt is capped. When the corpus outgrows the cap the generator keeps whole pages, in the order the sidebar presents them, and stops when the next one would not fit. The generator never truncates mid-page. A markdown document ending in the middle of a table is worse than a shorter complete one, because a model will happily finish the table from imagination. The header counts what survived (Pages: N of M.) and a ## Truncated block at the foot of the file counts what was dropped, names the shards to fetch instead, and then lists every dropped page by title with its .md URL and its status. Nothing is hidden: the manifest is budgeted for before any page is, and measured over the whole corpus so the reservation can never be an underestimate. That costs a few per cent of the cap, in exchange for a file that never drops a page without naming it.

Every shard carries its sections whole, and a shard that does not fails the build. That was not true until recently. At a 400 kB cap the three shards carried 65%, 46% and 32% of their own sections, which is not a smaller answer but a quieter wrong one: ask for the health doctrine, get half of it, with no way to know which half. A shard at its budget is a signal that it needs splitting rather than trimming, so it was split. Reference left the foundations shard and became a shard of its own, because Reference is 899 kB of which eight generated table pages are 800 kB, and a combined shard therefore spent its whole budget on tables and truncated before it reached the doctrine it was named for. The four shards are now 599, 630, 576 and 899 kB against a 1.2 MB cap.

Run against a server, check:llms reads the Pages: N of M. line out of every corpus file, and a shard that dropped even one page is a build failure naming the count and the coverage. It fails separately when an implemented component is missing from /llms-components.txt. That second check matters more than it sounds, because truncation drops pages in corpus order, so the components that fall off are the ones at the end rather than a random sample.

/llms-full.txt is the exception and is reported rather than failed. It is capped on purpose: the whole corpus is 3.8 MB, which is more context than most readers of that file have, so the cap is what makes the file usable at all. It still names every page it dropped, and check:llms fails if it ever drops one without printing that manifest.

If you need a specific page rather than a shard, fetch its markdown twin. See Raw markdown API. Fetching one page is almost always cheaper than fetching a shard and hoping the page is in it.

Verify it worked

curl -s https://opsinjs.pensievelabs.org/llms.txt | head -40

Troubleshooting

/llms-full.txt is missing pages I can see on the site. The cap. The header counts how many pages survived and the ## Truncated block at the foot lists every page that was dropped, with the .md URL to fetch it from. Read that block first rather than diffing the file against /llms.txt. Then fetch the twin, or take the shard that covers its section, remembering that eight of the sixteen sections have no shard at all.

A page's aliases are missing from the index. Aliases come from frontmatter. A page with none contributes none. That is a documentation gap, reportable as a docs issue.

The index is stale after I added a page. It is generated at build time. Rebuild, or in development restart. pnpm run generate does not rebuild the index on its own.

I want an index of only the built components. Filter the Components section on status: shipped; there are sixty in total. /r/index.json answers the same question in a field named implemented and is cheaper to parse than a text index.

Next

• Raw markdown API covers fetching one page instead of a shard.
• Offline docs bundle is the same corpus as one versioned JSON file.
• Writing docs for agents says what makes a page survive the trip into an index entry.