Evals

The task suite that checks whether these docs answer the questions an agent asks, at the address it would guess. What each task asserts, and how it is scored.

How this is generated

Source file: skills/opsinjs/evals/evals.json. Script: scripts/run-evals.mts. Command: pnpm run evals.

Each task in the source file is a question an agent asks in practice, the address the answer has to live at, and the things that answer cannot omit or claim. The script resolves each page from content/docs on disk, or from the served .md twins when it is given --base. It checks every assertion and writes a dated result file. No score on this page is typed by a person, and a task with no result renders as <NoDataYet> rather than as an optimistic blank.

What is being scored is the corpus, not a model: the script never contacts one. A failing task means a page is missing, has moved, or has stopped saying the thing it exists to say, which makes this a coverage probe for the docs-as-API contract rather than a benchmark. The suite is deliberately small and deliberately adversarial, and when a task starts failing the usual cause is that a page became ambiguous.

What a task looks like

A task pairs the question with assertions that can be checked mechanically. The assertions are the interesting half:

{
  "id": "normal-is-banned",
  "question": "How should I label a result that is inside the reference range?",
  "why": "Not \"normal\". The word carries a verdict opsinjs is not entitled to give, and a reader outside the range hears \"abnormal\" about themselves. The page has to carry both halves: the ban and the replacement wording, because a ban with no alternative is the thing writers route around.",
  "expect": {
    "page": "health/reference-ranges",
    "mustContain": [
      "normal",
      "is banned",
      "in the usual range",
      "describes the person, not the measurement"
    ],
    "alsoResolves": ["content/writing-status-and-alerts"]
  }
}

mustContain and mustNotContain are lowercase substring checks over the whole page, not regular expressions and not judgements by another model. The substrings are chosen because the page cannot be correct without them rather than because an author happened to phrase it that way. Each one is a rule the page exists to state, or a unit it exists to disambiguate. frontmatter and frontmatterAny pin a declared field, catalogueId and catalogueStatus check the catalogue row instead of the prose, and alsoResolves asserts that the pages a good answer would link to are actually there.

A needle also has to survive typography and serialisation, which is why so many of them are JSX attributes rather than sentences. The runner matches plain substrings against the page text, so a phrase lifted from running prose passes or fails on where the author happened to break the line. The sentence saying a component is implemented wraps on TrendSparkline and does not on ResultCard. An attribute (importPath="@/components/ui/result-card", status="shipped"), a token name, a table cell or a fenced command is on one line by construction. Prefer an attribute whose value is already a string, because those survive both forms of the page: the .md twin serialises every JSX expression attribute, so unbuilt={false} on disk arrives as unbuilt="false" over HTTP. A needle written in the brace form then stops matching under --base. As a mustContain it fails against a page that is correct, and as a mustNotContain it passes while measuring nothing. Where the suite has to pin one of those, it asserts both spellings.

mustNotContain is the more valuable half, because a page can be wrong in both directions: claiming more than its status supports, or still announcing an absence the registry has since filled. When one trips, check the page's status against registry/catalogue.ts before editing the page. The assertion may be the stale half.

What the twenty-two tasks cover

Doctrine. never-mix-the-axes, status-needs-a-word, normal-is-banned, units-are-correctness and no-invented-evidence check that the five rules an agent most often breaks are stated in words on the page an agent lands on when it asks the corresponding question. The last of them also checks that the page declares an evidence value rather than leaving the reader to guess.

Component identity. value-against-a-range checks that the component for the most common health-UI task resolves and says when not to use it. The other three check the honesty contract in both directions, because it can now fail either way. body-map-is-built asks for a working BodyMap snippet and asserts the page supplies one, pinned to the catalogue row as well as to the page. shipped-is-real-and-unreviewed asks how far ResultCard can be trusted and asserts the page names the import path that resolves and says the API may move in any release and that nobody has reviewed it: presenting a built component as settled fails it, and so does still calling it unbuilt. toast-is-built checks the same for the id that spent longest unbuilt.

Getting it installed. install-path, which-registry and tokens-not-values check that the mechanical route from documentation to working project is written down where a tool would look for it. That route is components.json, the @opsinjs namespace and the token layer.

Scope. touch-target-floor, machine-surface and scope-and-limits check the boundaries: the target-size floor, the address of the machine surface, and the statement that opsinjs is not a medical device.

The regression guard. Six tasks exist for no reason other than to make one particular corpus-wide failure loud. Before the component layer was built, every page in the corpus said that nothing was implemented; the repair that changed that is exactly the kind of work a later edit can quietly undo, page by page or wholesale. the-shipped-roster-is-not-empty pins the component index to its rendered roster and to a built id in the catalogue. status-pill-is-built, trend-sparkline-is-built, alert-banner-is-built and metric-tile-is-built each pin one built component to its catalogue row, its status frontmatter and an unwrappable marker in its body, so a regression has to defeat all three to go unnoticed. the-install-command-is-real pins the sixty-second entry point to the one command that works. They fail together, which is the point: a revert to "nothing is built" costs the suite six whole tasks rather than a rounding error.

The other direction. no-npm-package-yet is the only task that guards against the corpus overstating rather than understating. Nothing is published to npm, the decision to publish has not been made, and the package page has to keep saying so while the components it names remain installable by another route entirely. An agent that reads shipped as a release reaches for npm install next, so this is the failure adjacent to every task above it.

Scoring

A task's score is the fraction of its assertions that held. Every assertion counts the same: there is no weight field in the suite and no weighting in the script, because a weight is a claim about relative importance that nobody has justified in writing.

PASS is the result when every assertion held.
PART is the result when some did. The score is the fraction, and each failed assertion is named underneath the task, because "78%" is not actionable and "resolves health/reference-ranges: no page at that address" is.
FAIL is the result when none did, which for a task with a page usually means the page has moved or been renamed.

The headline figure is the mean of the task scores against the passMark declared in the suite file, currently 0.9. The mark is high because the suite is small and its failures are correlated: at 0.8 a suite this size absorbed four whole failing tasks without complaining, which is more than the regression guard above is willing to lose quietly.

Scores are advisory everywhere they run today. run-evals reports them, writes them to public/r/evals.json and never blocks a pull request; the nightly job scores the deployed .md twins with --base, uploads the result as an artefact and is marked continue-on-error, so a red task there is a finding to read rather than a broken build. --strict exits non-zero below the pass mark and nothing invokes it yet.

What this suite does not measure

It does not measure clinical correctness, and it cannot. Every assertion is a substring or a declared field. A corpus can score 100% and still carry a page a clinical reviewer would reject, and the safety argument continues to rest on the review checklist rather than on a score.

It does not measure a model. There is no prompt, no completion and no model endpoint anywhere in the script, so no number produced here is a fact about an assistant. <EvalResult> prints a model name beside a score for exactly that reason: a score with no model attached is one nobody may attribute to one.

It also does not measure whether the documentation is right. It measures whether the documentation is legible to a machine. That is a lower and more achievable bar, and it is the one this pillar is responsible for.

Results

Five of the twenty-two, the ones that carry the doctrine. Nothing here has been run: <EvalResult> takes its score as a prop and this page passes none, so each slot renders the honest absence rather than a number.

This table has not been generated. It is produced by scripts/run-evals.mts. Run pnpm run generate and reload.