---
title: "Evals"
description: "The task suite that checks whether these docs answer the questions an agent asks, at the address it would guess. What each task asserts, and how it is scored."
url: "https://opsinjs.pensievelabs.org/agents/evals"
source: "https://opsinjs.pensievelabs.org/agents/evals.md"
section: "Agents & automation"
kind: "reference"
aliases: ["eval suite", "agent scores", "benchmark", "model evaluation"]
---

> Elements written as `<PascalCase … />` below are opsinjs documentation
> components. Their attributes are the content: the values they render are
> generated from `tokens/*.json` and `registry/catalogue.ts` and are
> published separately at https://opsinjs.pensievelabs.org/r/index.json and under the Reference
> section.
> Nothing is missing from this page. The data simply does not live in
> the prose.

<PageTemplate kind="reference" />

## How this is generated [#how-this-is-generated]

Source file: `skills/opsinjs/evals/evals.json`. Script: `scripts/run-evals.mts`.
Command: `pnpm run evals`.

Each task in the source file is a question an agent asks in practice, the
address the answer has to live at, and the things that answer cannot omit or
claim. The script resolves each page from `content/docs` on disk, or from the
served `.md` twins when it is given `--base`. It checks every assertion and
writes a dated result file. No score on this page is typed by a person, and a
task with no result renders as `<NoDataYet>` rather than as an optimistic blank.

What is being scored is the corpus, not a model: the script never contacts one.
A failing task means a page is missing, has moved, or has stopped saying the
thing it exists to say, which makes this a coverage probe for the docs-as-API
contract rather than a benchmark. The suite is deliberately small and
deliberately adversarial, and when a task starts failing the usual cause is that
a page became ambiguous.

### What a task looks like [#what-a-task-looks-like]

A task pairs the question with assertions that can be checked mechanically. The
assertions are the interesting half:

```json
{
  "id": "normal-is-banned",
  "question": "How should I label a result that is inside the reference range?",
  "why": "Not \"normal\". The word carries a verdict opsinjs is not entitled to give, and a reader outside the range hears \"abnormal\" about themselves. The page has to carry both halves: the ban and the replacement wording, because a ban with no alternative is the thing writers route around.",
  "expect": {
    "page": "health/reference-ranges",
    "mustContain": [
      "normal",
      "is banned",
      "in the usual range",
      "describes the person, not the measurement"
    ],
    "alsoResolves": ["content/writing-status-and-alerts"]
  }
}
```

`mustContain` and `mustNotContain` are lowercase substring checks over the whole
page, not regular expressions and not judgements by another model. The
substrings are chosen because the page cannot be correct without them rather
than because an author happened to phrase it that way. Each one is a rule the
page exists to state, or a unit it exists to disambiguate. `frontmatter` and
`frontmatterAny` pin a declared field, `catalogueId` and `catalogueStatus` check
the catalogue row instead of the prose, and `alsoResolves` asserts that the
pages a good answer would link to are actually there.

A needle also has to survive typography and serialisation, which is why so many
of them are JSX attributes rather than sentences. The runner matches plain
substrings against the page text, so a phrase lifted from running prose passes
or fails on where the author happened to break the line. The sentence saying a
component is implemented wraps on `TrendSparkline` and does not on `ResultCard`.
An attribute (`importPath="@/components/ui/result-card"`, `status="shipped"`), a
token name, a table cell or a fenced command is on one line by construction.
Prefer an attribute whose value is already a string, because those survive both
forms of the page: the `.md` twin serialises every JSX expression attribute,
so `unbuilt={false}` on disk arrives as `unbuilt="false"` over HTTP. A needle
written in the brace form then stops matching under `--base`. As a `mustContain`
it fails against a page that is correct, and as a `mustNotContain` it passes
while measuring nothing. Where the suite has to pin one of those, it asserts
both spellings.

`mustNotContain` is the more valuable half, because a page can be wrong in both
directions: claiming more than its status supports, or still announcing an
absence the registry has since filled. When one trips, check the page's status
against `registry/catalogue.ts` before editing the page. The assertion may be
the stale half.

### What the twenty-two tasks cover [#what-the-twenty-two-tasks-cover]

**Doctrine.** `never-mix-the-axes`, `status-needs-a-word`, `normal-is-banned`,
`units-are-correctness` and `no-invented-evidence` check that the five rules an
agent most often breaks are stated in words on the page an agent lands on when
it asks the corresponding question. The last of them also checks that the page
declares an `evidence` value rather than leaving the reader to guess.

**Component identity.** `value-against-a-range` checks that the component for
the most common health-UI task resolves and says when *not* to use it. The other
three check the honesty contract in both directions, because it can now fail
either way. `body-map-is-built` asks for a working `BodyMap` snippet and asserts
the page supplies one, pinned to the catalogue row as well as to the page.
`shipped-is-real-and-unreviewed` asks how far `ResultCard` can be trusted and
asserts the page names the import path that resolves *and* says the API may move
in any release and that nobody has reviewed it: presenting a built component as
settled fails it, and so does still calling it unbuilt. `toast-is-built` checks the same for the id that spent
longest unbuilt.

**Getting it installed.** `install-path`, `which-registry` and
`tokens-not-values` check that the mechanical route from documentation to
working project is written down where a tool would look for it. That route is
`components.json`, the `@opsinjs` namespace and the token layer.

**Scope.** `touch-target-floor`, `machine-surface` and `scope-and-limits` check
the boundaries: the target-size floor, the address of the machine surface, and
the statement that opsinjs is not a medical device.

**The regression guard.** Six tasks exist for no reason other than to make one
particular corpus-wide failure loud. Before the component layer was built, every
page in the corpus said that nothing was implemented; the repair that changed
that is exactly the kind of work a later edit can quietly undo, page by page or
wholesale. `the-shipped-roster-is-not-empty` pins the component index to its
rendered roster and to a built id in the catalogue. `status-pill-is-built`,
`trend-sparkline-is-built`, `alert-banner-is-built` and `metric-tile-is-built`
each pin one built component to its catalogue row, its `status`
frontmatter and an unwrappable marker in its body, so a regression has to defeat
all three to go unnoticed. `the-install-command-is-real` pins the sixty-second
entry point to the one command that works. They fail together, which is the
point: a revert to "nothing is built" costs the suite six whole tasks rather
than a rounding error.

**The other direction.** `no-npm-package-yet` is the only task that guards
against the corpus overstating rather than understating. Nothing is published to
npm, the decision to publish has not been made, and
[the package page](../packages/opsinjs-react.mdx) has to keep saying so while
the components it names remain installable by another route entirely. An agent
that reads `shipped` as a release reaches for `npm install` next, so this is the
failure adjacent to every task above it.

### Scoring [#scoring]

A task's score is the fraction of its assertions that held. Every assertion
counts the same: there is no weight field in the suite and no weighting in the
script, because a weight is a claim about relative importance that nobody has
justified in writing.

* **PASS** is the result when every assertion held.
* **PART** is the result when some did. The score is the fraction, and each
  failed assertion is named underneath the task, because "78%" is not actionable
  and "resolves health/reference-ranges: no page at that address" is.
* **FAIL** is the result when none did, which for a task with a `page` usually
  means the page has moved or been renamed.

The headline figure is the mean of the task scores against the `passMark`
declared in the suite file, currently 0.9. The mark is high because the suite is
small and its failures are correlated: at 0.8 a suite this size absorbed four
whole failing tasks without complaining, which is more than the regression guard
above is willing to lose quietly.

Scores are advisory everywhere they run today. `run-evals` reports them, writes
them to `public/r/evals.json` and never blocks a pull request; the nightly job
scores the deployed `.md` twins with `--base`, uploads the result as an artefact
and is marked `continue-on-error`, so a red task there is a finding to read
rather than a broken build. `--strict` exits non-zero below the pass mark and
nothing invokes it yet.

### What this suite does not measure [#what-this-suite-does-not-measure]

It does not measure clinical correctness, and it cannot. Every assertion is a
substring or a declared field. A corpus can score 100% and still carry a page a
clinical reviewer would reject, and the safety argument continues to rest on
[the review checklist](../health/safety-review-checklist.mdx) rather than on a
score.

It does not measure a model. There is no prompt, no completion and no model
endpoint anywhere in the script, so no number produced here is a fact about an
assistant. `<EvalResult>` prints a model name beside a score for exactly that
reason: a score with no model attached is one nobody may attribute to one.

It also does not measure whether the *documentation* is right. It measures
whether the documentation is legible to a machine. That is a lower and more
achievable bar, and it is the one this pillar is responsible for.

### Results [#results]

Five of the twenty-two, the ones that carry the doctrine. Nothing here has been
run: `<EvalResult>` takes its score as a prop and this page passes none, so each
slot renders the honest absence rather than a number.

<NoDataYet script="scripts/run-evals.mts" />

<EvalResult task="value-against-a-range" />

<EvalResult task="toast-is-built" />

<EvalResult task="never-mix-the-axes" />

<EvalResult task="normal-is-banned" />

<EvalResult task="no-invented-evidence" />

<Todo owner="agents">
  Wire 

  `public/r/evals.json`

   into 

  `<EvalResult>`

   so a run
  reaches this page. Today the script writes the file and nothing reads it, which
  means a green suite and a suite nobody has run look identical here.
</Todo>

### Reproducing a run [#reproducing-a-run]

```bash
pnpm run evals                                    # score the corpus on disk
pnpm run evals -- --base http://127.0.0.1:4000    # score the served .md twins
```

There is no endpoint to configure and no key to set. The script reads
`content/docs` directly, or fetches the `.md` twins when `--base` is given, so a
run reproduces from a clone with nothing else installed. Running it against a
base URL is the stricter of the two: it scores what the site actually serves,
including whatever the twin renderer did to the page on the way out.

Results are written dated to `public/r/evals.json`. A run is attached to a
documentation version, so a regression can be traced to the change that caused
it. See [Docs freshness](../project/docs-freshness.mdx).