Specshift evaluates how well a documentation site serves an LLM trying to use the platform the docs describe. Not whether the docs are well-written. Whether the docs contain the concepts an agent would need for a correct answer, a working call, a passing build. Scores are content-coverage heuristics: no task is executed.
§ 01
What we measure.
Future suite: oscal (compliance crosswalk for FedRAMP-targeting platforms). Built — wiring it into the live suite list above is pending.
§ 02
Scoring.
Every test produces a score in [0, 1]. NaN means the test couldn't run (timeout, infrastructure failure) and is excluded from the suite roll-up. Suite scores are weighted averages of their tests. The overall score is the weighted average of suite scores.
Agent and retrieval scores measure documentation content coverage against the v1 Specshift corpus. Each expected concept is a synonym group, and a test that matches at least one group scores no lower than 0.5. No task is executed and no vector recall is measured.
Reports are reproducible. Every scorecard pins both the methodology version and the engine version. Re-run the same suite on the same target with the same engine and methodology and you get a deterministic score — or the methodology documents the source of randomness.
§ 03
How to run.
Paste a docs URL into /eval and Specshift runs all four suites against it right now — no install required. The scorecard, with the overall score, the per-suite breakdown, and every test, renders inline in the response.
npx @wentzel/specshift run https://docs.example.comNot yet published to npm — this is the CLI surface once it ships. /eval is the working way to run Specshift today.
§ 04