feat(typesafe-docs-drift): phase-1 docs-drift detector with fixture replay

[glifocat]

What changes?

Adds the typesafe-docs-drift skill, phase 1 of a docs-drift detector for NanoClaw. It ranks where the nanoclaw-docs portal disagrees with the code and writes to neither repository.

• Facts from code, not the model. Deterministic parsers read a NanoClaw checkout: ncl resources, verbs, access levels, declared or help-text flags and column enums (src/cli/resources/); environment variables with precedence and defaults (src/config.ts, gateway and driver defaults, .env.example); container_configs columns, defaults and cli_scope values; workspace skill names and descriptions; gateway selection rules (docs/gateway-seam.md when present, otherwise src/gateway-providers/index.ts); the ## Timestamps rules in CLAUDE.md. Each fact is {id, area, statement, evidence: file:line excerpt}.
• Candidate sections. A small BM25 index over the pages listed in docs.json, split on headings, returns the top 3 sections per fact.
• One TypeSafe fan-out request per fact x section: contradicts (noul), covers (noul), staleness (score over current / slightly outdated / wrong / dangerous if followed). documented_anywhere is the max covers per fact.
• Gate and report. Default 0.7 on the nouls, 0.5 on the score confidence, all flags. Ranked DRIFT / MISSING / UNSURE / OK rows with the doc page and heading, the model's numbers and the code evidence; per-area counts, tokens, wall time. --json, --fixture / --record replay, --facts-only and --plan for the key-free half. Raw answers go to a gitignored output directory.
• TYPESAFE_API_KEY is read from the environment at request time only; it is never read from a file, printed or saved, and error bodies that echo it are redacted.
• Docs: docs/typesafe-docs-drift.md (pipeline, gate, JSON contract for phase 2), SKILL.md, catalog, README and changelog entries; plugin version 0.12.0.

Independent of feat/typesafe-triage-dry-run; both bump the plugin to 0.12.0 and touch the same changelog, catalog and .gitignore lines, so whichever merges second needs a trivial conflict resolution.

Benchmark (second commit). drift-benchmark.py runs TypeSafe (recorded and live) and a prompt-and-parse LLM through headless claude -p (tools disabled, 10 pairs per call, strict JSON) on the same 60 recorded fact x section pairs. Only the recorded contender runs by default; network and CLI use is opt-in through --contenders.

Validation

Offline

• python3 -m unittest tests.test_typesafe_docs_drift: 47 tests pass (every extractor against a synthetic checkout, section splitting, lexical ranking, request shape, gate and fold logic, mocked HTTP transport with retry and redaction, concurrent and sequential pair execution with partial failures, fixture replay end to end, a mocked live run that records a replayable fixture, key never in stdout, stderr or saved JSON). No test contacts the network.
• Full suite python3 -m unittest discover -s tests: 286 tests, 10 failures and 25 errors, identical to main before this change (239 tests, same 35). They come from the missing pyte dependency on bare system Python 3.14 in the e2e wizard, Windows, macOS and Proxmox modules; none are in the new module.
• bash -n on all skill scripts, git diff --check, plugin manifest JSON parse: clean.
• Fixture replay command from the docs run by hand: exit 0, DRIFT 1 / MISSING 2 / UNSURE 1 / OK 2.

Live inference (one run)

2026-09-16 22:29 UTC, nanocoai/nanoclaw 6e5008fe against nanoclaw-docs e3e1546, jev-latest, default gates, concurrency 4: 142 facts, 426 requests, 609,315 input and 22,578 output tokens, 71.5 s. DRIFT 7, MISSING 1, UNSURE 11, OK 123. The top five DRIFT rows were checked by hand against the doc text and are real (for example the docs list an awaiting_reason approval status and an unknown_sender_decline_notify drop reason that the CLI enums do not have, and say WEBHOOK_PORT is process-environment only while src/config.ts also reads .env). The sanitized summary is on the doc page; the raw payload stays in the gitignored output directory.

Review

Codex adversarial review on the diff: 8 findings, all fixed before commit (redact before truncating error bodies, prefer declared args over flags mentioned in help text, multiline enums, non-transport exceptions in workers, replayable partial records, failed requests are UNSURE not MISSING, fixture mode honours --limit-facts / --plan / --facts-only, section line numbers match the source file).

Limits

Detection only. Candidate recall is lexical top-3. The shipped fixture's answers are hand-written in the documented response shapes. docs/gateway-seam.md and a non-empty .env.example are not on NanoClaw main at 6e5008fe; the run notes say so.

Benchmark run (live inference)

2026-09-17 10:12 to 10:34 UTC, one macOS machine, 60 pairs, two passes per live contender, claude CLI 2.1.273, jev-latest.

Contender	Wall time	p50 per pair	p50, startup subtracted	Tokens in / out	Cost	Parse failures	Flip rate	Precision	Recall
typesafe-recorded	40.1 s	0.66 s	0.66 s	95,720 / 3,180	not reported	0	n/a	1.00	0.92
typesafe-live	41.3 s	0.68 s	0.68 s	94,880 / 3,180	not reported	0 / 0	3.3% (2/60)	1.00	0.92
claude:haiku	359.6 s	5.57 s	5.09 s	81,994 / 32,776	$0.33	0 / 0	6.7% (4/60)	0.85	0.92
claude:sonnet	284.8 s	3.99 s	3.66 s	110,521 / 25,665	$0.70	0 / 0	11.7% (7/60)	0.83	0.83

Precision and recall are for contradicts on a 30-pair gold file (12 positive). The gold labels were written by Claude, the LLM-assisted reviewer that built this tool, by reading the code and the doc section without looking at contender answers; no human has verified them and 3 are marked borderline. The sample was stratified by TypeSafe's own verdicts, which favors its recall, and a Claude labeler may favor the Claude contenders. One positive pair moves recall by 8 points. Method and caveats: docs/typesafe-docs-drift.md.

• Offline: tests/test_drift_benchmark.py, 24 tests (mocked subprocess and transport). Last clean full suite: 308 tests with the same 35 pre-existing pyte failures. The final full run also showed 19 Proxmox subprocess timeouts under machine load; both Proxmox modules pass when rerun alone (28 tests).
• Codex adversarial review on the benchmark, capped at 3 rounds: 18 findings, 17 fixed; left: duplicate keys inside one JSON object are not rejected.

CI flake fix (fourth commit). The ubuntu job failed in tests/test_shared_terminal.py setUp with a JSON decode error: the server creates its state file with O_EXCL before writing the body, and setUp read it as soon as the path existed. Main's own run for the #25 merge failed the same way. setUp now polls until the file parses, within the same deadline. Full offline suite with pyte installed in a venv: 341 tests, OK, three consecutive green runs of the shared-terminal module.