Skip to content

feat(bin): add a hardened quality posture and the D2 receipt checker - #17

Open
BohnBawerick wants to merge 13 commits into
mainfrom
fm/qx-design-revisions
Open

feat(bin): add a hardened quality posture and the D2 receipt checker#17
BohnBawerick wants to merge 13 commits into
mainfrom
fm/qx-design-revisions

Conversation

@BohnBawerick

Copy link
Copy Markdown
Owner

Intent

Fix the D2 quality-gate receipt schema and the design's bounds before the quality loop controller is built on top of them. Evidence is data/qx-quality-pilot/report.md sections 8, 9, and 10 recommendations 5 and 6: the D2 receipt schema cannot express real findings, and the design's own worst failure mode is invisible through it. D4 consumes D2, so this must land first.

Implement, in the report's priority order:

  1. Add head_sha. Without it a drifted base_sha reports not-applicable and exits 0, stepping aside and letting a bad change through on the risk the design called easiest to get wrong and hardest to notice.
  2. Give the clean phase its own findings list, or generalise survivors[] into findings[] with a per-phase classification vocabulary. A complexity offender must not be filed as killable.
  3. Add a stable per-finding id, so no-progress means the same findings rather than the same count.
  4. Add engine name and version, plus the threshold the outcome was judged against. The same code scored 2 to 37 points apart on two runners; a receipt with no engine identity is not comparable.
  5. Make duration_ms required.
  6. Decide whether verify emits one object or two, and say why. One object cannot carry two phases without losing the mapping.
  7. Add a wall-clock bound to bounds. The design carries budget_usd only, but the pilot showed the cost that actually decides affordability is measurement wall clock, spent before any harness call exists to enforce spend against.

Out of scope:

  • Do not touch the two threshold numbers. They are captain decisions (qx-quality-pilot-decision-kill-rate-threshold-policy and qx-quality-pilot-decision-crap-max-threshold-policy).
  • Do not turn any quality bar on for the firstmate repository itself. This work builds the capability to run a quality bar; firstmate is not a project it gets applied to. Building the schema is in scope; enabling it here is not.
  • Do not build the loop controller (fm-quality-loop / bin/fm-quality.sh). That is blocked on this task finishing.

Acceptance:

  • Every item above is implemented or explicitly declined in the design document with a stated reason. Nothing silently dropped.
  • The schema change is a schema change, not a rewrite: existing receipts that are still valid stay valid, or state plainly what breaks and why breaking it is right.
  • The design document and any script header, skill, or doc that describes the old shape are updated to match.
  • Tests cover the new required fields and the failure the missing head_sha allowed, live beside the code they test, and are falsifiable (must fail if the thing they pin is replaced by a constant).
  • Keep it simple. Stay on the design and schema surface.

What Changed

  • bin/fm-project-mode.sh parses a position-independent +hardened registry token and answers --quality with one word, and bin/fm-brief.sh, bin/fm-spawn.sh, and bin/fm-promote.sh carry that posture into the brief line, refuse a spawn that disagrees with it, and print a deviation notice; +hardened beside no-mistakes-prod-only drops to standard with a warning.
  • New docs/quality-gate.md and docs/quality-receipt.schema.json revise the D2 receipt in place: head_sha is required, survivors[] becomes findings[] with per-phase classification and a stable id, detail replaces mutant, engine and threshold are required on clean and harden, duration_ms is a required top-level integer, verify emits one envelope with phases[], and bounds gains budget_minutes.
  • New bin/fm-quality-receipt.sh validates a receipt from a file, -, or stdin against the committed schema (overridable via FM_QUALITY_RECEIPT_SCHEMA), supports --check-head <git-dir> to pin head_sha to that tree's HEAD, and separates receipt faults (exit 1) from wiring faults (exit 2); tests/fm-quality-receipt.test.sh plus additions to the brief, task-delivery, and control-relaunch suites cover the new fields and the missing-head_sha fail-open.

Risk Assessment

✅ Low: Every intent item is implemented with behavior tests against the real validator and the committed schema, the out-of-scope constraints are respected, and the one remaining defect is a wrong exit code on an unusual schema edit rather than a wrong verdict on any receipt.

Testing

I ran the new targeted suite tests/fm-quality-receipt.test.sh (27 checks green), plus tests/fm-documentation-audiences.test.sh for the touched doc inventory and the test-runner routing/coverage inspections for the new file. Because passing unit tests alone would not show the intent, I drove bin/fm-quality-receipt.sh as an end user would, against a real throwaway git repository, and captured a CLI transcript covering all seven intent items: the Stage 0a fail-open receipt with no head_sha now exits 1, a constant and a stale head_sha are both caught by --check-head, a complexity offender can no longer be filed as killable, two survivors on one line stay distinct by id while a duplicate id is refused, missing engine/threshold/duration_ms each fail, verify validates as one envelope carrying per-phase children while the pilot's flat mixed object and a cross-tree glued wrapper are refused, and budget_minutes is present in the documented D1 bounds. I then mutation-tested the new tests six ways; every mutation turned the suite red and the worktree was restored clean. No product failures, flaky tests, or setup problems. Non-D2 commits in the base..target range belong to a prior intent and were not re-validated here.

Evidence: D2 receipt end-to-end CLI walk-through

Source: D2 receipt end-to-end CLI walk-through

--- 1. head_sha: the Stage 0a fail-open ------------------------------ The old receipt: base_sha drifted, no head_sha, "not-applicable", exit 0. {"schema_version":1,"phase":"clean","outcome":"not-applicable","base_sha":"5df8ff79..."} $ bin/fm-quality-receipt.sh validate old-stage0a.json fm-quality-receipt: $/head_sha: required -> exit 1 (was exit 0 before: a bad change walked through) A real sha that is no longer HEAD (the anchor drifted): $ bin/fm-quality-receipt.sh validate --check-head <repo> head-stale.json fm-quality-receipt: $/head_sha: 5df8ff79... is not HEAD eba51b08... -> exit 1 --- 2. per-phase classification vocabulary --------------------------- $ bin/fm-quality-receipt.sh validate complexity-as-killable.json fm-quality-receipt: $/findings/0/classification: expected 'over-threshold' -> exit 1 --- 3. stable per-finding id ---------------------------------------- no-progress key = ['42', '43'] $ bin/fm-quality-receipt.sh validate dup-id.json fm-quality-receipt: $/findings/1/id: duplicate id '42' (also $/findings/0/id) -> exit 1 --- 4/5. engine, threshold, duration_ms are required ----------------- fm-quality-receipt: $/engine: required -> exit 1 fm-quality-receipt: $/threshold: required -> exit 1 fm-quality-receipt: $/duration_ms: required -> exit 1 --- 6. verify emits ONE envelope carrying phases[] ------------------- CI reads one outcome: exhausted phase clean outcome=pass engine=eslint@10.1.0 findings=0 phase harden outcome=exhausted engine=@stryker-mutator/core@10.0.0 findings=1 $ validate verify-flat.json -> fm-quality-receipt: $/phases: required (exit 1) $ validate verify-glued.json -> $/phases/1/head_sha: must equal envelope head_sha (exit 1) --- 7. wall-clock bound in D1 bounds --------------------------------- D1 bounds as documented: {"max_iterations": "4", "no_progress_limit": "2", "budget_usd": "8", "budget_minutes": "20"} wall-clock bound present: True --- exit-code contract ----------------------------------------------- $ validate missing.json -> cannot read receipt ... -> exit 2 (wiring fault, not a verdict)

=== D2 quality receipt: end-to-end walk =====================================
project repo : /tmp/tmp.3gmdbSsnhR/proj
base_sha     : 5df8ff7985512f59c3ed8f4b304af144f9182890
head_sha     : eba51b080d63d78d70fc1e8c16127f45c82ec422
validator    : bin/fm-quality-receipt.sh
=============================================================================

--- 1. head_sha: the Stage 0a fail-open ------------------------------
The old receipt: base_sha drifted, no head_sha, "not-applicable", exit 0.
{"schema_version":1,"phase":"clean","outcome":"not-applicable","base_sha":"5df8ff7985512f59c3ed8f4b304af144f9182890"}

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate old-stage0a.json
fm-quality-receipt: $/head_sha: required
  -> exit 1 (was exit 0 before: a bad change walked through)

A receipt that stuffs a constant in head_sha, checked against the tree:

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate --check-head /tmp/tmp.3gmdbSsnhR/proj head-constant.json
fm-quality-receipt: $/head_sha: does not resolve in /tmp/tmp.3gmdbSsnhR/proj: fatal: Needed a single revision
  -> exit 1

A real sha that is no longer HEAD (the anchor drifted):

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate --check-head /tmp/tmp.3gmdbSsnhR/proj head-stale.json
fm-quality-receipt: $/head_sha: 5df8ff7985512f59c3ed8f4b304af144f9182890 is not HEAD eba51b080d63d78d70fc1e8c16127f45c82ec422
  -> exit 1

The honest receipt from that same tree:

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate --check-head /tmp/tmp.3gmdbSsnhR/proj clean-pass.json
  -> exit 0

--- 2. per-phase classification vocabulary ---------------------------
A complexity offender filed as "killable" (what the pilot was forced to do):

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate complexity-as-killable.json
fm-quality-receipt: $/findings/0/classification: expected 'over-threshold'
  -> exit 1

The same finding filed as "over-threshold":

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate complexity-ok.json
  -> exit 0

--- 3. stable per-finding id ----------------------------------------
Two distinct survivors on one line stay two findings:

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate harden-two-on-a-line.json
  -> exit 0
  no-progress key = ['42', '43']

Two findings sharing one id (count same, identity lost):

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate dup-id.json
fm-quality-receipt: $/findings/1/id: duplicate id '42' (also $/findings/0/id)
  -> exit 1

--- 4/5. engine, threshold, duration_ms are required -----------------

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate  # clean receipt with engine removed
fm-quality-receipt: $/engine: required
  -> exit 1

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate  # clean receipt with threshold removed
fm-quality-receipt: $/threshold: required
  -> exit 1

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate  # clean receipt with duration_ms removed
fm-quality-receipt: $/duration_ms: required
  -> exit 1

--- 6. verify emits ONE envelope carrying phases[] -------------------

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate --check-head /tmp/tmp.3gmdbSsnhR/proj verify.json
  -> exit 0
  CI reads one outcome: exhausted
    phase clean   outcome=pass       engine=eslint@10.1.0 findings=0
    phase harden  outcome=exhausted  engine=@stryker-mutator/core@10.0.0 findings=1

The pilot shape, one flat object mixing both phases:

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate verify-flat.json
fm-quality-receipt: $/phases: required
  -> exit 1

A wrapper gluing a child from another tree:

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate verify-glued.json
fm-quality-receipt: $/phases/1/head_sha: must equal envelope head_sha
  -> exit 1

--- 7. wall-clock bound in D1 bounds ---------------------------------
  D1 bounds as documented: {"max_iterations": "4", "no_progress_limit": "2", "budget_usd": "8", "budget_minutes": "20"}
  wall-clock bound present: True

--- exit-code contract the loop controller branches on ---------------
valid receipt            -> 0
schema/rule violation    -> 1  (the phase command is at fault)

$ /home/paiva/.no-mistakes/worktrees/3437026af8a8/01M0KX7PAKBDDAN72SW7REWCQ2/bin/fm-quality-receipt.sh validate /tmp/tmp.3gmdbSsnhR/missing.json  # unreadable path
fm-quality-receipt: cannot read receipt: [Errno 2] No such file or directory: '/tmp/tmp.3gmdbSsnhR/missing.json'
  -> exit 2  (wiring fault, not a verdict)

=== end =====================================================================
Evidence: Falsifiability sweep of the new tests

Source: Falsifiability sweep of the new tests

[baseline] unmodified tree SUITE PASSED <-- expected [A] head_sha removed from the committed schema's required lists SUITE FAILED <-- good not ok - missing head_sha fail-open: expected exit 1, got 0 [B] --check-head turned into a no-op (a constant head_sha would pass) SUITE FAILED <-- good not ok - head_sha pinned to the parent commit: expected exit 1, got 0 [C] duration_ms, engine and threshold removed from required SUITE FAILED <-- good not ok - duration_ms only inside metrics: expected exit 1, got 0 [D] per-finding id uniqueness rule disabled SUITE FAILED <-- good not ok - duplicate finding ids: expected exit 1, got 0 [E] clean/harden classification vocabularies merged into one enum SUITE FAILED <-- good not ok - clean finding classified killable: expected exit 1, got 0 [F] verify envelope allowed to carry a flat findings[] again SUITE FAILED <-- good not ok - verify with flat findings: expected exit 1, got 0 [restored] tree back to committed state SUITE PASSED <-- expected git status --porcelain: (nothing listed above = worktree clean)

=== Falsifiability of tests/fm-quality-receipt.test.sh ======================
Each mutation removes one thing the intent asked for, then reruns the suite.
A test that pins behavior must go red. Tree is restored after each mutation.

[baseline] unmodified tree
SUITE PASSED  <-- expected

[A] head_sha removed from the committed schema's required lists
SUITE FAILED  <-- good
      not ok - missing head_sha fail-open: expected exit 1, got 0

[B] --check-head turned into a no-op (a constant head_sha would pass)
SUITE FAILED  <-- good
      not ok - head_sha pinned to the parent commit: expected exit 1, got 0

[C] duration_ms, engine and threshold removed from required
SUITE FAILED  <-- good
      not ok - duration_ms only inside metrics: expected exit 1, got 0

[D] per-finding id uniqueness rule disabled
SUITE FAILED  <-- good
      not ok - duplicate finding ids: expected exit 1, got 0

[E] clean/harden classification vocabularies merged into one enum
SUITE FAILED  <-- good
      not ok - clean finding classified killable: expected exit 1, got 0

[F] verify envelope allowed to carry a flat findings[] again
SUITE FAILED  <-- good
      not ok - verify with flat findings: expected exit 1, got 0

[restored] tree back to committed state
SUITE PASSED  <-- expected
git status --porcelain:
(nothing listed above = worktree clean)
============================================================================

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

⚠️ **Rebase** - 1 warning
  • ⚠️ .agents/skills/project-management/SKILL.md - branch carries 7 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (12 file(s)) into the PR:
  • a7ca549 no-mistakes(document): document hardened quality contract in architecture and scripts inventory
  • 1875879 no-mistakes(review): document hardened token in architecture and README grammar
  • e3d0c65 no-mistakes(review): document hardened registration, refuse it on conditional policy
  • 32f566d no-mistakes(review): add promote quality notice, fix brief help, pin base capture
  • aaa2272 no-mistakes(review): fix test suite teardown, header, typo-fallback coverage
  • 9bb7d0c no-mistakes(review): add quality standing-posture notice, correct fallback header
  • 16b36f3 feat: plumb the registered hardened quality posture through to the task record

Push main to origin, or rebase your branch onto origin/main, before gating.

⚠️ **Review** - 1 warning
  • ⚠️ bin/fm-quality-receipt.sh:29 - The usage extractor uses sed -n &#39;2,21{...}&#39;, but line 21 of the file is set -eu, not a comment. Running bin/fm-quality-receipt.sh --help (or any usage-error path, which prints the same text to stderr) ends its output with a bare set -eu line. Confirmed by execution. The range should be 2,20. There is no help test in tests/fm-quality-receipt.test.sh, so nothing catches it.
  • ⚠️ bin/fm-quality-receipt.sh:70 - The -- case does shift; break, leaving the loop without consuming the remaining operands, and nothing after the loop reads &#34;$@&#34;. So validate -- receipt.json never sets FILE, falls through to the FILE=- default, and validates stdin instead of the named file. Confirmed: validate -- /tmp/r.json &lt;/dev/null reported "cannot read receipt" for stdin rather than reading /tmp/r.json. Worse than an error: if stdin happens to carry a different valid receipt, the tool validates the wrong document and exits 0. Fix by capturing FILE=${1:-} after the loop (respecting the existing duplicate-operand refusal).
  • ⚠️ bin/fm-quality-receipt.sh:224 - pattern is checked with re.search, not re.fullmatch. Python's $ also matches immediately before a trailing newline, so the sha pattern ^[0-9a-f]{7,40}$ accepts a trailing newline. Confirmed: a receipt with &#34;head_sha&#34;: &#34;bbbbbbb\n&#34; validates and exits 0, while --check-head on the same receipt would fail with a confusing git error because git rev-parse --verify cannot resolve the embedded newline. Use re.fullmatch (JSON Schema pattern is a search, but every pattern in this schema is fully anchored, so fullmatch is the correct reading here).
  • ⚠️ bin/fm-quality-receipt.sh:179 - validate() implements a hand-picked subset of draft 2020-12 and silently ignores every keyword it does not know: anyOf, oneOf, maximum, exclusiveMinimum, maxItems, uniqueItems, maxLength, patternProperties, dependentRequired, propertyNames. The script's own header declares docs/quality-receipt.schema.json the owner of the JSON shape, and the schema file advertises $schema: draft/2020-12, so a later editor tightening the schema with any of those keywords gets a no-op that still exits 0 - a fail-open in the one tool whose purpose is to not fail open. The code already raises on an unsupported type value, so extend the same treatment: raise SchemaError on any keyword outside the implemented set.
  • ⚠️ bin/fm-quality-receipt.sh:302 - The header contract (line 15) says "Exit 0 on a valid receipt, 1 on an invalid one, 2 on usage or tool errors", but git_rev_parse turns any git failure into a SchemaError anchored at $/head_sha, which main() reports as exit 1. Confirmed: validate --check-head /tmp/definitely-not-a-repo-xyz receipt.json prints "fm-quality-receipt: $/head_sha: fatal: cannot change to ..." and exits 1. A caller (the future loop controller) will read that as a bad receipt and blame the phase command for a wiring bug. Fail-closed is the safe direction, but the exit code should be 2 when git itself failed to run, and 1 only when git ran and the sha genuinely is not HEAD.
  • ⚠️ bin/fm-brief.sh:528 - The hardened quality-gate section names $FM_ROOT/bin/fm-quality.sh three times (lines 528, 532) as the authoritative driver, including bin/fm-quality.sh --help, but that script does not exist and the intent explicitly blocks building it in this task. fm-brief.sh &lt;id&gt; &lt;proj&gt; --mode no-mistakes --quality hardened works today and hands a worker instructions pointing at a missing binary. docs/quality-gate.md line 9 acknowledges the controller is not in this revision, so this looks deliberate staging, but nothing refuses --quality hardened while the driver is absent. Either gate --quality hardened on the script existing, or have the brief say the driver is not built yet.
  • ℹ️ docs/quality-receipt.schema.json:28 - The verify envelope's phases[] only constrains minItems: 1 and each child's phase to the enum [clean, harden]. A wrapper emitting two clean children, or a single clean with no harden, validates cleanly. Intent item 6 was about not losing the per-phase mapping; a duplicate phase reintroduces exactly that ambiguity for the D4 folding rule. Worth deciding whether phases[] should require at most one child per phase before D4 is written on top of it.

🔧 Fix: fix receipt validator operand, sha, keyword and exit-code defects
5 issues (1 error, 3 warnings, 1 info) still open:

  • 🚨 bin/fm-quality-receipt.sh:139 - The validator runs its Python program with exec python3 - &#34;$SCHEMA&#34; &#34;$CHECK_HEAD&#34; &#34;$FILE&#34; &lt;&lt;&#39;PY&#39;, so the program itself is delivered on stdin. When source == "-" (which is the default whenever no file operand is given, line 137, and the documented [&lt;file&gt;|-] form at header line 9), sys.stdin.read() returns the empty remainder of the consumed heredoc, json.loads(&#34;&#34;) raises JSONDecodeError, and main() prints "cannot read receipt" and returns 1. Confirmed by execution: ./bin/fm-quality-receipt.sh validate - &lt; receipt.json, ./bin/fm-quality-receipt.sh validate &lt; receipt.json, and cat receipt.json | ./bin/fm-quality-receipt.sh validate all print cannot read receipt: Expecting value: line 1 column 1 (char 0) and exit 1, for a receipt that exits 0 when passed as a file. The header (lines 9 and 13) and docs/quality-gate.md advertise stdin as a supported input, and D1's contract is that the verify command "prints one D2 verify receipt on stdout" - the natural CI wiring is a pipe, which is exactly the path that cannot work. Worse than a plain error: exit 1 is the "invalid receipt" verdict, so the loop controller will blame the phase command for a defect in this script. No test exercises stdin; every test passes a file, and tests/fm-quality-receipt.test.sh:668 even asserts stdin is NOT read, which passes vacuously. Fix by not feeding the program through stdin, e.g. capture the program into a variable or a temp file and use python3 -c &#34;$PROG&#34; / python3 &#34;$TMPPROG&#34;, leaving fd 0 free for the receipt, then add a stdin round-trip test.
  • ⚠️ bin/fm-quality-receipt.sh:70 - --check-head accepts an empty value: the guard at line 66 only requires that a second argument exists, so CHECK_HEAD=&#34;&#34; passes. main() then gates the check on if check_head_dir:, which is falsy for an empty string, so the HEAD verification is skipped entirely and the receipt validates. Confirmed: bin/fm-quality-receipt.sh validate --check-head &#34;&#34; bad.json exits 0 on a receipt whose head_sha is a constant that is not any commit, while the same receipt with --check-head /real/repo correctly exits 1. A caller building the command from an unset variable (--check-head &#34;$repo&#34;) therefore gets a silent pass from the one flag whose entire purpose is to catch a stuffed constant head_sha - the fail-open class this script exists to close. Refuse an empty --check-head value with exit 2, matching the non-empty-value refusals the sibling scripts already use.
  • ⚠️ bin/fm-quality-receipt.sh:437 - The receipt read lumps OSError together with json.JSONDecodeError and returns 1 for both. A malformed JSON document genuinely is the phase command's fault (exit 1 is right), but a path that does not exist, is a directory, or is unreadable is a wiring fault. Confirmed: validate /tmp/definitely-missing-abc.json exits 1 with "[Errno 2] No such file or directory", validate /tmp exits 1 with "[Errno 21] Is a directory", and a chmod 000 file exits 1 with "[Errno 13] Permission denied". docs/quality-gate.md:146 makes this split load-bearing ("a broken receipt is the phase command's fault, while git failing to run ... is a wiring fault and must not be read as a verdict on the change"), and the previous fix round already corrected exactly this class for git under --check-head. The file-open sibling is the same invariant at the same boundary and is still reachable: the loop controller will read a mistyped receipt path as a failing quality phase and blame the change. Split the handlers so OSError returns 2 and JSONDecodeError returns 1.
  • ⚠️ docs/quality-receipt.schema.json:127 - $defs/phase_result makes engine, threshold, and findings required on every clean and harden receipt, and $defs/engine requires a non-empty name and version while $defs/threshold requires minProperties 1. But docs/quality-gate.md:55 states the D1 rule "Everything else is optional, and a missing phase means that phase reports not-applicable". A phase that is not configured in .quality-gate.yaml has no command, no engine, and no threshold, so it cannot produce a schema-valid receipt without the loop controller inventing an engine name, a version, and a threshold number - fabricated data in the exact fields intent item 4 added so receipts would be comparable. Confirmed: a verify envelope carrying a clean child with outcome not-applicable and no engine is rejected with $/phases/0/engine: required. The existing test at tests/fm-quality-receipt.test.sh:168 covers the different case of a configured phase that found no relevant files, so this gap is untested. D4 is blocked on this schema, so it should be settled now: either allow engine/threshold to be absent when outcome is not-applicable (an if/then the checker already supports), or state in docs/quality-gate.md that an unconfigured phase produces no child receipt at all and D1's rule refers only to the folded envelope outcome.
  • ℹ️ docs/quality-receipt.schema.json:28 - The top-level properties.phases block (lines 28-32) is a byte-for-byte duplicate of the one inside the verify then branch (lines 48-52). The top-level copy is inert: a verify receipt is already constrained by the then copy, and a clean or harden receipt is rejected by else -&gt; $ref phase_result, whose additionalProperties:false has no phases property. Both copies apply under allOf so a divergence would not silently weaken validation, but two places to edit in the file that is declared the owner of the JSON shape is an avoidable drift surface. Drop the top-level copy and keep the one in the then branch.

🔧 Fix: fix receipt validator stdin, empty args and read exit codes
1 warning still open:

  • ⚠️ bin/fm-quality-receipt.sh:274 - validate() returns immediately after resolving a $ref (line 274-276), so any keyword written beside that $ref is silently ignored, and check_schema() does not catch it because every one of those keywords is in KNOWN_KEYWORDS. This contradicts the script's own header contract at line 23 ("An unenforced schema keyword is refused, never ignored") and docs/quality-gate.md:147. Confirmed by execution: setting properties.head_sha to {"$ref": "#/$defs/sha", "minLength": 64, "enum": ["nope"]} passes check_schema and validates a 40-char sha receipt with exit 0 - both added constraints are no-ops. That is the same fail-open class the maxItems refusal was added in the prior fix round to close. Two related symptoms at the same boundary: a broken $ref (e.g. "#/$defs/shaa") raises SchemaError from resolve(), which main() catches in the validate block and reports as exit 1 ("bad receipt") rather than exit 2 ("wiring fault"), so the loop controller blames the phase command for a broken schema file; and a broken $ref inside an "if" subschema is swallowed by matches() and silently flips which branch applies. Fix in check_schema(): resolve every $ref up front and refuse a $ref carrying any non-annotation sibling keyword, raising SchemaError so main() maps it to exit 2.

🔧 Fix: refuse unapplied $ref siblings and unresolvable schema refs
2 warnings still open:

  • ⚠️ bin/fm-quality-receipt.sh:324 - const is checked with Python's !=, where True == 1 is true, so a receipt whose schema_version is the boolean true validates as version 1. Confirmed by execution: a clean receipt with &#34;schema_version&#34;: true (everything else valid) exits 0, and json.load reads the field back as Python True. JSON Schema treats true and 1 as different types, so this is a wrong result with no error - exactly the fail-open class this script exists to close (header line 23, docs/quality-gate.md:147), and the same class the prior rounds fixed for re.search and for ignored keywords. enum at line 326 has the identical in-based comparison; no boolean is currently reachable through an enum in the committed schema, but the same fix applies. Compare type as well as value, e.g. reject when isinstance(instance, bool) != isinstance(schema[&#39;const&#39;], bool) before the equality test.
  • ⚠️ bin/fm-quality-receipt.sh:462 - json.loads(raw) uses Python's default lenient constant handling, so the non-standard literals NaN, Infinity and -Infinity are parsed into floats. is_number at line 252 accepts them, and metrics / threshold are both declared {&#34;type&#34;: &#34;number&#34;} in docs/quality-receipt.schema.json (lines 90, 94), so nothing rejects them. Confirmed by execution: a harden receipt with &#34;threshold&#34;: {&#34;kill_rate_min&#34;: Infinity} and &#34;metrics&#34;: {&#34;kill_rate&#34;: NaN} exits 0 as a valid passing receipt. This is reachable, not theoretical: a Python phase command whose engine reports zero mutants computes kill_rate = 0/0 = nan, and json.dumps emits the bare token NaN by default. The receipt then claims outcome: pass while carrying a metric that compares false against every threshold, which defeats intent item 4's whole purpose ("a receipt with no engine identity is not comparable"). Pass parse_constant= to json.loads so those tokens raise (mapping to the exit-1 "not JSON" path), or reject non-finite floats in is_number.

🔧 Fix: reject boolean consts and non-finite receipt numbers
1 warning still open:

  • ⚠️ bin/fm-quality-receipt.sh:231 - check_schema() does expected_type not in KNOWN_TYPES without first checking that the value is a string. JSON Schema allows type to be an array of type names, so a schema edit like &#34;notes&#34;: {&#34;type&#34;: [&#34;string&#34;, &#34;integer&#34;]} makes the membership test raise TypeError: unhashable type: &#39;list&#39;. The exception is not caught, Python prints a traceback and the process exits 1. Confirmed by execution: FM_QUALITY_RECEIPT_SCHEMA=&lt;schema with a type array&gt; ./bin/fm-quality-receipt.sh validate &lt;valid receipt&gt; prints TypeError: unhashable type: &#39;list&#39; and returns rc=1, while the same schema with the scalar &#34;type&#34;: &#34;boolean&#34; correctly prints schema is not enforceable: $/properties/notes: unsupported type &#39;boolean&#39; and returns rc=2. That contradicts the header contract at lines 20-22 ("Tool errors (exit 2) ... a schema using a keyword this checker does not implement") and docs/quality-gate.md:147, and it is the same fail-direction the previous round fixed for broken $refs: the loop controller reads exit 1 as "the phase command produced a bad receipt" and blames the change for a schema wiring fault. A type union is an ordinary schema edit (making a field nullable or accepting int-or-string), not a contrived one. Fix at line 231 by refusing a non-string type as an unsupported construct, e.g. if expected_type is not None and (not isinstance(expected_type, str) or expected_type not in KNOWN_TYPES): raise SchemaError(...). The same boundary has a second, more exotic symptom worth closing in the same edit: a cyclic $ref ($defs/loop: {&#34;$ref&#34;: &#34;#/$defs/loop&#34;}) passes check_schema's seen guard and then makes validate() recurse until RecursionError, also a traceback and exit 1.
✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-quality-receipt.test.sh (27 behavior checks, all pass)
  • bash tests/fm-documentation-audiences.test.sh (docs/documentation-audiences.json gained docs/quality-gate.md)
  • bin/fm-test-run.sh --list --changed --base 672c6d0 confirms the new test is routed for the changed paths
  • bin/fm-test-run.sh --check-coverage (lane coverage guard ok, total=160)
  • Manual CLI walk-through against a throwaway git repo: bin/fm-quality-receipt.sh validate [--check-head &lt;repo&gt;] &lt;receipt.json&gt; over the Stage 0a fail-open receipt, a constant head_sha, a stale (parent-commit) head_sha, an honest clean pass, a clean finding misfiled as killable, the same finding as over-threshold, two survivors on one line, a duplicate finding id, receipts with engine/threshold/duration_ms dropped, a valid verify envelope, the pilot's flat mixed verify object, a verify wrapper gluing a child from another tree, and an unreadable path for the exit-2 wiring-fault contract
  • Parsed the D1 bounds YAML block out of docs/quality-gate.md and asserted budget_minutes is present alongside max_iterations, no_progress_limit and budget_usd
  • Falsifiability sweep: 6 mutations (head_sha dropped from required; check_head made a no-op; duration_ms/engine/threshold dropped from required; finding-id uniqueness disabled; clean/harden classification vocabularies merged; verify envelope allowed a flat findings[]), each rerunning tests/fm-quality-receipt.test.sh and each failing, with the tree restored and git status --porcelain empty after
  • Out-of-scope check: no .quality-gate.yaml and no bin/fm-quality.sh exist; the two threshold numbers in docs/quality-gate.md are unchanged
🔧 **Document** - 1 issue found → auto-fixed ✅
  • ℹ️ docs/quality-gate.md:9 - Judgment call, left as is: bin/fm-brief.sh's hardened section instructs the worker to drive the loop with bin/fm-quality.sh, which does not exist yet. docs/quality-gate.md already states plainly that the loop controller is not in this revision and that the schema had to land first, and +hardened is not enabled for any project, so no extra warning was added. Closing this belongs to the follow-up loop-controller task, not to a doc edit here.

🔧 Fix: document unsupported array-valued type in receipt checker
✅ Re-checked - no issues remain.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…sk record

Reads a project's registered "+hardened" annotation and carries it to the
worker's instructions and the task's durable record, so the quality loop that
bin/fm-quality.sh will drive has a posture and a fixed base commit to work from.
That script is not part of this change; it is referenced by name only.

- bin/fm-project-mode.sh: --quality prints one word, standard or hardened. The
  two-word stdout its three callers parse is untouched, so it gets its own
  output path. The bracket grammar is now position-tolerant: a "+"-prefixed
  token is a flag and never a mode, so "[+hardened local-only]" resolves the
  mode behind it instead of reading the flag as an unknown mode. Unrecognized
  flags are still ignored rather than refused.
- bin/fm-brief.sh: --quality standard|hardened, defaulting to standard and
  refused on scout, dreamer, and secondmate scaffolds. A hardened brief records
  the sibling "Quality contract: quality=hardened" line and one short quality
  gate section; a standard brief records neither and stays byte-identical to
  the pre-quality scaffold.
- bin/fm-spawn.sh: the brief's quality line must agree with --quality, the same
  check the delivery line already gets, in both directions. quality= and
  base_sha= land in the task record; the base commit is captured once at spawn
  and read back on relaunch, never recaptured, because the loop commits each
  round and a later capture would narrow the gate while still reporting success.
- AGENTS.md: one sentence placing quality resolution at intake.

Tests execute the real interfaces. The load-bearing ones prove a project
without "+hardened" and a brief scaffolded without --quality behave exactly as
before: the two-word stdout is pinned across every annotation form, the two
scaffolds are compared byte for byte, and the task record's key set is pinned
so only quality= and base_sha= are additive.
The stage 0a pilot showed the receipt cannot express real findings
and that a missing head_sha makes a drifted base report
not-applicable and exit 0. This revises unpublished schema v1 in
place: require head_sha, duration_ms, engine, threshold, and a
stable finding id; replace survivors[] with per-phase findings[];
and make verify one envelope with phases[]. bounds.budget_minutes
is the missing wall-clock bound.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant