Skip to content

Commit 52bc5ce

Browse files
igorcostaAutohand Evolve
andcommitted
Protect the existing CLI option contract with an additive snapshot
Capture the actual Commander graph and reject changes against the golden committed on main. Keep the production change to exporting the existing program, and fetch main history for CI comparisons. Record the main baseline and live Fantail transcript. Aggregate candidate verification remains outstanding because repeated host sleeps invalidated both long-running checks; this change is for a draft PR only. Co-authored-by: Autohand Evolve <code-noreply@autohand.ai>
1 parent fcd92ec commit 52bc5ce

7 files changed

Lines changed: 2224 additions & 1 deletion

File tree

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,8 @@ jobs:
7474
shard: [1, 2, 3]
7575
steps:
7676
- uses: actions/checkout@v7
77+
with:
78+
fetch-depth: 0
7779

7880
- name: Setup Node.js
7981
uses: actions/setup-node@v7
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Claude Code compatibility guard
2+
3+
F-0158.0 establishes the option contract before the compatibility flags are implemented. The test imports the actual Commander program from `src/index.ts`; importing it does not parse arguments or invoke a command action.
4+
5+
The golden at `tests/startup/__snapshots__/cli-options.json` records 189 built-in option definitions across 51 command paths, including 94 root options, hidden flags and commands with no options. It captures flags, required/optional/variadic arity, negation, and defaults. `hasDefault` distinguishes an omitted default from an explicit null or false default. Runtime extension flags depend on the user's installed extensions and are outside this built-in contract.
6+
7+
The tests compare the program with both the working golden and the golden committed on `origin/main`. Updating both code and the golden therefore cannot silently change an existing option. CI fetches history so `origin/main` is available. The first guard PR bootstraps the golden when main does not have one yet.
8+
9+
To record additive flags or commands locally:
10+
11+
```sh
12+
git fetch origin main
13+
UPDATE_CLI_OPTIONS_SNAPSHOT=1 bun run test -- tests/startup/cliOptionsSnapshot.test.ts
14+
bun run test -- tests/startup/cliOptionsSnapshot.test.ts
15+
```
16+
17+
The updater rejects removed or changed entries and refuses to write in CI. A newly added entry must appear in the golden in the same PR. Changes to existing semantics need an explicitly reviewed change to this guard; regenerating the golden alone will not approve them.
18+
19+
`baseline.txt` records the untouched main checks before the candidate's full validation. Each compatibility PR must include its own full-suite diff against that baseline, snapshot diff and real-model headless transcript. The product TRD requires Igor's approval before any merge, with this guard merged before feature branches.
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
Source: origin/main fcd92ec12f30ea113f44ef4887ea0d8c51556331
2+
Node: v24.3.0
3+
Bun: 1.4.0
4+
5+
bun run typecheck: PASS (exit 0)
6+
bun run lint: PASS (exit 0)
7+
bun run test:ci: FAIL (exit 1)
8+
Test Files 1 failed | 851 passed | 5 skipped (857)
9+
Tests 2 failed | 12194 passed | 72 skipped (12268)
10+
Duration 706.39s (tests 93%, import 4%, setup 2%, transform 1%)
11+
12+
Existing failing tests:
13+
tests/core/agent/AgentDependencyComposer.outcomes.test.ts > hook context delivery > keeps subagent-start context out of the lead conversation (async: false)
14+
tests/core/agent/AgentDependencyComposer.outcomes.test.ts > hook context delivery > keeps subagent-progress context out of the lead conversation (async: false)
15+
16+
No source changes in the baseline checkout. Dependencies are linked through a node_modules symlink; the same dependencies are used by the candidate.
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# F-0158.0 validation
2+
3+
Status: draft. Full regression verification is outstanding. Do not merge.
4+
5+
Base: `origin/main` at `fcd92ec12f30ea113f44ef4887ea0d8c51556331`. The original local main and unrelated worktrees were preserved.
6+
7+
## Main baseline
8+
9+
`baseline.txt` contains the exact two existing hook-context failures. Untouched main completed `bun run test:ci` in 706.39 seconds: 12,194 passed, two failed, 72 skipped. Typecheck and lint passed.
10+
11+
## Focused candidate evidence
12+
13+
- The initial contract test failed because the existing Commander program was not exported; it passed after adding only `export` to that declaration.
14+
- The committed-main regression test failed before its implementation and passed afterward. It verifies that changing both the working golden and an existing option still fails against the committed main contract, and that a missing main ref produces an actionable error.
15+
- All 13 focused tests passed. Removal, spelling, required/optional/variadic arity, negation and undefined/null/false default changes are covered.
16+
- Candidate typecheck, lint and build passed.
17+
- Root help, resume help and MCP-add help matched untouched main byte for byte on stdout and stderr, with exit code zero.
18+
- Snapshot diff: initial golden adds 189 option records across 51 command paths (94 root options). No existing flag definition changed. CI's test job fetches history for the committed-main comparison.
19+
20+
## Real-model headless transcript
21+
22+
Both untouched source main and the built candidate used the existing Autohand account with Fantail in isolated temporary settings. Hooks, MCP and sync were disabled in those temporary settings; stored user settings were preserved. Test credentials are not part of this repository.
23+
24+
```sh
25+
bun dist/index.js -p 'Reply with exactly CC_SNAPSHOT_OK. Do not use tools.' \
26+
--model fantail --config "$TEST_CONFIG" --path "$TEST_WORKSPACE" \
27+
--ephemeral --offline --restricted
28+
```
29+
30+
Candidate stdout:
31+
32+
```text
33+
› Reply with exactly CC_SNAPSHOT_OK. Do not use tools.
34+
CC_SNAPSHOT_OK
35+
Completed in 0m 03s · 31.4k tokens used
36+
```
37+
38+
Exit code: 0. The baseline also returned the standalone expected answer and exited 0.
39+
40+
A preliminary baseline attempt with `--bare` was rejected because bare mode requires an explicit API key/helper. It did not reach a model. The account-authenticated transcript above omits `--bare`.
41+
42+
## Aggregate validation boundary
43+
44+
The candidate's final `bun run test:ci` crossed repeated macOS sleep/wake cycles. It lasted 64,056.53 seconds, reported 154 failures and six worker errors, and left some files unexecuted. Individual tests exceeded their configured 30-second deadlines by many minutes. These results are preserved locally and are not a valid zero-new-failures comparison against the uninterrupted 706-second main run.
45+
46+
The required `bun run proof` was invoked after wake. Further sleep cycles again invalidated test timing, and the run was deliberately interrupted. `ioreg` confirmed the Mac lid was closed. No global power settings were changed.
47+
48+
The 13 focused tests and live transcript do not replace aggregate regression proof. A fresh uninterrupted `bun run test:ci` and required proof run remain necessary. Their failure sets must be compared with `baseline.txt` before a merge-ready or zero-new-failures claim.
49+
50+
The TRD requires this guard to merge first and prohibits merging until Igor gives the go. No feature branches are stacked on this unmerged guard. F-0168 continues to wait for F-0113; F-0170 remains design/spike only. No package release is part of this PR.

‎src/index.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -320,7 +320,7 @@ import { isDiscoveryInvocation, registerDiscoveryCommand } from './discovery/cli
320320

321321
installProcessErrorHandlers();
322322

323-
const program = new Command();
323+
export const program = new Command();
324324

325325
const collectRepeatable = (value: string, previous: string[] = []): string[] => [...previous, value];
326326

0 commit comments

Comments
 (0)