Repository navigation
fix(mt#5101): Assert the driven-session host's MCP-config contract, not the exact server set - #3728
Conversation
…dependent server set Two mt#3377 tests in driven-session-host.test.ts asserted the --mcp-config payload's keys equal exactly ["minsky"]. Since mt#4239 the resolver also inherits entries from the daemon checkout's real .mcp.json (default set ["minsky", "github"]), so the exact key list depends on the machine running the tests: CI has no .mcp.json and passes; an operator machine with a github server fails both with received ["github", "minsky"]. Reproduced identically under bun 1.3.14 and 1.4.2 — not a runtime change. Both sites now assert what the host owes on every machine: minsky is present, it is the LAST key (synthesized after the inherited entries so nothing can shadow it), and its args carry the session cwd as --repo. The inherited set is the resolver's contract and is covered with an injected reader in driven-session-mcp-config.test.ts. No production change: startDrivenSession reaches the resolver through the transport with no readServers seam, and threading one through three files for two assertions is disproportionate. Negative control: the old assertions on this machine → 92 pass / 2 fail at :206 and :901. After: 94/0, and the full gated suite 18,359 pass / 0 fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0147JJ3SVgqRz9JyUD4NZBz9
Minsky Reviewer StatusVerdict: APPROVED — no blocking findings Commands
|
There was a problem hiding this comment.
Independent adversarial review (Chinese-wall)
Reviewer: minsky-reviewer[bot] via openai:gpt-5
Tier: 2
Scoped, test-only PR that replaces environment-dependent exact-set assertions with host-contract assertions looks correct and matches the task spec. I verified the two modified sites now assert minsky presence, last-key position, and preserved --repo arg; explanatory comments are included. One non-blocking suggestion: relying on key order via Object.keys().at(-1) can be brittle — consider strengthening the check by asserting the synthesized minsky entry’s distinguishing shape in addition to presence. No production behavior changed; no docs impact. Approving.
Findings
- [NON-BLOCKING] src/cockpit/driven-session-host.test.ts:210 — Order-based assertion on
Object.keyscan be brittle; consider asserting synthesizedminskyshape instead of relying on key order
The tests now assertserverNames.at(-1) === "minsky"(src/cockpit/driven-session-host.test.ts:210and:909). While JS preserves string-key insertion order forObject.keys, this couples the test to the serialization/build order of the payload. A future refactor that builds the object via a sorted entries path (or merges maps in a different order) would fail this test even if the host contract still holds (i.e., inheritedminskycannot shadow synthesizedminsky). Consider also asserting on the synthesizedminskyentry's distinguishing properties (e.g., thatparsed.mcpServers.minsky.argsincludes--repowithSCRATCH_CWD, and optionally that itstype/commandmatches the synthesized local-command shape) to reduce reliance on key order. This is advisory; current assertions are valid given today's builder guarantees.
Spec verification
| Criterion | Status | Evidence |
|---|---|---|
Both tests stop asserting the exact mcpServers key set and assert the HOST's own contract instead — which holds on every machine: the synthesized payload carries a minsky entry; it is the LAST key (mt#4239 writes it after the inherited entries so it always wins); its args carry the session cwd as --repo. The inherited set is NOT asserted here: it is the resolver's contract, covered with an injected reader in src/cockpit/driven-session-mcp-config.test.ts (AT1, AT2, "an inherited minsky entry can never shadow the synthesized one"). (Amended at planning, 2026-09-11 — the original "inject the operator server set" form is not reachable from these tests; see ## Planning Audit (READY) → gate (b).) |
Met | src/cockpit/driven-session-host.test.ts:206-216 and :901-911 replace toEqual(["minsky"]) with toContain("minsky") and expect(serverNames.at(-1)).toBe("minsky"), and retain expect(parsed.mcpServers.minsky.args).toContain(SCRATCH_CWD). |
The full gated suite (MINSKY_PREPUSH_FULL_SUITE=1 bun scripts/run-tests-gated.ts) passes on a machine whose .mcp.json declares a github server — the condition that fails today. |
Unverifiable | Execution outcome is not present in the diff and cannot be verified from repository contents alone. Criterion depends on a local machine state and CI run outside this PR. |
Negative control: the un-fixed assertion, run on such a machine, observed failing with received ["github", "minsky"]. (Observed at planning, 2026-09-11 18:30Z: 92 pass / 2 fail at driven-session-host.test.ts:206 and :901, both + "github"; grep -c '"github"' .mcp.json → 1. The implementer restates this in the PR body's test-first evidence — mt#3244 requires the negative control there.) |
Unverifiable | This is an execution observation, not encoded in the repo. The PR description claims it, but it cannot be verified from the diff. |
A one-line note in the test explains why the exact key set is not asserted (mt#4239's inheritance makes it a function of the daemon machine's .mcp.json). |
Met | src/cockpit/driven-session-host.test.ts:206-213 and :903-906 include explanatory comments stating inheritance from the daemon checkout’s .mcp.json and why the exact set is not asserted. |
Documentation impact
- no-update-needed — Test-only change in
src/cockpit/driven-session-host.test.tsadjusting assertions; no production behavior or public API changed. No docs updated in this PR and none appear necessary.
Cockpit PreviewPreview URL: https://cockpit-preview-production.up.railway.app Deployed from commit Updated by Cockpit Preview. |
There was a problem hiding this comment.
Independent adversarial review (Chinese-wall)
Reviewer: minsky-reviewer[bot] via openai:gpt-5
Tier: 2
Verification round: the prior non-blocking concern stands (order-based key assertion), but the test-only changes meet the spec. Both modified sites now assert the host contract (minsky present, last key, and --repo arg contains the session cwd) and include explanatory comments. No production code touched; no critical new defects introduced. I recommend optionally strengthening the assertion by relying less on key order, but this is advisory. Approving.
Findings
- [NON-BLOCKING] src/cockpit/driven-session-host.test.ts:214 — Order-based assertion on
Object.keys().at(-1)may be brittle; consider asserting synthesizedminskyshape too
The revised tests assertserverNames.at(-1) === "minsky"(start path at src/cockpit/driven-session-host.test.ts:214 and resume path at :909). While this matches today’s builder guarantee (synthesizedminskyappended after inherited entries), it couples the tests to key order. A future refactor that constructsmcpServersvia sorted entries or a different merge order would fail these tests even if the host contract ("inheritedminskycannot shadow synthesizedminsky") still holds. Consider additionally asserting on the synthesizedminskyentry’s distinguishing properties — e.g., thatparsed.mcpServers.minsky.argsincludes--repowithSCRATCH_CWD(already present) and, optionally, itstype/commandshape — and relaxing the last-key assertion to presence + shape if order is not essential.
Spec verification
| Criterion | Status | Evidence |
|---|---|---|
Both tests stop asserting the exact mcpServers key set and assert the HOST's own contract instead — which holds on every machine: the synthesized payload carries a minsky entry; it is the LAST key (mt#4239 writes it after the inherited entries so it always wins); its args carry the session cwd as --repo. The inherited set is NOT asserted here: it is the resolver's contract, covered with an injected reader in src/cockpit/driven-session-mcp-config.test.ts (AT1, AT2, "an inherited minsky entry can never shadow the synthesized one"). (Amended at planning, 2026-09-11 — the original "inject the operator server set" form is not reachable from these tests; see ## Planning Audit (READY) → gate (b).) |
Met | src/cockpit/driven-session-host.test.ts:206-216 and :901-911 replace toEqual(["minsky"]) with expect(serverNames).toContain("minsky") and expect(serverNames.at(-1)).toBe("minsky"), and retain expect(parsed.mcpServers.minsky.args).toContain(SCRATCH_CWD); comments explain the contract-based assertion. |
The full gated suite (MINSKY_PREPUSH_FULL_SUITE=1 bun scripts/run-tests-gated.ts) passes on a machine whose .mcp.json declares a github server — the condition that fails today. |
Unverifiable | Execution outcome depends on local machine state and CI; not present in the diff. Cannot be verified from repo contents. |
Negative control: the un-fixed assertion, run on such a machine, observed failing with received ["github", "minsky"]. (Observed at planning, 2026-09-11 18:30Z: 92 pass / 2 fail at driven-session-host.test.ts:206 and :901, both + "github"; grep -c '"github"' .mcp.json → 1. The implementer restates this in the PR body's test-first evidence — mt#3244 requires the negative control there.) |
Unverifiable | Pre-change execution evidence is not encoded in the repo; cannot be verified from this diff. |
A one-line note in the test explains why the exact key set is not asserted (mt#4239's inheritance makes it a function of the daemon machine's .mcp.json). |
Met | src/cockpit/driven-session-host.test.ts:206-213 and :903-906 include explanatory comments about inherited entries from the daemon checkout’s .mcp.json and why the exact set is not asserted. |
Documentation impact
- no-update-needed — This PR modifies only tests (
src/cockpit/driven-session-host.test.ts) to assert contract-based behavior; no production code or public API changed. No relevant docs are impacted or invalidated.
Summary
Two mt#3377 tests in
src/cockpit/driven-session-host.test.ts— "production default provisions the minsky MCP server" and "a resume re-provisions the MCP config rather than dropping it" — assertedObject.keys(parsed.mcpServers)equals exactly["minsky"]. Since mt#4239 the payload also INHERITS entries from the daemon checkout's real.mcp.json(readOperatorMcpServers(process.cwd()/.mcp.json), default set["minsky", "github"]), so the full key list is a function of the machine running the tests: CI has no.mcp.jsonand passes; any operator machine with agithubserver fails both with received["github", "minsky"]. Found by mt#5097's local full-suite run — the only two failures in 18,369 — and reproduced identically under bun 1.3.14 and 1.4.2, so this is not a runtime change.Key Changes
src/cockpit/driven-session-host.test.ts(only file): both sites now assert the HOST's own contract, which holds on every machine —minskyis present, it is the LAST key (mt#4239 synthesizes it after the inherited entries so nothing inherited can shadow it), and itsargscarry the session cwd as--repo(unchanged line). A comment at each site explains why the exact set is not asserted.driven-session-mcp-config.test.ts(AT1/AT2, "an inheritedminskyentry can never shadow the synthesized one").Why not inject the server set (the spec's original SC1):
startDrivenSessionreaches the resolver viaClaudeStreamJsonTransport.spawn→resolveMcpConfigForSpawn→resolveDrivenSessionMcpConfig(repoPath, { names }), which passes noreadServers/sourceDir. The host-level seams aremcpConfig(bypasses synthesis — the thing under test) andmcpServerNames. Threading a reader seam throughdriven-session-host.ts→driver-transport.ts→claude-transport.tson both the start and resume paths for two assertions is disproportionate, and the file already tolerates the analogous machine-dependence ofinvocation.command(the bun binary path) by not asserting on it. Recorded in the spec's## Planning Audit (READY); SC1 was amended at planning to the contract-assertion form.Alternative weighed and rejected:
mcpServerNames: ["minsky"]pins the set deterministically but changes what the test proves (an explicit set rather than the production default) and still reads the real file.Testing
Execution evidence:
grep -c '"github"' .mcp.json→ 1), in the session workspace:.mcp.jsonon the runner; the assertion is invariant across both cases).:206and:901now readexpect(serverNames).toContain("minsky"); expect(serverNames.at(-1)).toBe("minsky");with the existingtoContain(SCRATCH_CWD)line kept.MINSKY_PREPUSH_FULL_SUITE=1 bun scripts/run-tests-gated.ts, session workspace):## Outcome); the 10 skips are pre-existing (18,357 + 2 + 10 = 18,369).Negative control: the un-fixed assertions on the same machine (main checkout at
1c4eba401, pre-change tree) —bun test --preload ./tests/setup.ts --timeout=15000 src/cockpit/driven-session-host.test.ts→ 92 pass / 2 fail, bothexpect(received).toEqual(expected)atdriven-session-host.test.ts:206and:901with diff+ "github"above"minsky". That is the full pre-fix state (the change is the test itself), not a partial revert.Defect class: a test asserting an exact value derived from real machine state through a production default. Other members checked: every other
mcpServerskey assertion undersrc/is either fixture-backed (driven-session-mcp-config.test.tsinjectsreadServers) or in an unrelated module (src/mcp/setup/local-http-*); the--strict-mcp-configtest in this file asserts flag presence only. No other site reaches the real.mcp.json.Typecheck: pass (8 projects, session workspace). Lint: pass (4,460 files, session workspace).
Deploy verification:
src/**is a push trigger fordeploy-minsky-mcp.yml(the image COPYs the wholesrctree, tests included), so this merge redeploys minsky-mcp with no behavioural change. Post-merge:deployment_wait-for-latest(service: "minsky-mcp", notBefore: <mergeDate>, expectCommitSha: <mergeSha>)→ SUCCESS withbuildIdentityread, then/healthassertingservice: minsky-mcp. Recorded on the task spec.🤖 Generated with Claude Code
https://claude.ai/code/session_0147JJ3SVgqRz9JyUD4NZBz9