fix(bin): stop dead harness agents from reporting stale liveness - #2501
Open
SimTet wants to merge 5 commits into
Open
fix(bin): stop dead harness agents from reporting stale liveness#2501SimTet wants to merge 5 commits into
SimTet wants to merge 5 commits into
Conversation
…fallback Consult the existing recovery-grade fm_backend_agent_state classifier before trusting a frozen busy-hook record or a stale status-log line, and broaden the fleet-snapshot heartbeat liveness check beyond secondmates. Only a confident dead/missing verdict changes behavior; every inconclusive verdict and every backend without a classifier falls through unchanged.
…d panes The raw event-stream drain in fm_backend_herdr_wait_transition forwards whatever pane_id the reader delivers with no cross-check against the caller's own subscribed window list. A torn-down task's former pane, or its surviving husk shell, could therefore keep delivering agent-status edges long after teardown removed its state/<id>.meta, each one waking firstmate with nothing actionable to act on. handle_push_transition now requires the window to match a currently-recorded task before treating an edge as actionable; an unrecorded pane's transition is committed (so it is never re-evaluated as fresh) and absorbed silently. A live, recorded task's detection is untouched.
…ws fallback Broadening fm-fleet-snapshot.sh's agent liveness check (previous commit) now calls the recovery-grade tmux classifier for every kind, not just secondmates. This fixture's fake tmux had no list-windows handler, so the classifier's session-inventory read silently succeeded with empty output and was read as a confident "missing" endpoint, turning an expected endpoint.status of "unknown" into a false "dead". Fail list-windows generically instead, matching real tmux's behavior for an unconfigured session, so the classifier reads unreadable and the existing assertion holds.
handle_push_transition now also skips a pane no task currently records, not just secondmate endpoints and declared pauses - keep the exemption list here accurate to match.
…ate pre-check before busy state
This was referenced Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
bin/fm-crew-state.sh: when a crew has no matching run, consult the recovery-gradefm_backend_agent_stateclassifier before falling back to busy-hook or status-log state; a confidentdead/missingverdict now reportsunknown · agent-stateinstead of trusting a frozen busy read or stale log line from a harness process that has died but whose pane/shell still answers.bin/fm-fleet-snapshot.sh:endpoint.agent_aliveis now populated for every task with a recorded target (not only secondmates), reusing the samefm_backend_agent_aliveclassifier; tasks with no recorded target still reportnot_checked.bin/fm-push-transition-lib.sh: the push-transition handler now checksfm_backend_meta_for_windowbefore routing a transition; a pane with no currently recorded task (a torn-down task's former pane, or an endpoint Firstmate never owned) has its transition committed and absorbed instead of being misrouted to the wrong crew.docs/architecture.mdanddocs/herdr-backend.mdto describe the new agent-state pre-check ordering and the push-transition retired-pane exemption, and added/extended tests intests/fm-crew-state.test.sh,tests/fm-supervision-events.test.sh, andtests/fm-bearings-snapshot.test.shcovering these paths.Risk Assessment
✅ Low: Two well-scoped liveness-detection fixes (crew-state no-run fallback and herdr push-transition retired-pane absorption) reuse existing, single-owner classifiers at the correct shared boundary, preserve byte-identical behavior for every inconclusive verdict, and are backed by regression tests that exercise real function behavior (fake tmux/herdr process trees, real fm_backend_agent_state calls) rather than source-text matching; the author also caught and fixed their own test-fixture fallout from the broadened fm-fleet-snapshot.sh scope in a follow-up commit.
Testing
All five targeted bash test suites relevant to this branch's crew-liveness and retired-pane fixes pass cleanly (133 assertions total across the suites), including every new regression test the branch adds for the crew-exit-probe incident (confirmed-dead vs confirmed-alive vs inconclusive agent-state verdicts, and retired-pane push-edge absorption vs live-pane wake); this is a bash CLI/library codebase with no UI surface, so the CLI test transcripts are the appropriate evidence and are captured below rather than as visual artifacts.
Evidence: fm-crew-state.test.sh — new confirmed-dead-agent regression assertions
Evidence: fm-supervision-events.test.sh — retired-pane absorption vs live-pane wake
Pipeline
Updates from git push no-mistakes
⏭️ **intent** - skipped
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
./tests/fm-crew-state.test.sh (54/54 pass, incl. 5 new: confirmed-dead agent overrides stale status log / frozen busy hook, confirmed-alive agent still reports working, ambiguous+unreadable verdicts fall through byte-identical to baseline, unverified backend falls through unchanged)./tests/fm-supervision-events.test.sh (9/9 pass, incl. 2 new: a pane with no recorded task is absorbed - never wakes the supervisor - while the identical pane on a live recorded task still wakes)./tests/fm-bearings-snapshot.test.sh (40/40 pass; exercises fm-fleet-snapshot.sh's now-broadened agent_alive check end-to-end and validates the dfabbeb fixture fallback fix for list-windows)./tests/fm-secondmate-liveness.test.sh (15/15 pass; exercises the underlying fm_backend_agent_state/fm_backend_agent_alive classifier reused by this branch's callers, confirming the classifier itself is untouched)./tests/fm-fleet-snapshot-view.test.sh (15/15 pass; confirms secondmate agent-liveness rendering is unaffected)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.