Skip to content

fix(bin): stop dead harness agents from reporting stale liveness - #2501

Open
SimTet wants to merge 5 commits into
kunchenguid:mainfrom
SimTet:fm/fm-liveness
Open

fix(bin): stop dead harness agents from reporting stale liveness#2501
SimTet wants to merge 5 commits into
kunchenguid:mainfrom
SimTet:fm/fm-liveness

Conversation

@SimTet

@SimTet SimTet commented Aug 16, 2026

Copy link
Copy Markdown

What Changed

  • bin/fm-crew-state.sh: when a crew has no matching run, consult the recovery-grade fm_backend_agent_state classifier before falling back to busy-hook or status-log state; a confident dead/missing verdict now reports unknown · agent-state instead of trusting a frozen busy read or stale log line from a harness process that has died but whose pane/shell still answers.
  • bin/fm-fleet-snapshot.sh: endpoint.agent_alive is now populated for every task with a recorded target (not only secondmates), reusing the same fm_backend_agent_alive classifier; tasks with no recorded target still report not_checked.
  • bin/fm-push-transition-lib.sh: the push-transition handler now checks fm_backend_meta_for_window before routing a transition; a pane with no currently recorded task (a torn-down task's former pane, or an endpoint Firstmate never owned) has its transition committed and absorbed instead of being misrouted to the wrong crew.
  • Updated docs/architecture.md and docs/herdr-backend.md to describe the new agent-state pre-check ordering and the push-transition retired-pane exemption, and added/extended tests in tests/fm-crew-state.test.sh, tests/fm-supervision-events.test.sh, and tests/fm-bearings-snapshot.test.sh covering these paths.

Risk Assessment

✅ Low: Two well-scoped liveness-detection fixes (crew-state no-run fallback and herdr push-transition retired-pane absorption) reuse existing, single-owner classifiers at the correct shared boundary, preserve byte-identical behavior for every inconclusive verdict, and are backed by regression tests that exercise real function behavior (fake tmux/herdr process trees, real fm_backend_agent_state calls) rather than source-text matching; the author also caught and fixed their own test-fixture fallout from the broadened fm-fleet-snapshot.sh scope in a follow-up commit.

Testing

All five targeted bash test suites relevant to this branch's crew-liveness and retired-pane fixes pass cleanly (133 assertions total across the suites), including every new regression test the branch adds for the crew-exit-probe incident (confirmed-dead vs confirmed-alive vs inconclusive agent-state verdicts, and retired-pane push-edge absorption vs live-pane wake); this is a bash CLI/library codebase with no UI surface, so the CLI test transcripts are the appropriate evidence and are captured below rather than as visual artifacts.

Evidence: fm-crew-state.test.sh — new confirmed-dead-agent regression assertions
ok - a confirmed-dead agent overrides a frozen working status log
ok - a confirmed-dead agent overrides a frozen busy hook record
ok - an explicitly confirmed-alive agent still reports working from its busy record
ok - ambiguous and unreadable agent-state verdicts fall through byte-identical to baseline
ok - a backend with no recovery-grade classifier falls through unchanged
all fm-crew-state tests passed
Evidence: fm-supervision-events.test.sh — retired-pane absorption vs live-pane wake
ok - handle_push_transition: a pane with no recorded task (torn down, or never one) is absorbed - never a wake
ok - handle_push_transition: the identical pane still wakes once it is a recorded live task (fix is scoped to unrecorded panes)
# fm-supervision-events.test.sh: all assertions passed

Pipeline

Updates from git push no-mistakes

⏭️ **intent** - skipped

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • ./tests/fm-crew-state.test.sh (54/54 pass, incl. 5 new: confirmed-dead agent overrides stale status log / frozen busy hook, confirmed-alive agent still reports working, ambiguous+unreadable verdicts fall through byte-identical to baseline, unverified backend falls through unchanged)
  • ./tests/fm-supervision-events.test.sh (9/9 pass, incl. 2 new: a pane with no recorded task is absorbed - never wakes the supervisor - while the identical pane on a live recorded task still wakes)
  • ./tests/fm-bearings-snapshot.test.sh (40/40 pass; exercises fm-fleet-snapshot.sh's now-broadened agent_alive check end-to-end and validates the dfabbeb fixture fallback fix for list-windows)
  • ./tests/fm-secondmate-liveness.test.sh (15/15 pass; exercises the underlying fm_backend_agent_state/fm_backend_agent_alive classifier reused by this branch's callers, confirming the classifier itself is untouched)
  • ./tests/fm-fleet-snapshot-view.test.sh (15/15 pass; confirms secondmate agent-liveness rendering is unaffected)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

SimTet added 5 commits August 16, 2026 22:14
…fallback

Consult the existing recovery-grade fm_backend_agent_state classifier before
trusting a frozen busy-hook record or a stale status-log line, and broaden
the fleet-snapshot heartbeat liveness check beyond secondmates. Only a
confident dead/missing verdict changes behavior; every inconclusive verdict
and every backend without a classifier falls through unchanged.
…d panes

The raw event-stream drain in fm_backend_herdr_wait_transition forwards
whatever pane_id the reader delivers with no cross-check against the
caller's own subscribed window list. A torn-down task's former pane, or its
surviving husk shell, could therefore keep delivering agent-status edges
long after teardown removed its state/<id>.meta, each one waking firstmate
with nothing actionable to act on. handle_push_transition now requires the
window to match a currently-recorded task before treating an edge as
actionable; an unrecorded pane's transition is committed (so it is never
re-evaluated as fresh) and absorbed silently. A live, recorded task's
detection is untouched.
…ws fallback

Broadening fm-fleet-snapshot.sh's agent liveness check (previous commit) now
calls the recovery-grade tmux classifier for every kind, not just
secondmates. This fixture's fake tmux had no list-windows handler, so the
classifier's session-inventory read silently succeeded with empty output and
was read as a confident "missing" endpoint, turning an expected
endpoint.status of "unknown" into a false "dead". Fail list-windows
generically instead, matching real tmux's behavior for an unconfigured
session, so the classifier reads unreadable and the existing assertion holds.
handle_push_transition now also skips a pane no task currently records, not
just secondmate endpoints and declared pauses - keep the exemption list here
accurate to match.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant