Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 32 additions & 5 deletions bin/fm-crew-state.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# fixed mapping logic, no heuristics and no LLM. Output is one stable, parseable,
# token-tight line firstmate can read every heartbeat:
#
# state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|none> · <detail>
# state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|agent-state|none> · <detail>
#
# Logic, in order:
# 1. Resolve worktree + backend target + kind from state/<id>.meta.
Expand All @@ -39,13 +39,19 @@
# the run-step shows the run moved on, the log is deterministically stale and
# is flagged superseded. A genuinely parked run plus a needs-decision log
# agree, and are reported as parked.
# 4. No run for this crew (pre-validation, or kind=scout): fall back to the
# 4. No run for this crew (pre-validation, or kind=scout): consult the
# recovery-grade agent-state classifier (fm_backend_agent_state, owned by
# bin/fm-backend.sh) first; a CONFIDENT dead/missing verdict reports
# unknown · agent-state instead of trusting a frozen busy-hook read or the
# status log - closes the incident where a harness process died but its
# pane/shell kept answering as "working" forever. Any other verdict
# (unverified/unreadable/ambiguous) changes nothing: fall back to the
# recorded backend's pane busy state, then the status log's last line only
# when its verb maps to a recognized run-state. Decision-only events such as
# `resolved` never become current state or detail.
# 5. Missing meta or torn-down worktree: report unknown · none. If no run is
# attributed to this crew, a dead endpoint also reports unknown · none rather
# than trusting a stale status log.
# 5. Missing meta or torn-down worktree: report unknown · none. An unreadable
# backend target (pane/session itself gone) also reports unknown · none;
# see 4 for the agent-state check applied while the target still answers.
#
# Read-only and side-effect free. Always exits 0 on a successful read regardless
# of state; exit 2 only on a usage error (no id).
Expand Down Expand Up @@ -539,6 +545,27 @@ fi
[ -n "$BACKEND_TARGET" ] || emit unknown none "no backend target recorded"
pane_readable "$BACKEND_TARGET" || emit unknown none "backend target gone: $BACKEND_TARGET"

# Recovery-grade agent liveness, ahead of both the busy-hook read and the
# status-log fallback below: pane_readable only proves the pane/shell answers,
# not that a harness process is actually running in it, and both
# crew_busy_verdict (frozen hook state) and the status log's last line can keep
# reading "working" forever once that process is dead while its shell survives.
# fm_backend_agent_state is the one owner of this question (bin/fm-backend.sh;
# already used by fm-control.sh, fm-spawn.sh, fm-bootstrap.sh, fm-watch.sh,
# fm-stow-cascade.sh) - reused here rather than forking a second classifier.
# Only a CONFIDENT dead/missing verdict changes behavior: unverified (backend
# has no classifier - today only tmux and herdr do), unreadable, and ambiguous
# all fall through with zero effect, byte-identical to before this check
# existed, because a false "crew is dead" would trigger recovery against a live
# worker - worse than the silent-working gap this closes. Cost is one bounded
# backend-native call (a handful of local tmux/ps reads, or one herdr API call)
# added to the no-run fallback path only, never the run-step path above and
# never a network call.
AGENT_STATE=$(fm_backend_agent_state "$TASK_BACKEND" "$BACKEND_TARGET" 2>/dev/null) || AGENT_STATE=unreadable
case "$AGENT_STATE" in
dead|missing) emit unknown agent-state "harness process confirmed $AGENT_STATE: $BACKEND_TARGET" ;;
esac

# Secondmates idle on their own watcher (idle pane = healthy), so the busy
# state is not meaningful for them; read their state from the status log only.
# Only an exact busy verdict reports working here, and only an exact idle
Expand Down
11 changes: 8 additions & 3 deletions bin/fm-fleet-snapshot.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,13 @@
# against current_state; hints.pending_decision and hints.blocked_event are
# booleans derived from that set.
# endpoint.exists is the cheap backend endpoint-presence read.
# endpoint.agent_alive is populated for secondmates only, where it is useful
# return-channel supervision data; other tasks use "not_checked".
# endpoint.agent_alive is the recovery-grade agent-state verdict
# (fm_backend_agent_alive, bin/fm-backend.sh) for every task with a recorded
# target, remote or local: alive/dead when the backend has a confident
# classifier, else "unknown" (no classifier for this backend, or an
# inconclusive read) - never a false dead. A remote secondmate target reuses
# its own alive/dead/missing/other mapping above instead of this call.
# "not_checked" appears only when the task has no recorded target at all.
# scout_reports[]: present data/<id>/report.md pointers.
# main_inventory: {valid,reason,orphan_in_flight[],unstructured_current_count} -
# main-home current-inventory checks shared with secondmate_home_summary_json
Expand Down Expand Up @@ -510,7 +515,7 @@ task_json_lines() {
endpoint_exists=false
fi
fi
if [ "$kind" = secondmate ] && [ -n "$target" ]; then
if [ -n "$target" ]; then
agent_alive=$(fm_backend_agent_alive "$backend" "$target" 2>/dev/null || printf unknown)
fi
fi
Expand Down
15 changes: 15 additions & 0 deletions bin/fm-push-transition-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,21 @@ handle_push_transition() { # <backend> <session> <record>
to=$(fm_transition_to_status "$record")
[ -n "$pane_id" ] || { sleep 1; return; }
window="$session:$pane_id"
# The raw stream drain (bin/backends/herdr.sh's fm_backend_herdr_wait_transition)
# forwards whatever edge the reader delivers - it is not re-filtered against the
# caller's own subscribed pane list - so a torn-down task's former pane, or its
# surviving husk shell (crew-exit-probe incident, 2026-08-16: the harness process
# died but the pane/shell kept answering), can keep delivering edges long after
# teardown removed its state/<id>.meta. Recorded-task membership is the one
# authoritative test (matches recorded_windows' own poll-path filter): commit the
# transition so herdr's own escalation marker still advances - the same edge is
# never re-evaluated as fresh on a later drain - then absorb silently. A live,
# currently-recorded task is completely unaffected: this check is a no-op for it.
if ! fm_backend_meta_for_window "$window" "$STATE" >/dev/null; then
fm_backend_commit_transition "$backend" "$STATE" "$session" "$record" || exit 1
triage_log "absorbed push $to (no recorded task for this pane - retired or unknown endpoint): $window"
return
fi
task=$(window_to_task "$window" "$STATE")
if status_is_paused "$(last_status_line "$STATE/$task.status")"; then
triage_log "absorbed push $to (declared pause, awaiting external): $window"
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Any direct or remaining historical annotation prints every status line unread at
The script header owns the exact run-head ancestry rules.
During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`.
The most recent recognized ci log marker wins, so checks-green monitoring reports done while a later re-arm, failed-check, or issue marker returns the crew to working.
Only when no matching run exists does it consult semantic busy state; exact busy reports working, exact idle permits fallback to a status-log event whose verb maps to a recognized run-state, and unknown or a dead pane stays unknown instead of trusting a stale log.
Only when no matching run exists does it consult the recovery-grade agent-state classifier first, ahead of semantic busy state: a confident dead or missing verdict reports unknown before busy state is even read, closing the case where a harness process died but its pane kept answering as busy. Any other verdict falls through unchanged: exact busy reports working, exact idle permits fallback to a status-log event whose verb maps to a recognized run-state, and unknown or a dead pane stays unknown instead of trusting a stale log.
Decision-only events such as `resolved` never become current state or leak their prose into the current-state detail.
In that status-log fallback, a declared external wait reports the distinct `paused` state with its reason.
The semantic branch reports working only on an exact busy verdict and names the source that produced it; an unknown verdict never becomes working, never permits the status-log fallback, and never becomes a silent idle.
Expand Down
2 changes: 1 addition & 1 deletion docs/herdr-backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ Mid-session secondmate liveness is not implemented because idle secondmates are
Protocol 16 can subscribe to `pane.agent_status_changed` over one bounded Unix-socket reader.
`bin/fm-transition-lib.sh` owns the backend-neutral transition vocabulary and policy.
The Herdr adapter subscribes before reconciling current levels, buffers edges during reconciliation, and returns fresh blocked transitions for this home's panes.
The watcher maps the pane back to the task and skips secondmate endpoints and declared `paused:` waits.
The watcher maps the pane back to the task and skips secondmate endpoints, declared `paused:` waits, and any pane no task currently records (a torn-down task's former pane, or one Firstmate never owned).

The push path only shortens latency.
Polling runs every cycle and remains the permanent fallback when protocol 16, the event schema, Python, connection, subscription, or repeated reader execution is unavailable.
Expand Down
8 changes: 8 additions & 0 deletions tests/fm-bearings-snapshot.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,14 @@ case "${1:-}" in
*) printf 'all quiet\n> \n' ;;
esac
;;
list-windows)
# fm_backend_agent_alive's session inventory (bin/fm-fleet-snapshot.sh's
# endpoint.agent_alive, now populated for every kind, not just secondmates).
# This fixture has no real tmux session behind it, so fail generically -
# NOT one of the classifier's recognized missing-session/server/socket
# patterns - so it reads unreadable rather than a false "missing"/"dead".
echo "list-windows not configured for this fixture" >&2
exit 1 ;;
esac
exit 0
SH
Expand Down
Loading
Loading