tracking: unified predicate-based stall detector — supersede the whack-a-mole pattern across releases

[nathanschram]

Audit-filed by untether-staging monitor loop, run 20260513T015418Z (passes 1-8 + retrospective synthesis).

Source: untether.service on lba-1, 0.35.3rc13.

Type: Tracking / design discussion (ENH-PATCH candidate for v0.35.4 implementation).

Context

Across four releases (v0.35.0 → v0.35.3) the stall detector has been refined for specific false-positive scenarios on every cycle — but each release adds another variant of "Untether thinks Claude is hung, but Claude isn't hung". The current design (warn on JSONL silence + CPU heuristic) keeps shipping single-context fixes; the underlying signal model is what's not scaling.

What's been fixed (and is still in the codebase)

Release	Issue	What it added
v0.35.0	#92	Foundation: stall warn after 5 min of no progress events
v0.35.0	#95	Detect stalls when no events arrive after StartedEvent
v0.35.0	#99	Auto-cancel sleep-then-stall loop after laptop sleep
v0.35.0	#105	Suppress when tool execution is active
v0.35.0	#115	Liveness watchdog false-positive auto-cancel guard
v0.35.0	#121	Suppress during CPU-active extended thinking
v0.35.0	#154	Longer threshold + contextual messaging for MCP tool calls
v0.35.0	#155	Frozen ring buffer escalation broadened beyond MCP
v0.35.0	#168	Suppress when main proc sleeping but children CPU-active
v0.35.0	#188	Verbose/misleading messaging cleanup for long-running tools
v0.35.1	#264	False positives during normal Agent/Bash workflows
v0.35.2	#349	Half-fix: visible chat indicator for rate-limit waits added, but the structured stall_warning still fires alongside it

What's still firing in v0.35.3 (OPEN)

Issue	Variant
#470	Post-result idle (after last_event_type=result) keeps firing stall warnings every 3 min — known benign noise
#481	Long-running Bash + ScheduleWakeup waits look hung in chat
#499	ExitPlanMode approval-pending session fires 22 stall warnings while user reads the plan
#526	Approval-pending + rate-limit-retry combo fires stall warnings despite Untether's own chat indicator showing the rate-limit
#482	Upstream Bash tool_result interim deltas (sibling — partial mitigation of #481)

Today's audit produced fresh evidence for both #499 (brand-copilot, peak_idle 5621s, 22 stall_warning fires this morning) and #526 (legal-librarian-local, 900s idle window mixing approval-pending + rate-limit-retry).

Proposal

Replace the implicit "stall = no JSONL events + tree_active check" with an explicit predicate-based decision at stall-detection time. The stall warning fires only when none of the predicates are true.

Predicates (initial set)

Predicate	True when	Source signal
approval_pending	A control_request is awaiting user response; inline keyboard is still rendered	_OUTLINE_REGISTRY / control-request registry / progress_edits keyboard state
rate_limit_waiting	The last rate_limit_event was within the past 60s and no assistant/result event has cleared it	claude.rate_limit_event log line (already emitted)
bash_running	An action with last_action='tool:Bash …' started < N seconds ago and hasn't shown (done)	existing last_action field
mcp_in_flight	An MCP tool call started < N seconds ago	existing MCP-aware threshold path (#154)
extended_thinking	Claude is in assistant.thinking mode with CPU active	existing tree_active + CPU check
post_result_idle	last_event_type=result AND idle_seconds < watchdog_threshold	#470's framing

Behavioural change

• Today: stall_warning fires every 3 min on JSONL silence + CPU heuristic. All 6 of the above scenarios produce warnings.
• Proposed: stall_warning fires only when no predicate is true. Each predicate gets its OWN structured event (subprocess.approval_pending, subprocess.rate_limit_waiting, etc.) with a much longer re-fire interval (e.g. 30 min) so they remain observable for operators but don't false-alarm.
• Chat-side: existing per-scenario UX (rate-limit indicator from enhancement: surface rate_limit_event as a visible "waiting for API" indicator, not a silent cancel #349; bash/wait surface from feat(progress): surface long-running Bash/ScheduleWakeup waits in chat — silent 5–10 min holds look hung #481; approval keyboard) keeps working unchanged. The stall message is suppressed in those states.

Daemon integration

The always-on untether-issue-watcher daemon currently treats subprocess.liveness_stall and progress_edits.stall_detected as auto-fileable. Once predicates ship, the daemon should filter on the predicate fields (pending_approval=true → skip, rate_limit_waiting=true → skip, …) so it doesn't keep auto-filing benign waits.

Related code

• src/untether/runner_bridge.py (stall detection + escalation; _should_emit_stall_warning)
• src/untether/runners/claude.py (_OUTLINE_REGISTRY, _DISCUSS_APPROVED, control-request registry, rate_limit_event handler)
• src/untether/utils/proc_diag.py (existing tree_active / cpu_active checks — keep, just demoted to fallback)
• ~/.local/bin/untether-issue-watcher (predicate-aware filter for auto:error-report filings)

Effort estimate

M — touches 2-3 files in src/untether/, requires a structured event taxonomy decision, and needs daemon-side dedup updates. Predicates can ship incrementally (start with approval_pending + rate_limit_waiting for the biggest wins; add bash_running / mcp_in_flight as #481/#482 land).

Why this is worth the refactor

• Eliminates 4 distinct OPEN issues in one design pass (stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470, feat(progress): surface long-running Bash/ScheduleWakeup waits in chat — silent 5–10 min holds look hung #481, monitor: unhandled warning 'progress_edits.stall_detected' fires on 'brand-copilot' #499, ENH-PATCH: differentiate approval-pending stalls from genuine hangs in stall warnings + Telegram messaging #526)
• Closes the half-fix from enhancement: surface rate_limit_event as a visible "waiting for API" indicator, not a silent cancel #349
• Stops the daemon (auto:error-report) and /monitor (auto:monitor-audit) from both auto-filing benign waits — today both treat subprocess.liveness_stall as automatically fileable, contributing to the comment-saturation pattern noted in the current audit run
• Makes the stall detector behaviour debuggable from a single decision site instead of N scattered suppressors

Out of scope

• Genuine subprocess hangs (post-OOM, deadlock, signal-death) — these continue to fire subprocess.liveness_stall as today; that's exactly the signal that should remain after benign cases are filtered out
• The session.summary.last_event_type field semantics observation from this audit (separate observability gap, sibling to bug: runner's proc.wait() doesn't fire when CC subprocess exits cleanly post-result — watchdog is the only safety net #502)
• Upstream Claude Code rate-limit retry behaviour (bug: Stream idle timeout - partial response received on long opus 4.7 plan-mode runs (upstream Anthropic API stall) #438) — orthogonal

— filed by untether-staging monitor loop, run 20260513T015418Z retrospective synthesis

[nathanschram]

Audit-filed by untether-staging monitor loop, pass 1, run 20260516T032233Z, at 2026-05-16T03:23Z.

Source: untether.service on lba-1, 0.35.3rc16.
Host: lba-1

First post-#527 evidence ping from a /monitor run. The whack-a-mole pattern this issue calls out is still active on rc16:

signal	source	last 24h on lba-1 staging	already-tracked?
progress_edits.stall_detected	runner_bridge	7 events across brand-copilot+tax	yes — #499
subprocess.liveness_stall	runners.claude	4 events (2 sessions × 2 chats)	yes — daemon auto:error-report
session.summary with stall_warnings>0	runner_bridge	2 sessions (brand-copilot stall_warnings=4, tax stall_warnings=3)	yes — #495
outbox.skipped (directory)	telegram.outbox_delivery	13 events / 3 projects / 6 directories	yes — #524

All four signals on the brand-copilot session at pid=194527 (2026-05-15T21:10:46 onward) describe the same underlying state: session healthy, parked on user-approval, watchdog fires — yet each signal is reported through a different code path, with different cadences, and different downstream issues. That's the exact fragmentation #527 proposes to consolidate.

Concrete pattern observed today (one session, multiple signals from same parked-on-approval state):

1. subprocess.liveness_stall fires at idle=600s (10 min) — Telegram warning feat: interactive enhancements — AskUserQuestion, export, browse, cost, triggers #1
2. progress_edits.stall_detected fires at idle=1819s (30 min) → 2179s (36 min) → 2359s (39 min) — three more Telegram warnings, escalating tone
3. session.summary writes liveness_stalls=1 stall_warnings=4 peak_idle_seconds=2359.6 — one summary number that conflates "stalled" (bad) with "awaiting approval" (fine)
4. outbox.skipped (post-approval, on a directory the agent tried to deliver) — fifth distinct signal from the same logical session, again with no agent feedback loop

The proposed unified predicate (per the issue body: "the session is in a known waiting-on-user state") would correctly suppress signals 1-2-3 and reclassify the session.summary numerator on signal 4 — replacing four user-visible Telegram nag messages with zero.

Suggested concrete predicate to land in rc17: is_awaiting_user_input(state) := last_action.startswith("warning:Permission Request") OR _PENDING_ASK_REQUESTS.get(session_id) is not None. Wire this into both subprocess.liveness_stall and progress_edits.stall_detected as a "downgrade to info + reframe text" check before sending Telegram.

— filed by untether-staging monitor loop, pass 1

[nathanschram]

Moved to v0.35.5. v0.35.4 is now a pure bug-fix patch, so the broad unified predicate-based stall detector redesign (an enhancement) belongs here. The narrow bug fix for the false-alarm stall warnings stays in v0.35.4, tracked under #495/#499/#500: add the approval_pending + rate_limit_waiting predicates so the detector stops firing while a session legitimately awaits user approval. This issue then carries the remaining predicate taxonomy (bash_running, mcp_in_flight, extended_thinking, post_result_idle) and the structured-event/dedup work.