Nothing watches whether a bound connector is receiving — WhatsApp was dark 9 days and only an unrelated audit noticed

[eumemic]

The gap

Nothing in the company watches whether a bound connector is actually receiving.

Established 2026-08-15: the WhatsApp connector — connection conn_01KTW15C6ARW02CT4AS6CK0CF5, unarchived, active allow-list, bound to session sess_01KTVY2HE8B7J5EF20ZP1KEFY2 — has had no session activity since 2026-08-06. Nine days. Its container (aios-whatsapp) is crashlooping roughly once a minute.

The only thing that noticed was an unrelated audit, whose alarm was titled "audit cron down" — naming the detector rather than the finding — so it accumulated 29 comments pointing at the healthy component while a user-facing channel stayed dark.

Why the existing monitoring could not catch it

The company watches disk, model-pool utilization, deploy lag, PR gates, hold staleness, issue limbo, lane liveness, sweep freshness, migration chains. All of those watch things that PRODUCE a signal when unhealthy.

A silent connector produces nothing. And "no inbound messages" is exactly what a healthy, quiet channel looks like. Absence of inbound is not evidence of health — and today there is no signal that tells the two apart.

This is the same shape as the two lane reconcilers that sat dead for 15–18 days (eumemic-company#251): a healthy-looking system with nothing asserting it is alive. That one was fixed with a liveness detector; connectors have no equivalent.

Proposal

A connector-liveness check over every unarchived connection:

• For each bound connection, compare its container/process state against its bound session's last activity.
• Alarm on the CONJUNCTION, which is the part that makes it low-noise: the transport is unhealthy AND the session has been silent beyond a per-channel threshold. Either alone is normal — a quiet channel is fine, and a brief restart is fine. Together they mean the channel is dark and nobody knows.
• Report the finding, not the detector: "whatsapp: container restarting, no session activity in 9d".

Thresholds should be per-channel — nine days of silence is alarming on an active support channel and meaningless on a rarely-used one. A single global threshold would produce exactly the noise that makes an alarm ignorable (see eumemic-ops#438).

Acceptance

• A connector whose container is unhealthy and whose session has been silent past its threshold produces an alarm naming both facts
• A quiet-but-healthy channel produces nothing — the true-negative, and the thing that decides whether this detector gets read or ignored
• Mutation test: stop a connector container in a test environment → the alarm fires. An untested detector is how the lane reconcilers went 18 days unnoticed.
• health_check_enabled is currently false on aios-whatsapp — worth asking whether every connector container should define what "healthy" means, since a restart loop with no success condition cannot converge.

Provenance: filed from the eumemic-ops#276 investigation, 2026-08-15.

[eumemic]

Superseded in part — the channel is retired, not broken (2026-08-15)

The chairman's answer to this issue's core question: nobody is really using WhatsApp. The container has been stopped deliberately rather than repaired.

The detector this issue proposes is still worth building, but its motivating example was partly wrong and that is worth recording:

The seat treated unarchived connection + active allow-list + bound session as evidence the channel was in use. It was evidence the channel was configured. Nine days of silence was the normal state of a channel nobody uses, not an outage.

This sharpens the design rather than invalidating it. A connector-liveness check that alarms on silence + unhealthy transport would have fired here and been wrong — exactly the false positive that turns a detector into wallpaper.

The missing third input is INTENT: is this channel supposed to be carrying traffic? That is not derivable from the connection record, which is why the seat got it wrong. Options: archive retired connections (making intent structural), or carry an explicit expectation per channel. The first is better — it makes the answer visible to every reader and detector rather than to one that knows to ask.

The seat's own connector_liveness_watch.sh will now correctly go quiet on WhatsApp — not because it is healthy, but because a stopped container plus a silent session no longer means anything is wrong. That is the bug in the current threshold design, found on its first day.

[eumemic]

Stall detector: stranded issue (no reconciler transition driving it)

Issue #2153 (shovel-ready ∧ approved) has had NO state change in 24.2h (> 24h band) and the reconciler is not driving it.

Why it looks stalled: shovel-ready ∧ approved with NO claim label and NO open PR — build-eligible but never dispatched; owner()'s build branch should have fired.

This is the uncorrelated stall alarm (READ + ESCALATE only — it does NOT act). The fix is to root-cause the missing/wrong owner() transition and patch it, then clear needs:human/stalled to release the item. (Verdict drawn from GitHub state, a substrate the reconciler cannot rewrite.)

[eumemic]

Design note (seat, 09:45Z): split PR #2208 — the shared runner health hook lands alone; per-connector readiness semantics become one follow-up each

#2208 hit the review-fix cycle cap (needs:human/livelock) after ~50 verdicts in 31 h. The last eight rounds alternated between three unrelated surfaces: the runner's heartbeat-claim path (now passing at 3ef6f2c0), Telegram long-poll readiness (Updater.start_polling() returns before the first successful getUpdates; PTB retries inside its background task, so the connection reads serving through an outage), and WhatsApp readiness (the connectionState=connected edge can fire before RpcListener.connect() subscribes, so a connection that is already receiving never becomes ready). Each is a real, connector-specific correctness problem; bundling them means every fix exposes the next and the PR never converges.

Scope for a narrowed PR (this issue): packages/aios-connector-http only — heartbeat attribution converging to the authoritative connection set after restart (the four-round fix that now passes), the /health receive-liveness surface, and the runner tests. Connectors keep their current behaviour; the hook is available for them to call.

Follow-ups (one issue each, needs-design until the runner lands): Telegram: mark ready only after the first successful getUpdates, unready on sustained poll failure. WhatsApp: readiness derived from daemon state on subscribe, not from observing an edge. Signal / Slack / Matrix / SMS: audit whether their start returns before a receive path is proven; likely the same shape.

The lane can rebuild from the existing branch with the connector files reverted, or the fixer can carve the runner commits out; either is fine. The livelock label stays until the narrowed PR exists.