MCP server ac.inference.sh/mcp has persistent liveness probe failures throughout the day (15+ events)

[Al629176]

Summary

The MCP server ac.inference.sh/mcp (server_id 00bbad23-4dab-443e-9f0b-ae73af1d6bb3) consistently fails liveness probes throughout the session, triggering multiple reconnect cycles. The reconnection loop itself also fails.

Log evidence (sample — 15+ occurrences observed)

05:49:48 WRN the liveness probe did not answer in time; keeping the session  outcome="timed_out" probe_timeout_seconds=8 consecutive_timeouts=1 teardown_after=3
06:17:46 WRN the transport failed its liveness probe; reconnecting  outcome="broken" elapsed=4.60s
06:28:55 WRN timed_out  consecutive_timeouts=1
06:51:06 WRN timed_out  consecutive_timeouts=1
07:07:57 WRN timed_out  consecutive_timeouts=1
07:19:07 WRN timed_out  consecutive_timeouts=1
07:51:17 WRN timed_out  consecutive_timeouts=1
08:09:27 WRN timed_out  consecutive_timeouts=1
08:42:18 WRN the transport failed; reconnecting  outcome="broken" elapsed=4.99s
09:10:36 WRN timed_out
10:21:54 WRN broken  elapsed=1.41s
10:33:03 WRN timed_out
10:55:01 WRN broken  elapsed=842ms
10:56:27 WRN reconnecting failed  failures=1 retry_in_seconds=5

Impact

• The MCP server for ac.inference.sh (inference toolkit) is effectively unavailable for large portions of the day.
• Any agent tool that routes through this MCP will silently fail or timeout.
• The consecutive_timeout counter resets to 1 on each observation but teardown_after=3 is never reached — the session is kept alive indefinitely despite being broken.

Questions

• Is ac.inference.sh a first-party service? If so, why is it consistently unreachable?
• teardown_after=3 — does this counter reset between WRN events, preventing teardown?
• Should a broken MCP session be tombstoned after N reconnect failures rather than retried indefinitely?

Expected behaviour

After teardown_after (3) consecutive failures, the session should be torn down and a user-visible notification shown. Reconnection should use exponential backoff, not a fixed 5-second interval.

Environment

• OpenHuman 0.63.17 (aarch64-apple-darwin)
• server_id: 00bbad23-4dab-443e-9f0b-ae73af1d6bb3
• qualified_name: ac.inference.sh/mcp
• Log date: 2026-09-09

[YellowSnnowmann]

Same server, same log pattern, and same conclusions as #5931 (closed 2026-09-03), whose fix — #5955 — shipped in v0.63.20; 0.63.17 predates it. Closing as a duplicate. Three claims in the description are worth correcting because they invert what the lines mean.

consecutive_timeouts=1 on every line means nothing was torn down and nothing was stuck. The counter is a streak (tinymcp ≥ v0.3.2, registry/supervisor/types.rs): one probe over the 8 s window keeps the session, three in a row tear it down, and a successful probe resets it. Fourteen lines at =1 spread over five hours means every one of them was followed by successful probes. The session was not "kept alive indefinitely despite being broken"; it was working between those lines.

The broken lines are the reconnects, and they reconnect. outcome="broken" is a transport-level failure of the probe's tools/list (reset/EOF/TLS/decode) and triggers an immediate reconnect; reconnecting failed failures=1 retry_in_seconds=5 is the first step of the reconnect backoff.

The backoff is exponential, not a fixed 5 s. vendor/tinymcp/crates/tinymcp/src/registry/supervisor/backoff.rs:16-30: BACKOFF_BASE * 2^(failures-1), capped at BACKOFF_MAX. 5 s is failures=1.

ac.inference.sh/mcp is a third-party server (inference.sh), not first-party. Measured from another network on 2026-09-02 it answered tools/list in 0.18–0.28 s across 49 probes with zero failures on both reused and fresh connections (table in #5931), so the timeouts are specific to the reporting machine's path, and the transport error text after the colon in the broken lines — truncated in both reports — is what would pin it.

What #5955 added (v0.63.20+): every probe outcome (ProbeTimedOut, TransportDropped with the error text as detail, Reconnected, ReconnectFailed, Parked) is an mcp_client domain event in Settings → Developer → Event Log, and a notification fires when a server stays down after a tick, comes back after failing, or is parked (src/openhuman/mcp/registry/supervisor_events.rs). A single slow probe stays Event-Log-only by design, so the pattern above produces rows, not banners. Upgrading gives you both the visibility and the error text asked for here.

[M3gA-Mind]

#6152 — ac.inference.sh/mcp "persistent liveness probe failures"

Verdict: MISDIAGNOSED. The log lines are real and were emitted as designed. All three defect claims in
the issue are contradicted by the code, and two of them are contradicted by the issue's own log evidence.

Not reproducible on this machine: zero liveness / consecutive_timeouts lines across every log in
~/.openhuman/logs/ (2026-09-03 through 2026-09-09). Alan's log is from his own build. So this is assessed
from code, not from a local repro.

ac.inference.sh/mcp is not first-party. The only occurrences of that string in the repo are test
fixtures (src/openhuman/desktop/notifications/bus_tests.rs:339,361,385,
app/src/lib/nativeNotifications/__tests__/*.ts). It is a third-party server the user installed.

Claim 1 — "consistently fails… effectively unavailable for large portions of the day"

Not supported by the evidence given. Every quoted timeout reads consecutive_timeouts=1. The streak
is cleared by a successful probe at
vendor/tinymcp/crates/tinymcp/src/registry/supervisor/types.rs:326-327 (ProbeOutcome::Alive →
self.timeouts.remove(&server_id)). A streak that is 1 on every observation is proof the server answered
in between — the opposite of persistently down.

With tick_interval = 60s (types.rs:64), ~15 timeouts across the quoted 05:49–10:56 window is roughly 15
slow probes out of ~300. That is a ~5% slow-probe rate against a remote HTTP server on the public internet.
Flapping latency, not an outage.

Claim 2 — "teardown_after=3 never reached — does the counter reset, preventing teardown?"

It resets, and that is correct. judge_timeout (types.rs:394-445) increments a per-server streak and
tears down at CONSECUTIVE_TIMEOUTS_BEFORE_TEARDOWN (3) in a row; a successful probe clears it
(:326), as does a Broken outcome (:355). Never reaching 3 means it never failed three consecutive
ticks — exactly what the counter is for.

The design is documented at types.rs:33-42: probe_timeout (8s) is deliberately shorter than
REMOTE_REQUEST_TIMEOUT (30s, registry/connections/types.rs:26,37), so a probe timeout means slow, not
dead, and one costs nothing on purpose.

Claim 3 — "Reconnection should use exponential backoff, not a fixed 5-second interval"

False, and refuted by the line quoted in the issue. vendor/tinymcp/.../supervisor/backoff.rs:20-32:
delay_after(failures) = 5s * 2^(failures-1), capped at 300s (BACKOFF_MAX, :14). Alan's
failures=1 retry_in_seconds=5 is the base case — the first failure. failures=2 yields 10s, =3
yields 20s, and so on. Nothing is fixed at 5s.

Claim 4 (impact) — "any agent tool that routes through this MCP will silently fail or timeout"

Wrong, and this is the one that matters. A probe timeout returns AfterProbe::Keep (types.rs:415):
the session is not disconnected. Real tool calls get the 30s REMOTE_REQUEST_TIMEOUT, not the 8s probe
window. A server answering in 9s fails the probe and serves every tool call normally. The probe is
telemetry, not a gate.

Claim 5 — "a broken session should be tombstoned rather than retried indefinitely… with a user-visible notification"

Both already exist, and the scoping is deliberate:

• Terminal parking: types.rs:266-281 inserts into self.terminal and emits SupervisorEvent::Parked.
• User-visible notification: src/openhuman/desktop/notifications/bus.rs:450-468 — "MCP server can't
  start / … will not be retried … Install the missing runtime, then disable and re-enable the server."
  with a deep link to /connections?tab=mcp.

Parking is scoped to error.is_missing_runtime() — a local prerequisite the user must fix. Network
failures are deliberately not parked; backoff.rs:10-14 states why: "the reason a server is down is
usually somewhere else entirely — its operator may fix it without anyone touching this host." Tombstoning
on N network failures would reverse a documented decision. That is a product call, not a bug fix.

The one real, already-known issue

probe_timeout = 8s is tight for a remote HTTP MCP over the public internet, and it produces exactly these
warnings for a merely-slow server. types.rs:44-58 already explains why it cannot simply be widened:
Supervisor::tick probes installs in sequence, so the probe window bounds the worst-case cycle;
widening to 30s multiplies the cycle by ~3.75 and past a handful of unresponsive installs the cycle outruns
tick_interval. The same comment names the prerequisite: "it needs the probe loop bounded first
(concurrent probes, or a cycle deadline)."

Blast radius (if anything is changed)

Everything above is in vendor/tinymcp, not openhuman — a submodule bump, and a gitlink bump is a
separate deliverable. openhuman only drives it: src/openhuman/mcp/registry/mod.rs:419-480 calls
Supervisor::tick from its own timer (which types.rs:52-53 explicitly flags as the host that gets no
MissedTickBehavior protection — though mod.rs:438 and :477 do set Delay and reset()).

Tests that pin current behaviour: vendor/tinymcp/crates/tinymcp/src/registry/supervisor/test.rs;
openhuman-side src/openhuman/mcp/registry/supervisor_events_tests.rs:44-138 and
src/core/events_tests.rs:520-533 pin teardown_after: 3 and the consecutive_timeouts values verbatim;
src/openhuman/desktop/notifications/bus_tests.rs:339-385 pins the parked notification text.

Proposed fix

No code change. The behaviour is correct, documented, and the mechanisms Alan asks for already exist.

If we want to act on anything, it is one of two small things, in order of laziness:

1. Nothing. Close as working-as-intended with the explanation. This is my recommendation.
2. If the warnings are genuinely too loud: make probe_timeout configurable per install so a user with a
   known-slow remote can raise it, rather than raising the global default. SupervisorConfig is already
   public and mod.rs:420 constructs it from default() — the seam exists. tinymcp change.

The thing not to build is the issue's "expected behaviour" (tear down after 3 + tombstone + exponential
backoff): two of those three are already implemented and the third reverses a documented decision.

Severity

Alan labelled P2. Disagree — this is P4 / not-a-bug. Nothing fails, nothing is lost, no tool call is
affected. The entire observable footprint is ~15 WRN lines a day about a third-party server that was
occasionally slower than 8 seconds.

Could not determine

• Whether ac.inference.sh was actually slow or actually down on Alan's machine on 2026-09-09. His log
  would settle it: grep -c "reconnected" against those timestamps. A reconnected … after_failures=N
  line between the timeouts (types.rs:254-265) confirms recovery. I do not have his log.
• Whether the 6 [socket] Ping timeout reconnects in this machine's log at 16:02–16:59 share a network
  cause with his MCP timeouts. Different subsystem (ws_loop_part_01.rs:590), and I did not chase it.