samohost-trigger cycle exits FAILURE when a single app faults (samograph port-squatter / cd apps/web; game-changers config.js) — needs per-app isolation

[samo-agent]

Symptom

The samohost-trigger service exits status=1/FAILURE on ~every 3-minute cycle, marking samohost-trigger.service red, even though most of the fleet deploys/provisions cleanly. The whole cycle is reported as a failure because a small number of specific, unrelated apps fault each pass:

• samograph: a port-squatter error, plus cd apps/web: No such file or directory during its deploy script.
• game-changers: a config.js conflict.

Meanwhile other apps in the same fleet (e.g. friends-of-twin-peaks, gregg-brandalise) deploy/provision fine on the same cycle. So a couple of bad apps drag the entire trigger cycle to FAILURE and turn the unit red every 3 minutes — a persistent, recurring failure/alarm-fatigue signal that hides whether anything is actually fleet-wide wrong.

Root cause (shape)

Per-app faults are not isolated from the overall cycle result: one app's non-zero/error outcome propagates to the cycle exit code, so the unit goes red regardless of how many other apps succeeded. This is the same "keep per-app isolation" concern that #52 called out for the CI-pending case, but here it's for runtime app faults (port squatting, cwd/script errors, a stale/conflicting config file) rather than CI state.

Recommendation

1. Isolate per-app failures so one bad app cannot fail the entire trigger cycle. A cycle should aggregate per-app outcomes and only exit non-zero on a genuine fleet-level problem; individual app faults should be counted/reported per-app (and surfaced in the journal) without poisoning the unit's overall status. (Complements trigger: pending/red CI should be a transient SKIP, not a FAILED (unattended timer marks every pre-CI cycle as failed) #52's per-app isolation, and the diagnostics-surfacing asks in app deploy swallows remote script output — failures report outcome=incomplete with zero diagnostics #123/trigger: PR-preview failures are swallowed (noop sinks) and excluded from the failed counter — timer reports success while previews fail every cycle #125.)
2. Fix or park the underlying app faults themselves:
   • samograph: the port-squatter and cd apps/web: No such file or directory errors (the latter is the same cwd/monorepo-subdir class as app deploy: script cwd leaks between phases — buildCmd's cd apps/web breaks the subsequent migrate phase #122 — confirm it isn't a regression of that fix).
   • game-changers: the config.js conflict.

Impact

systemctl status samohost-trigger.service is red on essentially every cycle, so it can no longer be trusted as a health signal for the fleet — a real fleet-wide breakage would look identical to the current couple-of-bad-apps state. This is an active, recurring failure worth its own investigation.

Related

• trigger: pending/red CI should be a transient SKIP, not a FAILED (unattended timer marks every pre-CI cycle as failed) #52 (CLOSED): per-app isolation for the CI-pending-vs-failed case — same isolation principle, different trigger (CI state vs runtime fault).
• app deploy: script cwd leaks between phases — buildCmd's cd apps/web breaks the subsequent migrate phase #122 (CLOSED): cd apps/web cwd-leak in the deploy script — likely the same class as the samograph cd apps/web error.
• app deploy swallows remote script output — failures report outcome=incomplete with zero diagnostics #123 / trigger: PR-preview failures are swallowed (noop sinks) and excluded from the failed counter — timer reports success while previews fail every cycle #125: swallowed deploy/preview diagnostics — the reporting side of making per-app faults visible without failing the whole cycle.