bug(sync): /sync/status never clears consecutive_failures, last_error or last_sync_at after a successful sync — healthy devices report degraded forever

[thomaspufahl]

📋 Pre-flight Checks

• I have searched existing issues and this is not a duplicate
• I understand this issue needs status:approved before a PR can be opened

📝 Bug Description

GET /sync/status keeps reporting a device as degraded long after cloud sync has recovered. phase, consecutive_failures, reason_code, last_error and last_sync_at are never reset on a successful sync, so the endpoint describes a permanently broken device while replication is demonstrably working.

This makes the endpoint unusable as a health signal for per-device sync monitoring: it produces false negatives that are indistinguishable from a genuinely mute device.

🔄 Steps to Reproduce

1. Device with cloud autosync enabled (ENGRAM_CLOUD_AUTOSYNC=1, ENGRAM_CLOUD_SERVER and ENGRAM_CLOUD_TOKEN set) and engram serve running.
2. Cause repeated push failures so consecutive_failures climbs and backoff_until is set. In our case the trigger was the poisoned-prompt 500 from fix(sync): empty prompt content accepted at capture poisons cloud push for all projects; repair cannot fix prompt entities #686: mutation push: status 500: ... mutations[1]: prompt payload content is required for upsert.
3. Let backoff_until expire and let sync recover.
4. Write a new observation on that device and confirm it reaches the cloud (compare the project counts on the cloud dashboard).
5. curl http://127.0.0.1:7437/sync/status

✅ Expected Behavior

After a successful sync, /sync/status should reflect current state: phase back to healthy, consecutive_failures reset to 0, last_sync_at set to the successful timestamp, and last_error / reason_message cleared or explicitly marked as historical rather than current.

❌ Actual Behavior

The fields stay frozen at the failure state indefinitely while sync works. Observed on a device that was actively replicating:

phase:                degraded
consecutive_failures: 10
last_sync_at:         null
backoff_until:        2026-07-30T20:34:37.363077112-03:00    (expired ~14 h earlier)
reason_code:          transport_failed
last_error:           push: transport push project "owlvision": cloud: mutation push:
                      status 500: insert mutations: cloudstore: canonicalize materialized
                      mutation batch chunk: mutations[1]: prompt payload content is
                      required for upsert

End-to-end canary proving sync worked at that exact moment:

	before	after (90 s)
device local infra	373 obs / 80 sessions	374 / 81
cloud infra	373 obs / 80 sessions	374 / 81

On the same device, engram sync --cloud --project owlvision returned Nothing new to sync for project "owlvision" — all memories already exported, and /sync/status was byte-identical afterwards.

last_sync_at: null deserves special mention: the daemon had ~14 h uptime and was successfully pushing, yet the field was never populated at all.

Operating System

Linux (Arch / CachyOS), kernel 7.0.8-1-cachyos, systemd user unit running engram serve.

Engram Version

1.20.0 (client and self-hosted cloud server both on v1.20.0)

Agent / Client

Claude Code

📋 Relevant Logs

{"backoff_until":"2026-07-30T20:34:37.363077112-03:00","consecutive_failures":10,"dead_count":0,"deferred_count":0,"enabled":true,"last_sync_at":null,"phase":"degraded","reason_code":"transport_failed","reason_message":"push: transport push project \"owlvision\": cloud: mutation push: status 500: insert mutations: cloudstore: canonicalize materialized mutation batch chunk: mutations[1]: prompt payload content is required for upsert"}

💡 Additional Context

Impact: we are building per-device sync monitoring to alert when a workstation silently stops pushing. /sync/status is the only place that exposes sync lifecycle, and it currently cannot be trusted in either direction — see the companion issue about engram doctor returning status: ok while pending mutations are blocked.

Diagnosing a real incident with these two signals produced three consecutive wrong conclusions, in both directions. The only method that turned out to be reliable was the end-to-end canary above: write an observation and measure the cloud count delta.

[thomaspufahl]

Companion issue: #688 — engram doctor reports status: ok while those same pending mutations are blocked. Together they mean both available sync signals are unreliable, in opposite directions: this one is pessimistic (reports degraded while syncing), #688 is optimistic (reports ok with an undeliverable backlog).