bug: intermittent 503 Service Unavailable from Railway (~10% hit rate)

[r1ckyIn]

Summary

Production backend returns 503 Service Unavailable (plain text, not JSON) intermittently, affecting ~10% of requests.

Evidence

Stability test (20 requests, 2026-04-11):

• /health: 9/10 → 200, 1/10 → 503
• /api/v1/sync/status: 9/10 → 401 (correct), 1/10 → 503

Key observation: 503 response is plain text Service Unavailable, not FastAPI's JSON error format. This indicates Railway's reverse proxy returning 503, not the Python app.

Possible Causes

1. Railway container OOM restart (memory pressure from sync + APScheduler + DB pool)
2. Railway sleep/wake cycle on hobby plan
3. DB pool exhaustion causing worker hangs → Railway health check timeout → restart

Impact

• Frontend requests may randomly fail
• Sync background tasks may be interrupted mid-execution
• User experience: occasional loading errors requiring retry

Monitoring

Phase 31.1 (#60) added Sentry observability — check Sentry for correlating errors during 503 windows.

UAT Reference

Phase 31 backend health probe, 2026-04-11

🤖 Generated with Claude Code