Investigate why 5xx alert mitigations did not work

[chronark]

Grafana IRM reported a "Response Status = 5xx" alert for the api service. Existing mitigations for these incidents did not work as expected. Investigate why the mitigations failed to prevent or resolve this alert and identify any necessary fixes or changes.

[linear]

ENG-2289

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• unkey#4013

Related Issues

• Investigate 5xx response status alerts from API #4013

👤 Suggested Assignees

• chronark

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[coderabbitai]

Coding Plan

Approach

Ticket Summary

Grafana IRM reported a 5xx error alert for the API service, but existing mitigations failed to prevent or resolve the incident. Investigation reveals that critical monitoring and resilience mechanisms are missing or disabled: CloudFlare Workers observability is disabled in production, no 5xx-specific Prometheus alerts exist, circuit breakers are not implemented, and the health check endpoint is deprecated. This plan establishes foundational monitoring, implements key resilience patterns, and adds verification gates to ensure future incidents can be properly detected, mitigated, and resolved.

Observations

Codebase Summary

The API service is built with the Hono framework running on CloudFlare Workers, with the entry point at apps/api/src/worker.ts and organized packages under apps/api/src/pkg/. The service has strong retry mechanisms and structured error handling but lacks circuit breakers and comprehensive monitoring. CloudFlare Workers configuration is managed via apps/api/wrangler.toml. Internal packages (internal/error, internal/logs, internal/worker-logging) provide error handling and logging infrastructure. A parallel Go backend exists with more mature resilience patterns. Alerting currently relies on Checkly health checks rather than metric-based thresholds. CI/CD workflows deploy through staged environments (staging → canary → production) without automated health verification or rollback capabilities.

Assumptions

Assumption 1: Observability Configuration Strategy

Options Considered:

• Enable only CloudFlare native observability (Workers Analytics and Logs)
• Implement custom metrics export to external Prometheus
• Use both CloudFlare native features and custom metrics

Chosen Option: Use both CloudFlare native features and custom metrics

Rationale: Enable CloudFlare Workers observability in wrangler.toml while enhancing custom metrics collection in the application code. This provides both platform-level visibility (through CloudFlare's native tools) and fine-grained application metrics (through custom instrumentation). CloudFlare's observability gives immediate insights into platform-level 5xx errors, while custom metrics allow tracking business logic errors and specific failure modes. This dual approach aligns with the existing structured logging patterns and provides redundancy.

Assumption 2: Circuit Breaker Implementation Location

Options Considered:

• Implement circuit breakers in the TypeScript API service (apps/api)
• Add circuit breaker middleware at the Hono framework level
• Rely on existing Go service circuit breakers and route through those services

Chosen Option: Add circuit breaker middleware at the Hono framework level

Rationale: Implement circuit breakers as Hono middleware in the TypeScript API service. The CloudFlare Workers API is the primary service mentioned in the alert and needs its own resilience mechanisms. Adding circuit breakers at the middleware level follows the existing pattern of middleware-based cross-cutting concerns (rate limiting, authentication, metrics) in apps/api/src/pkg/middleware/. This prevents cascading failures to downstream dependencies (database, cache, external APIs) without requiring architectural changes to route through the Go backend.

Assumption 3: Alert Definition and Routing Strategy

Options Considered:

• Replace Checkly with pure Prometheus alertmanager
• Add Prometheus alerts alongside existing Checkly checks
• Define alerts in code vs. manual Grafana UI configuration

Chosen Option: Add Prometheus alerts alongside existing Checkly checks

Rationale: Add Prometheus alert rules defined as code (configuration files in the repository) while maintaining existing Checkly health checks. Route both Prometheus alerts and Checkly checks to incident.io for unified incident management. Checkly provides synthetic monitoring and endpoint availability checks, while Prometheus alerts catch metric threshold violations (5xx error rates, latency spikes). Defining alerts in code ensures they're version-controlled and deployed consistently across environments. This follows the infrastructure-as-code pattern visible in the existing CloudFlare Workers configuration and aligns with developer preference for declarative, reviewable configurations.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Enable Observability and Metrics Foundation

Establish the monitoring foundation by enabling CloudFlare Workers observability and implementing comprehensive 5xx error metrics collection. This phase ensures future incidents are visible and measurable.

Task 1: Enable CloudFlare Workers Observability

Enable observability in the CloudFlare Workers configuration to capture platform-level logs and analytics.

• Update apps/api/wrangler.toml to set observability = true for production and canary environments (currently disabled globally)
• Enable logpush = true for all environments to ensure logs are exported to centralized logging (currently only enabled in production)
• Configure retention policies if needed based on CloudFlare Workers documentation

Task 2: Enhance 5xx Error Metrics Collection

Augment existing error handling to emit detailed metrics for 5xx responses that can be consumed by Prometheus.

• Update error handlers in apps/api/src/pkg/errors/http.ts to increment counters for 5xx responses with labels for error type, endpoint, and status code
• Add histogram metrics for response latency segmented by status code range (2xx, 4xx, 5xx) to correlate latency spikes with errors
• Extend the error context in internal/error/src/errors/base.ts to capture additional metadata (dependency name, retry count, circuit breaker state) for root cause analysis
• Ensure metrics follow Prometheus naming conventions (e.g., http_requests_total, http_request_duration_seconds, http_errors_total)

Task 3: Create Metrics Export Endpoint

Add a dedicated endpoint to expose Prometheus-formatted metrics from the CloudFlare Worker.

• Create a new route handler in apps/api/src/routes/ for /v1/metrics that exports metrics in Prometheus text format
• Aggregate metrics from the global metrics collector initialized in apps/api/src/pkg/hono/app.ts
• Implement caching of metrics export (5-10 second TTL) to avoid performance impact under high scrape frequency
• Add authentication/authorization to the metrics endpoint to prevent public access (consider CloudFlare Access or API key validation)

Task 4: Define Prometheus Alert Rules

Create alert rule definitions for 5xx error rates and related conditions.

• Create a new directory apps/api/monitoring/ for alert rule definitions
• Define Prometheus alert rules in YAML format covering:
  • High 5xx error rate (warning at 1% of requests, critical at 5% over 5-minute window)
  • Sustained 5xx errors (any 5xx errors for more than 15 minutes)
  • Circuit breaker open state (when implemented)
  • Dependency failure patterns (database connection errors, cache unavailability)
• Include annotations with runbook links and suggested remediation steps
• Add labels for severity, service name, and environment

🤖 Prompt for AI agents

This phase establishes monitoring foundation by enabling CloudFlare Workers
observability and implementing 5xx error metrics collection.

Enable CloudFlare Workers observability:
- Set `observability = true` in `apps/api/wrangler.toml` for production and
canary environments
- Enable `logpush = true` for all environments

Enhance 5xx error metrics:
- Update error handlers in `apps/api/src/pkg/errors/http.ts` to increment 5xx
counters with labels (error type, endpoint, status code)
- Add histogram metrics for response latency by status code range
- Extend error context in `internal/error/src/errors/base.ts` with metadata
(dependency name, retry count, circuit breaker state)
- Follow Prometheus naming conventions

Create metrics export endpoint:
- Add `/v1/metrics` route handler in `apps/api/src/routes/` that exports
Prometheus text format
- Aggregate metrics from global collector in `apps/api/src/pkg/hono/app.ts`
- Implement metrics caching with 5-10 second TTL
- Add authentication/authorization to metrics endpoint

Define Prometheus alert rules:
- Create `apps/api/monitoring/` directory
- Define YAML alert rules for high 5xx error rate, sustained 5xx errors, circuit
breaker state, dependency failures
- Include annotations with runbook links and remediation steps
- Add labels for severity, service name, environment

Phase 2: Implement Resilience Mechanisms

Add circuit breaker protection and modernize health checks to prevent cascading failures and enable automated recovery.

Task 1: Implement Circuit Breaker Middleware

Create a circuit breaker implementation to protect against cascading failures when downstream dependencies fail.

• Create a new circuit breaker utility in apps/api/src/pkg/util/circuit-breaker.ts with configurable failure thresholds, timeout periods, and half-open state logic
• Implement circuit breaker middleware in apps/api/src/pkg/middleware/ that wraps critical dependency calls (database queries, cache operations, external API calls)
• Configure circuit breakers for key dependencies with appropriate thresholds (e.g., 50% error rate over 10 requests, 30-second timeout, 10-second retry)
• Integrate with existing retry logic in apps/api/src/pkg/util/retry.ts to avoid duplicate retry attempts when circuit is open
• Emit metrics when circuit breaker state changes (closed → open, open → half-open, half-open → closed)

Task 2: Modernize Health Check Endpoint

Replace the deprecated health check endpoint with a comprehensive readiness/liveness probe.

• Create a new route handler in apps/api/src/routes/ for /v1/health (separate from deprecated /v1/liveness)
• Implement two probe types:
  • Liveness probe: Basic service responsiveness check (always returns 200 unless worker is completely unresponsive)
  • Readiness probe: Validates critical dependencies (database connectivity, cache availability, Durable Objects reachability)
• Return structured JSON response with dependency status details and overall health status
• Add configurable timeout for dependency checks (fail fast after 2-3 seconds to avoid blocking health checker)
• Update the deprecated /v1/liveness endpoint to redirect or return deprecation warning

Task 3: Add Automatic Recovery Triggers

Implement mechanisms that can automatically attempt recovery when degraded state is detected.

• Update queue handlers in apps/api/src/worker.ts to include circuit breaker state checks before processing messages
• Add fallback logic in critical paths (e.g., apps/api/src/pkg/keys/service.ts) to use stale cached data when circuit is open rather than failing requests
• Implement graceful degradation patterns where non-critical features can be disabled during high 5xx error rates (emit warning logs when degradation is active)
• Add automatic circuit breaker reset triggers based on successful health check responses from dependencies

🤖 Prompt for AI agents

This phase implements circuit breaker protection and modernizes health checks to
prevent cascading failures.

Implement circuit breaker middleware:
- Create circuit breaker utility in `apps/api/src/pkg/util/circuit-breaker.ts`
with configurable thresholds, timeouts, and half-open state logic
- Add circuit breaker middleware in `apps/api/src/pkg/middleware/` wrapping
critical dependency calls
- Configure thresholds for key dependencies (50% error rate over 10 requests,
30-second timeout, 10-second retry)
- Integrate with existing retry logic in `apps/api/src/pkg/util/retry.ts` to
avoid duplicate retries
- Emit metrics on circuit breaker state changes

Modernize health check endpoint:
- Create `/v1/health` route handler in `apps/api/src/routes/`
- Implement liveness probe (basic responsiveness) and readiness probe (validates
dependencies: database, cache, Durable Objects)
- Return structured JSON with dependency status and overall health
- Add configurable timeout for dependency checks (2-3 seconds)
- Update deprecated `/v1/liveness` endpoint with redirect or deprecation warning

Add automatic recovery triggers:
- Update queue handlers in `apps/api/src/worker.ts` to check circuit breaker
state before processing
- Add fallback logic in critical paths (e.g.,
`apps/api/src/pkg/keys/service.ts`) to use stale cached data when circuit is
open
- Implement graceful degradation for non-critical features during high 5xx error
rates
- Add automatic circuit breaker reset based on successful health checks

Phase 3: Deployment Verification and Incident Response

Add automated checks to catch problems during deployments and integrate monitoring with incident management systems.

Task 1: Add Post-Deployment Health Verification

Implement health checks in CI/CD workflows that verify service health after each deployment stage.

• Update .github/workflows/job_deploy_api_staging.yaml to add health check steps that call the new /v1/health endpoint after deployment completes
• Add similar health verification to .github/workflows/job_deploy_api_canary.yaml and production deployment workflow
• Configure health check to validate both liveness and readiness probes, waiting for readiness before considering deployment successful
• Set reasonable timeout (e.g., 60 seconds) with retry logic to allow service to stabilize after deployment
• Fail the deployment workflow if health checks don't pass within the timeout period

Task 2: Implement Smoke Tests for 5xx Detection

Create integration tests that verify critical endpoints are not returning 5xx errors after deployment.

• Update test workflows in .github/workflows/job_test_api_staging.yaml and .github/workflows/job_test_api_canary.yaml to include smoke tests specifically checking for 5xx responses
• Test critical endpoints (key verification, rate limiting, authentication) with realistic payloads
• Assert that response status codes are in expected ranges (2xx for success, 4xx for validation errors, never 5xx)
• Configure tests to run immediately after deployment and again after 5 minutes to catch delayed failures

Task 3: Integrate Prometheus Alerts with incident.io

Configure alert routing from Prometheus to incident.io for automated incident creation when 5xx thresholds are breached.

• Create alert channel configuration in internal/checkly/src/alert-channels.ts or equivalent monitoring configuration to route Prometheus alerts to incident.io
• Map alert severity levels to incident.io severity (warning → low, critical → high)
• Configure alert grouping to avoid creating multiple incidents for the same root cause
• Add context to incident payload including affected service, environment, alert query, current error rate, and runbook link
• Test integration with a low-threshold test alert before enabling production alerts

🤖 Prompt for AI agents

This phase adds deployment verification checks and integrates monitoring with
incident management.

Add post-deployment health verification:
- Update `.github/workflows/job_deploy_api_staging.yaml` with health check steps
calling `/v1/health` after deployment
- Add health verification to `.github/workflows/job_deploy_api_canary.yaml` and
production deployment workflow
- Configure health checks to validate liveness and readiness probes, wait for
readiness before success
- Set 60-second timeout with retry logic for service stabilization
- Fail deployment workflow if health checks don't pass

Implement smoke tests for 5xx detection:
- Update `.github/workflows/job_test_api_staging.yaml` and
`.github/workflows/job_test_api_canary.yaml` with 5xx smoke tests
- Test critical endpoints (key verification, rate limiting, authentication) with
realistic payloads
- Assert response codes are in expected ranges (2xx, 4xx, never 5xx)
- Run tests immediately after deployment and again after 5 minutes

Integrate Prometheus alerts with incident.io:
- Create alert channel configuration in `internal/checkly/src/alert-channels.ts`
routing Prometheus alerts to incident.io
- Map severity levels (warning → low, critical → high)
- Configure alert grouping to avoid duplicate incidents
- Add incident payload context (service, environment, alert query, error rate,
runbook link)
- Test with low-threshold test alert before production enable

Phase 4: Documentation and Operational Runbooks

Document the new mitigations and create operational procedures for responding to 5xx incidents.

Task 1: Create 5xx Incident Response Runbook

Develop comprehensive runbook for operators responding to 5xx error alerts.

• Create a new runbook in apps/engineering/content/docs/runbooks/services/api/ specifically for "5xx Error Response" incidents
• Document diagnostic steps: check Grafana dashboards for error rate trends, review CloudFlare Workers logs for error patterns, verify circuit breaker states, check dependency health
• Include remediation procedures: manual circuit breaker reset process, cache invalidation commands, deployment rollback instructions, scaling considerations
• Link to relevant Grafana dashboards, log queries, and metrics endpoints
• Reference the existing incident.io escalation process mentioned in other runbooks

Task 2: Document New Monitoring Architecture

Create documentation explaining the monitoring and resilience mechanisms added in this plan.

• Add architecture documentation in apps/engineering/content/docs/ describing the monitoring stack: CloudFlare Workers observability, custom metrics collection, Prometheus alerting, Grafana visualization
• Document circuit breaker configuration and behavior: thresholds, timeout periods, fallback strategies
• Explain the relationship between Checkly synthetic checks, Prometheus metric-based alerts, and incident.io incident creation
• Include diagrams showing alert flow from metrics → Prometheus → incident.io → on-call engineer
• Document the deployment verification process and when rollbacks should be triggered

Task 3: Update Development Guidelines

Add guidelines for developers to maintain and extend the resilience patterns.

• Update development documentation to include guidelines for adding metrics to new endpoints (where to emit metrics, naming conventions, required labels)
• Document circuit breaker usage patterns and when to apply circuit breaker protection to new dependencies
• Provide examples of proper error handling that integrates with metrics collection and circuit breakers
• Add testing guidelines for verifying circuit breaker behavior and metrics emission in integration tests
• Create checklists for pre-deployment verification that developers should complete before merging changes affecting critical paths

🤖 Prompt for AI agents

This phase documents the new mitigations and creates operational procedures for
5xx incidents.

Create 5xx incident response runbook:
- Create runbook in `apps/engineering/content/docs/runbooks/services/api/` for
"5xx Error Response" incidents
- Document diagnostic steps (check Grafana dashboards, review CloudFlare Workers
logs, verify circuit breaker states, check dependency health)
- Include remediation procedures (circuit breaker reset, cache invalidation,
deployment rollback, scaling)
- Link to Grafana dashboards, log queries, metrics endpoints
- Reference incident.io escalation process

Document new monitoring architecture:
- Add architecture documentation in `apps/engineering/content/docs/` describing
monitoring stack (CloudFlare Workers observability, custom metrics, Prometheus
alerting, Grafana)
- Document circuit breaker configuration (thresholds, timeouts, fallback
strategies)
- Explain relationship between Checkly checks, Prometheus alerts, and
incident.io
- Include alert flow diagrams (metrics → Prometheus → incident.io → on-call
engineer)
- Document deployment verification and rollback triggers

Update development guidelines:
- Add guidelines for adding metrics to new endpoints (emission points, naming
conventions, labels)
- Document circuit breaker usage patterns and when to apply protection
- Provide error handling examples integrating with metrics and circuit breakers
- Add testing guidelines for circuit breaker behavior and metrics emission
- Create pre-deployment verification checklists for critical path changes

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

This phase establishes monitoring foundation by enabling CloudFlare Workers
observability and implementing 5xx error metrics collection.

Enable CloudFlare Workers observability:
- Set `observability = true` in `apps/api/wrangler.toml` for production and
canary environments
- Enable `logpush = true` for all environments

Enhance 5xx error metrics:
- Update error handlers in `apps/api/src/pkg/errors/http.ts` to increment 5xx
counters with labels (error type, endpoint, status code)
- Add histogram metrics for response latency by status code range
- Extend error context in `internal/error/src/errors/base.ts` with metadata
(dependency name, retry count, circuit breaker state)
- Follow Prometheus naming conventions

Create metrics export endpoint:
- Add `/v1/metrics` route handler in `apps/api/src/routes/` that exports
Prometheus text format
- Aggregate metrics from global collector in `apps/api/src/pkg/hono/app.ts`
- Implement metrics caching with 5-10 second TTL
- Add authentication/authorization to metrics endpoint

Define Prometheus alert rules:
- Create `apps/api/monitoring/` directory
- Define YAML alert rules for high 5xx error rate, sustained 5xx errors, circuit
breaker state, dependency failures
- Include annotations with runbook links and remediation steps
- Add labels for severity, service name, environment
===============================================================================

Task: 2

This phase implements circuit breaker protection and modernizes health checks to
prevent cascading failures.

Implement circuit breaker middleware:
- Create circuit breaker utility in `apps/api/src/pkg/util/circuit-breaker.ts`
with configurable thresholds, timeouts, and half-open state logic
- Add circuit breaker middleware in `apps/api/src/pkg/middleware/` wrapping
critical dependency calls
- Configure thresholds for key dependencies (50% error rate over 10 requests,
30-second timeout, 10-second retry)
- Integrate with existing retry logic in `apps/api/src/pkg/util/retry.ts` to
avoid duplicate retries
- Emit metrics on circuit breaker state changes

Modernize health check endpoint:
- Create `/v1/health` route handler in `apps/api/src/routes/`
- Implement liveness probe (basic responsiveness) and readiness probe (validates
dependencies: database, cache, Durable Objects)
- Return structured JSON with dependency status and overall health
- Add configurable timeout for dependency checks (2-3 seconds)
- Update deprecated `/v1/liveness` endpoint with redirect or deprecation warning

Add automatic recovery triggers:
- Update queue handlers in `apps/api/src/worker.ts` to check circuit breaker
state before processing
- Add fallback logic in critical paths (e.g.,
`apps/api/src/pkg/keys/service.ts`) to use stale cached data when circuit is
open
- Implement graceful degradation for non-critical features during high 5xx error
rates
- Add automatic circuit breaker reset based on successful health checks
===============================================================================

Task: 3

This phase adds deployment verification checks and integrates monitoring with
incident management.

Add post-deployment health verification:
- Update `.github/workflows/job_deploy_api_staging.yaml` with health check steps
calling `/v1/health` after deployment
- Add health verification to `.github/workflows/job_deploy_api_canary.yaml` and
production deployment workflow
- Configure health checks to validate liveness and readiness probes, wait for
readiness before success
- Set 60-second timeout with retry logic for service stabilization
- Fail deployment workflow if health checks don't pass

Implement smoke tests for 5xx detection:
- Update `.github/workflows/job_test_api_staging.yaml` and
`.github/workflows/job_test_api_canary.yaml` with 5xx smoke tests
- Test critical endpoints (key verification, rate limiting, authentication) with
realistic payloads
- Assert response codes are in expected ranges (2xx, 4xx, never 5xx)
- Run tests immediately after deployment and again after 5 minutes

Integrate Prometheus alerts with incident.io:
- Create alert channel configuration in `internal/checkly/src/alert-channels.ts`
routing Prometheus alerts to incident.io
- Map severity levels (warning → low, critical → high)
- Configure alert grouping to avoid duplicate incidents
- Add incident payload context (service, environment, alert query, error rate,
runbook link)
- Test with low-threshold test alert before production enable
===============================================================================

Task: 4

This phase documents the new mitigations and creates operational procedures for
5xx incidents.

Create 5xx incident response runbook:
- Create runbook in `apps/engineering/content/docs/runbooks/services/api/` for
"5xx Error Response" incidents
- Document diagnostic steps (check Grafana dashboards, review CloudFlare Workers
logs, verify circuit breaker states, check dependency health)
- Include remediation procedures (circuit breaker reset, cache invalidation,
deployment rollback, scaling)
- Link to Grafana dashboards, log queries, metrics endpoints
- Reference incident.io escalation process

Document new monitoring architecture:
- Add architecture documentation in `apps/engineering/content/docs/` describing
monitoring stack (CloudFlare Workers observability, custom metrics, Prometheus
alerting, Grafana)
- Document circuit breaker configuration (thresholds, timeouts, fallback
strategies)
- Explain relationship between Checkly checks, Prometheus alerts, and
incident.io
- Include alert flow diagrams (metrics → Prometheus → incident.io → on-call
engineer)
- Document deployment verification and rollback triggers

Update development guidelines:
- Add guidelines for adding metrics to new endpoints (emission points, naming
conventions, labels)
- Document circuit breaker usage patterns and when to apply protection
- Provide error handling examples integrating with metrics and circuit breakers
- Add testing guidelines for circuit breaker behavior and metrics emission
- Create pre-deployment verification checklists for critical path changes

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord or schedule a call!