Phase F: Add Prometheus alerting rules (alert_rules.yml)

[geoffjay]

Goal

infra/prometheus/prometheus.yml has the alerting and rule_files blocks commented out, and no alert_rules.yml exists. We have visibility but no notification — problems are only caught when someone happens to look at a dashboard.

Add a baseline set of alert rules covering availability, errors, latency, and resource exhaustion.

Tasks

1. Create infra/prometheus/alert_rules.yml

Baseline alerts (start here; tune thresholds based on observed behavior):

groups:
  - name: agentd_availability
    interval: 30s
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} is down"
          description: "Service {{ $labels.service }} has been unreachable for 1m"

  - name: agentd_errors
    interval: 30s
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} HTTP 5xx rate > 5%"

      - alert: WorkflowDispatchFailures
        expr: |
          sum(rate(workflow_dispatches_total{status="failed"}[5m]))
            / sum(rate(workflow_dispatches_total[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Workflow dispatch failure rate > 10%"

  - name: agentd_latency
    interval: 30s
    rules:
      - alert: HighP99Latency
        expr: |
          histogram_quantile(0.99,
            sum by (le, job) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} p99 HTTP latency > 1s"

  - name: agentd_queues
    interval: 30s
    rules:
      - alert: MessageQueueBacklog
        expr: messages_queued > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Message queue backlog > 100 for 5m"

      - alert: ApprovalsBacklog
        expr: approvals_pending > 20
        for: 10m
        labels:
          severity: info
        annotations:
          summary: "Pending approvals > 20 for 10m"

  - name: agentd_resources
    interval: 30s
    rules:
      - alert: HighCpuUsage
        expr: system_cpu_usage_percent > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Host CPU > 90% for 5m"

      - alert: HighMemoryUsage
        expr: (system_memory_used_bytes / system_memory_total_bytes) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Host memory > 90% used for 5m"

      - alert: DiskAlmostFull
        expr: system_disk_usage_percent > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk {{ $labels.mountpoint }} > 85% full"

2. Uncomment prometheus.yml blocks

rule_files:
  - "alert_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

Note: this assumes Alertmanager will run at localhost:9093. If Alertmanager isn't set up yet, the rules still evaluate and appear in /alerts on the Prometheus UI; they just won't notify.

3. (Optional) Document Alertmanager setup

Add a short section to infra/README.md (or create one) explaining how to run Alertmanager locally.

Files Touched

• infra/prometheus/alert_rules.yml (new)
• infra/prometheus/prometheus.yml
• infra/README.md (optional)

Verification

1. promtool check rules infra/prometheus/alert_rules.yml — passes
2. Restart Prometheus; localhost:9090/rules lists all groups
3. Stop a service; within 1m localhost:9090/alerts shows ServiceDown firing
4. Restart the service; alert clears

[geoffjay]

✅ Triage Complete

Field	Value
Type	enhancement
Complexity	medium
Milestone	v0.11.0 - Experiment Tracking & Observability
Labels	enhancement, complexity:medium

Summary: This is a well-specified infra/observability task — Phase F of the Prometheus instrumentation plan. It establishes the alerting layer by creating alert_rules.yml with a baseline set of SRE-style alerts (availability, error rate, latency, queue depth, resource exhaustion) and uncommenting the rule_files / alerting blocks in prometheus.yml that have been commented out.

Scope breakdown:

• infra/prometheus/alert_rules.yml — new file, ~80 lines of YAML (full content provided in issue body across 5 alert groups)
• infra/prometheus/prometheus.yml — uncomment 6-7 lines (rule_files + alerting blocks)
• infra/README.md — optional short Alertmanager setup section

Milestone rationale: Assigned to v0.11.0 (Experiment Tracking & Observability), which explicitly includes Prometheus metric labeling and the local Prometheus/Grafana observability stack. This is the direct predecessor to Phase G (#1247) which depends on this issue's rule_files config being in place.

Threshold note: Alert thresholds (5% error rate, 1s p99, 90% CPU, etc.) are marked as baselines to tune after observing real behavior — this is intentional and not a blocker.

No clarifying questions needed — full YAML content, affected files, and promtool-based verification steps are all provided.