Phase G: Add SLO recording rules

[geoffjay]

Goal

Dashboards and alerts currently compute percentiles and error rates on the fly. At scale this is expensive and slow. Pre-aggregate via Prometheus recording rules so dashboards just read the pre-computed series, and burn-rate alerts (multi-window SLO alerting) become straightforward.

Tasks

1. Create infra/prometheus/recording_rules.yml

Standard SRE SLI/SLO pattern. Naming convention: <scope>:<metric>:<aggregation>.

groups:
  - name: agentd_sli
    interval: 30s
    rules:
      # Per-service availability SLI: success ratio over 5m
      - record: agentd:http_request_success:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status!~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))

      # Per-service availability SLI: 1h
      - record: agentd:http_request_success:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{status!~"5.."}[1h]))
            / sum by (job) (rate(http_requests_total[1h]))

      # Per-service latency SLI (p99 over 5m)
      - record: agentd:http_request_duration:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (le, job) (rate(http_request_duration_seconds_bucket[5m]))
          )

      # Workflow dispatch success ratio
      - record: agentd:workflow_dispatch_success:ratio_rate5m
        expr: |
          sum(rate(workflow_dispatches_total{status="dispatched"}[5m]))
            / sum(rate(workflow_dispatches_total[5m]))

2. Add to prometheus.yml

Append to the rule_files block:

rule_files:
  - "alert_rules.yml"
  - "recording_rules.yml"

3. Add burn-rate alerts (optional, advanced)

If SLO targets are defined (e.g. 99.9% availability), add multi-window burn-rate alerts using the recorded ratios. See SRE Workbook ch. 5 for the pattern.

4. Migrate dashboard queries

Update infra/grafana/dashboards/service-overview.json (and others) to use the recorded series instead of computing on the fly. E.g.:

- histogram_quantile(0.99, sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))
+ agentd:http_request_duration:p99_5m

Files Touched

• infra/prometheus/recording_rules.yml (new)
• infra/prometheus/prometheus.yml
• infra/grafana/dashboards/*.json (queries updated to use recorded series)

Verification

1. promtool check rules infra/prometheus/recording_rules.yml passes
2. After restart, localhost:9090/api/v1/query?query=agentd:http_request_success:ratio_rate5m returns data
3. Dashboards still render correctly with the migrated queries
4. Dashboard load time is noticeably faster (especially for the percentile panels)

Depends On

Phase F — the recording_rules.yml is read alongside alert_rules.yml, so the rule_files config block in prometheus.yml needs Phase F's changes first

[geoffjay]

✅ Triage Complete

Field	Value
Type	enhancement
Complexity	medium
Milestone	v0.11.0 - Experiment Tracking & Observability
Labels	enhancement, complexity:medium

Summary: This is a well-specified infra/observability task — the final phase (G) of the Prometheus instrumentation plan. It adds Prometheus recording rules to pre-aggregate SLI/SLO metrics that dashboards and burn-rate alerts currently compute on the fly.

Scope breakdown:

• infra/prometheus/recording_rules.yml — new file, ~40 lines of YAML (fully provided in issue body)
• infra/prometheus/prometheus.yml — 1-line addition to rule_files
• infra/grafana/dashboards/*.json — query migration across one or more dashboard files (extent depends on how many dashboards currently use the raw histogram/rate expressions)
• Task 3 (burn-rate alerts) is explicitly optional/advanced and can be a follow-on

Milestone rationale: Assigned to v0.11.0 (Experiment Tracking & Observability), which explicitly includes Prometheus metric labeling and the local Prometheus/Grafana observability stack. All sibling Phase A-F issues belong to the same plan (#1240) and should be coordinated under this milestone.

Dependency note: Blocked on Phase F (#1246) — prometheus.yml must already have a rule_files block before this issue's changes can be applied.

No clarifying questions needed — acceptance criteria are explicit and verifiable (promtool check rules pass, query endpoint returns data, dashboards render correctly).