Plan: Improve Prometheus instrumentation across services

[geoffjay]

Context

Audit of Prometheus instrumentation across the 11 agentd service crates revealed strong foundations (shared HTTP middleware, per-service /metrics endpoints, well-instrumented orchestrator and monitor) but significant gaps elsewhere — most notably, two services with no metrics at all (hook, mcp), four services emitting only service_info (ask, wrap, core, index), an unbounded label cardinality risk in communicate, no alerting rules, and a Grafana dashboard referencing metrics that aren't emitted.

This roadmap closes those gaps in shippable phases. Each sub-issue is independently executable and can be assigned to a separate PR.

Current Inventory

Service	Custom Metrics	Status
orchestrator	16+ counters/gauges across agents, workflows, approvals, messages, cost	Strong
monitor	CPU, memory, disk, load, collections	Strong
notify	created/responded/dismissed/deleted/pending	Good
communicate	rooms, participants, messages	Cardinality risk (room_id label)
memory	searched/deleted/created	Thin — no latency
ask	only service_info	No business metrics
wrap	only service_info	No business metrics
core	only service_info	No business metrics
index	only service_info	No business metrics
hook	nothing	Not scraped at all
mcp	nothing	Not scraped at all

Roadmap

Each phase is a sub-issue and is independently shippable. Listed in priority order.

• Phase A — Fill critical service gaps (hook, mcp, ask, wrap, core, index)
• Phase B — Latency histograms + per-operation error counters
• Phase C — Wire the experiment dashboard (experiment_* metrics)
• Phase D — Fix cardinality risks (room_id labels in communicate)
• Phase E — Metric descriptions (describe_*! calls for # HELP text)
• Phase F — Alerting rules (infra/prometheus/alert_rules.yml)
• Phase G — SLO recording rules

Shared Infrastructure (already in place)

• crates/common/src/server.rs::metrics_layer() — Tower middleware emitting http_requests_total{method,path,status} and http_request_duration_seconds{method,path} for every request, with UUID path normalization
• metrics-exporter-prometheus + PrometheusBuilder::new().install_recorder() pattern in each service's main.rs
• Scrape config at infra/prometheus/prometheus.yml
• Grafana dashboards under infra/grafana/dashboards/

Success Criteria

• All 11 service crates expose /metrics and are scraped
• Every service emits at least: service_info, business counters for create/delete/process operations, and gauges for active counts
• Every histogram-worthy operation (DB query, embedding call, dispatch, search) has a *_duration_seconds histogram
• No unbounded label cardinality
• /metrics output has # HELP lines for every metric
• alert_rules.yml covers service-down, error-rate, latency, queue-depth, dispatch-failure baselines
• All four existing dashboards render real data (currently experiment-tracking.json is empty)