Phase B: Add latency histograms and per-operation error counters

[geoffjay]

Goal

The HTTP middleware in crates/common/src/server.rs captures request-level latency for every route, but it doesn't measure the actual work — the embedding API call, the LanceDB search, the workflow dispatch. When those slow down, p99 HTTP latency tells us something is wrong but not which operation.

Add operation-level histograms and error counters so we can answer "what's slow?" and "what's failing?" precisely.

Tasks

Histograms

Service	Metric	Location
orchestrator	workflow_dispatch_duration_seconds	crates/orchestrator/src/scheduler/runner.rs (around the dispatch call)
memory	memory_search_duration_seconds	crates/memory/src/api.rs (search handler)
memory	memory_embedding_duration_seconds	crates/memory/src/ (wherever embeddings are computed)
index	index_query_duration_seconds	crates/index/src/api.rs (search handler)
index	index_embedding_duration_seconds	embedding call site
notify	notification_delivery_age_seconds	histogram of (delivered_at - created_at) — crates/notify/src/api.rs

Pattern:

let start = std::time::Instant::now();
let result = do_work().await;
metrics::histogram!("memory_search_duration_seconds").record(start.elapsed().as_secs_f64());

Per-operation error counters

The HTTP middleware already counts 5xx via http_requests_total{status="5xx"}, but that's coarse. For each operation that can fail in a recoverable way (embedding API timeout, LanceDB error, workflow dispatch error), emit a labeled counter:

• memory_errors_total{operation, kind} — operation = search|embed|store, kind = timeout|api_error|db_error
• index_errors_total{operation, kind}
• workflow_dispatch_errors_total{reason} (reason = template_render | agent_unavailable | etc.)
• embedding_errors_total{service, provider} (cross-service — both memory and index call embeddings)

Files Touched

• crates/orchestrator/src/scheduler/runner.rs
• crates/memory/src/api.rs + embedding caller
• crates/index/src/api.rs + embedding caller
• crates/notify/src/api.rs

Verification

1. cargo build clean
2. Exercise each operation; confirm via /metrics that the histogram has populated buckets
3. Force an error (e.g. point embedding service at a bad URL); confirm the error counter increments
4. Dashboard query histogram_quantile(0.99, sum(rate(memory_search_duration_seconds_bucket[5m])) by (le)) returns a real value

Depends On

Phase A — index and ask need their /metrics endpoints functional first (index already has one; verify before starting)

[geoffjay]

✅ Triage Complete

Field	Value
Type	enhancement
Complexity	large
Milestone	v0.11.0 - Experiment Tracking & Observability
Labels	enhancement, complexity:large

Summary: Phase B of the Prometheus instrumentation plan. Adds operation-level latency histograms and per-operation error counters across four service crates — replacing the coarse "something is slow" signal from HTTP middleware with precise "which operation is slow/failing" observability.

Scope breakdown:

• crates/orchestrator/src/scheduler/runner.rs — workflow_dispatch_duration_seconds histogram + workflow_dispatch_errors_total{reason} counter
• crates/memory/src/api.rs + embedding caller — memory_search_duration_seconds, memory_embedding_duration_seconds histograms + memory_errors_total{operation, kind} counter
• crates/index/src/api.rs + embedding caller — index_query_duration_seconds, index_embedding_duration_seconds histograms + index_errors_total{operation, kind} counter
• crates/notify/src/api.rs — notification_delivery_age_seconds histogram
• Cross-service: embedding_errors_total{service, provider} at embedding call sites in both memory and index

Why complexity:large: Touches 4 separate service crates, 6 histogram additions, 4 error counter types, and requires locating embedding call sites within memory and index that aren't explicitly identified. Total instrumentation across 6-8 files in 4 crates is well over 200 lines.

Milestone rationale: v0.11.0 (Experiment Tracking & Observability) — same series as Phase C-G, all under plan #1240.

Dependency note: Blocked on Phase A (#1241) — index and ask need their /metrics endpoints functional before this work is fully verifiable. The orchestrator, memory, and notify portions can proceed independently.

No clarifying questions needed — metric names, locations, and the timing pattern are all provided.