Phase C: Wire experiment_* metrics to populate experiment-tracking dashboard

[geoffjay]

Goal

The Grafana dashboard infra/grafana/dashboards/experiment-tracking.json references three metrics that no code emits:

• experiment_dispatches_total{experiment, status}
• experiment_cost_usd_total{experiment}
• experiment_dispatch_duration_seconds_bucket{experiment}

Result: the dashboard is permanently empty. User has confirmed experiments is a planned feature. Wire the metrics so the dashboard works.

Tasks

1. Locate the experiment dispatch path

Experiments are not yet a first-class concept in the codebase. Likely entry points to check:

• crates/orchestrator/src/scheduler/ — experiments may live alongside workflows
• Workflow metadata — check whether WorkflowResponse has an experiment field
• Any prompt template variables that label dispatches with an experiment ID

If experiments don't exist yet as a concept: this issue may need to be re-scoped as "add experiment tracking" rather than just emitting the metrics.

2. Emit the three metrics

// On dispatch completion
metrics::counter!("experiment_dispatches_total",
    "experiment" => experiment_id,
    "status" => status_label, // "dispatched" | "failed"
).increment(1);

metrics::counter!("experiment_cost_usd_total",
    "experiment" => experiment_id,
).increment_by(cost_usd);

metrics::histogram!("experiment_dispatch_duration_seconds",
    "experiment" => experiment_id,
).record(duration_secs);

Cardinality note: experiment label is bounded by the number of distinct experiments — acceptable if experiments are limited (≤ 100s). If experiments can grow unbounded, consider hashing/bucketing.

3. Validate the dashboard

Open experiment-tracking.json in Grafana with Prometheus connected and confirm all panels render data.

Files Touched

• crates/orchestrator/src/scheduler/runner.rs (or wherever dispatch happens)
• Possibly crates/orchestrator/src/types.rs if experiment becomes a workflow field

Verification

1. Run an experiment dispatch (mechanism TBD based on discovery in task 1)
2. curl localhost:17006/metrics | grep experiment_ shows the three metrics
3. Grafana panels in experiment-tracking.json show non-empty time series

[geoffjay]

✅ Triage Complete

Field	Value
Type	enhancement
Complexity	medium
Milestone	v0.11.0 - Experiment Tracking & Observability
Labels	enhancement, needs-refinement, complexity:medium

Summary: Phase C of the Prometheus instrumentation plan. The experiment-tracking Grafana dashboard references three metrics that no code currently emits. The goal is to wire those metrics so the dashboard works.

Why needs-refinement: The issue itself flags a critical precondition in Task 1 — experiments may not exist as a first-class concept in the codebase yet. If that's the case, the scope changes from "emit 3 metric calls" to "implement experiment tracking and emit metrics," which is a much larger undertaking. The issue should be re-evaluated after Task 1's discovery step:

• If experiments exist (e.g., as workflow metadata or a scheduler concept): complexity drops to small — emit 3 metric calls in the dispatch path (~15 lines)
• If experiments don't exist yet: this issue should be split or re-scoped, with the experiment entity implementation tracked separately (likely the bulk of v0.11.0 scope)

Milestone rationale: v0.11.0 (Experiment Tracking & Observability) explicitly includes experiment tracking entities and Prometheus metric labeling — this is exactly where the concept should be built.

Complexity set to medium as a conservative estimate pending Task 1's findings. Adjust to small or large after discovery.