Benchmark v1.6 vs v1.7 orchestration cost and latency

[jrpbuilds]

Parent

Part of #46.

Relationships

Depends on #63 and #64.

Goal

Quantify whether the v1.7 architecture actually improves SpecOps efficiency and reliability rather than relying on prompt-size intuition alone.

Use representative scenarios from #64 to compare a v1.6 baseline with the final v1.7 implementation.

Metrics

Capture, where the provider/runtime exposes them:

• total input tokens;
• Coordinator input tokens;
• total output tokens;
• reasoning tokens where available;
• number of model calls;
• number of Coordinator activations;
• number of tool calls;
• Todo/progress-specific model/tool calls removed by v1.7;
• wall-clock run duration;
• time spent in Coordinator calls if separable;
• malformed handoffs/retries;
• tool misuse/invalid action attempts;
• remediation iterations;
• final workflow success/failure.

Also capture static/expanded prompt size so token/runtime changes can be correlated with the architectural changes.

Model matrix

At minimum compare:

• one lower-cost/fast Coordinator model; and
• one stronger Coordinator model.

A fast model such as the currently considered DeepSeek Flash class is particularly useful because v1.7 is intended to reduce the amount of SpecOps protocol reasoning expected from a cheap orchestrator.

Keep specialist model choices stable across paired runs wherever possible so the comparison isolates Coordinator/workflow architecture rather than unrelated model changes.

Method

• Use identical task/scenario inputs between versions.
• Record model/provider/version/configuration with each result.
• Run enough repetitions to distinguish obvious architecture wins from one-off provider latency noise.
• Report medians/ranges rather than overclaiming precision from a tiny sample.
• Note where v1.6/v1.7 behaviour differs materially and makes a direct metric comparison invalid.

Success criteria

There is no mandatory wall-clock percentage target, but v1.7 should demonstrate a clear reduction in prompt/input overhead without reducing workflow success.

Desired evidence would look like:

Coordinator context: materially lower
Total input tokens: lower
Mechanical tool/model churn: lower
Median runtime: lower or neutral
Workflow success: unchanged or better

If a metric regresses, document why before release rather than hiding it behind prompt-size gains.

Acceptance criteria

• A reproducible benchmark harness/report compares v1.6 and v1.7 on representative scenarios.
• Token, call-count, latency and success metrics are captured where available.
• Both a cheap/fast and stronger Coordinator configuration are exercised.
• Results include enough repetitions/context to avoid presenting provider jitter as architectural fact.
• The report calls out regressions as well as improvements.
• Benchmark results can be summarized in the v1.7 release notes/documentation.

Out of scope

• Guaranteeing provider-independent latency percentages.
• Optimizing specialist model mappings as part of this issue.