Skip to content

Commit 93dc6bc

Browse files
committed
Merge origin/main into feat/voice-dictation-v2
Resolve keybindings.go import conflict: keep both fmt (from main, used by conflict-warning Sprintf calls) and strconv (from this branch, used by function-key label Itoa).
2 parents 85c263e + 008bc9b commit 93dc6bc

36 files changed

Lines changed: 2450 additions & 29 deletions
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

‎.config/go/telemetry/local/upload.token‎

Whitespace-only changes.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2

‎.zero/specialists/storyteller.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
name: "storyteller"
3+
description: "Storyteller"
4+
tools:
5+
- "read-only"
6+
- "plan"
7+
---
8+
9+
You are a Storyteller. Create imaginative and engaging stories for the user.

‎docs/MODEL_ROUTING_RESEARCH.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Multi-Model Routing, Refinement & Panel Deliberation — Design
2+
3+
**Status:** Design for a future release, scope confirmed (build all three tracks
4+
plus a combined `auto` mode). No implementation code written yet.
5+
**Date:** 2026-07-05
6+
7+
## 1. The ask
8+
9+
Route different parts of an agent session to different models based on which
10+
model is good at that kind of work, for two payoffs at once: lower cost and
11+
higher per-task quality. Two products were named as inspiration —
12+
OpenRouter's "Fusion" and Sakana AI's multi-model work — plus the
13+
cost-focused academic pattern (RouteLLM/FrugalGPT). Decision: build all
14+
three, composed into one adaptive `auto` mode rather than three disconnected
15+
toggles.
16+
17+
## 2. What the named prior art actually does
18+
19+
### OpenRouter Fusion — deliberation, not routing
20+
`openrouter/fusion` sends one prompt to a **panel** of models in parallel,
21+
then a **judge** model reads every panel response, identifies consensus,
22+
contradictions, and blind spots, and produces one synthesized answer.
23+
OpenRouter's own docs scope it to "scenarios where accuracy and thoroughness
24+
are critical and the cost of being wrong outweighs the higher latency and
25+
cost," and call it "overkill for simple, tactical prompts." Billing is the
26+
**sum** of every panel completion plus the judge — strictly more expensive
27+
than one model call. Quality-maximizing, not cost-saving.
28+
29+
### Sakana AI — Multi-LLM AB-MCTS / "Fugu"
30+
Treats several frontier models as a collaborative pool and runs Monte Carlo
31+
Tree Search with adaptive branching: at each node, go "wider" (try a
32+
different model for a fresh candidate) or "deeper" (refine the current
33+
candidate using external feedback, e.g. execute code and check the result),
34+
balanced via Thompson sampling. Explicitly an inference-time-scaling
35+
technique — spends more compute per query for a better answer on hard,
36+
scorable benchmarks (SWE-bench-class). Productized as "Sakana Fugu," a
37+
single endpoint that conducts/routes/verifies/synthesizes internally.
38+
[reported per sakana.ai and third-party coverage, not independently
39+
re-verified against a primary paper]
40+
41+
Full MCTS assumes many rollouts scored against a numeric reward — that setup
42+
fits competitive-programming/math-style tasks with a clean pass/fail signal.
43+
Most day-to-day dev tasks either have a *weaker* verifiable signal (tests,
44+
build, lint — binary-ish, not a smooth reward) or *no* verifiable signal at
45+
all (design judgment calls). The design below adapts the wider/deeper
46+
**idea** without building literal MCTS.
47+
48+
### RouteLLM / FrugalGPT — the actual cost-saving pattern
49+
RouteLLM trains a classifier on preference data to predict whether a "weak"
50+
model matches a "strong" model on a given query and routes accordingly.
51+
FrugalGPT cascades: cheap model first, a judge/verifier checks the answer,
52+
escalate to a stronger model only on failure. Both are reported to cut cost
53+
40–85% while retaining roughly 95% of frontier-model quality.
54+
55+
Both of Sakana/OpenRouter's patterns spend *more* for a *better* answer;
56+
RouteLLM/FrugalGPT spend *less* by not over-using frontier models. They are
57+
opposite levers — the design below keeps them as separate mechanisms and
58+
lets a third policy (`auto`) decide when to use which, rather than merging
59+
them into one setting.
60+
61+
## 3. Zero's existing substrate (grounds every "reuse" claim below)
62+
63+
| Piece | File | Relevance |
64+
|---|---|---|
65+
| Per-specialist model override, wired end-to-end | `internal/specialist/manifest.go:30` (`Metadata.Model`), applied in `internal/specialist/exec.go:693-705` (`appendModelArgs`) | Direct substrate for **Route**. Empty inherits parent; set value is passed to the spawned subprocess today. |
66+
| Per-swarm-member model override | `internal/swarm/definition.go:33`, resolved in `internal/swarm/team.go:209-214` | Same override pattern, swarm side. |
67+
| Coarse manual mode presets | `internal/modelregistry/modes.go:30-63` | `smart`/`deep`/`fast`/`large`/`precise`. **`smart` stays as-is — do not repurpose or rename.** |
68+
| Tiered, cache-aware cost computation | `internal/modelregistry/cost.go` | Feeds `zero usage`; reused as-is for Route's savings telemetry. |
69+
| Live pricing/context-limit catalog | `internal/modelregistry/modelsdev.go` | Keeps routing cost data fresh without hardcoding. |
70+
| Model-family prompt tuning | `internal/agent/system_prompt_models.go` | Precedent for "different models get different treatment," at the prompt layer. |
71+
| **Post-edit verify + corrective feedback loop, already shipped** | `internal/agent/selfcorrect.go` (`SelfCorrector.AfterEdit`, `Outcome` enum: `Disabled`/`Passed`/`Correcting`/`Reported`/`Aborted`) | This **is** the "deeper" half of Refine. Runs LSP diagnostics + `verify.Run`, synthesizes feedback, retries same-model, bounded at `defaultSelfCorrectMaxAttempts = 3`. |
72+
| **The exact gap**, precisely located | `internal/agent/loop.go:602`: `if feedback, _ := options.SelfCorrect.AfterEdit(ctx, ...); feedback != "" { … }` | The `Outcome` return value is discarded. `OutcomeAborted` (still failing after budget) currently just... ends up wherever the run's normal stop condition takes it. No escalation policy reads this signal today. **This is the hook point for "wider."** |
73+
| No fan-one-task-to-N-models primitive | Checked `internal/swarm/coordinator.go` | Confirmed absent. Swarm decomposes into distinct per-member sub-tasks; it does not broadcast one identical task to N members. **Panel needs new dispatch code**, though it can reuse `exec.go`'s subprocess-launch-with-model-override underneath. |
74+
75+
## 4. Design: three mechanisms plus one policy that composes them
76+
77+
### Naming
78+
`modelregistry.Mode`'s existing `"smart"` preset (fixed `claude-sonnet-4.5`,
79+
balanced daily driver) is unchanged — renaming or repurposing it would
80+
silently change behavior for anyone currently running `/mode smart`. The new
81+
adaptive/combined mode is named **`auto`**: it more precisely says "the
82+
system is choosing for you" than "smart" would, and it doesn't collide.
83+
84+
### 4.1 Route (cost-saving — RouteLLM/FrugalGPT pattern)
85+
86+
- New `modelRouting.roles` config block (user or project config): role name
87+
→ model id/alias, e.g. `{"roles": {"explorer": "haiku-4.5", "code-review":
88+
"gpt-5.5", "worker": ""}}` (empty = inherit, matching existing manifest
89+
semantics). Resolved through the existing `modelregistry.Resolve` — same
90+
resolver every other model input already goes through, so aliases and
91+
deprecation fallbacks are handled for free.
92+
- Ship sensible defaults directly on the three builtins in
93+
`internal/specialist/builtin.go`: `explorer` (read-only enumeration) to a
94+
small/fast tier, `code-review` to a strong reasoning tier, `worker` stays
95+
inherit/balanced. This is close to a one-line-per-specialist change given
96+
the field and its wiring already exist.
97+
- Classification signal: the specialist/role name at the `Task` call site.
98+
This is a free, reliable signal — Zero already knows which role is being
99+
invoked because the calling model names it explicitly. No text classifier
100+
needed, which sidesteps RouteLLM's hardest problem (it has no such label
101+
in a raw chat stream; Zero's architecture already segments work by role).
102+
- Savings telemetry: extend `zero usage` with a comparison line — "routing
103+
saved $X vs. running every call on your primary model" — by reusing
104+
`internal/usage`'s existing per-model `CostBreakdown` aggregation.
105+
106+
### 4.2 Refine + Escalate (Sakana-inspired "wider vs. deeper," pragmatic)
107+
108+
**Deeper (already shipped, no new work required):** `SelfCorrector.AfterEdit`
109+
verifies, synthesizes feedback, retries the same model, gives up after 3
110+
attempts (`OutcomeAborted`).
111+
112+
**Wider (new):**
113+
1. Stop discarding the `Outcome` at `loop.go:602`; branch on it.
114+
2. On `OutcomeAborted`, for a **verifiable** task (tests/build/lint exist to
115+
check against):
116+
- Simplest correct version: retry once more on a stronger single model
117+
(classic FrugalGPT-style cascade escalation), carrying the same
118+
corrective feedback `SelfCorrector` already synthesized.
119+
- Stronger version (follow-up, not required for first ship): fan the
120+
still-failing task to a small panel (2 models) and let the *existing
121+
verifier* pick whichever candidate actually passes — cheap because the
122+
test suite is the judge; no extra judge-model call needed.
123+
3. For an **unverifiable** task with no tests/build to check against, Refine
124+
has nothing to referee an escalation with — hand off to Panel instead
125+
(§4.3), don't try to force a verify-loop where none is meaningful.
126+
127+
This keeps "deeper" exactly as it already works and adds "wider" as a
128+
genuinely new but narrowly-scoped escalation policy sitting on top of an
129+
existing, tested signal.
130+
131+
### 4.3 Panel (quality-maximizing — OpenRouter Fusion pattern)
132+
133+
- New dispatch primitive: fan one prompt to N diverse models in parallel
134+
(reusing `exec.go`'s subprocess-launch-with-model-override underneath),
135+
collect N responses, synthesize with a judge call that surfaces consensus,
136+
contradictions, and blind spots (mirroring Fusion's actual synthesis
137+
shape, not just picking one response).
138+
- Explicit opt-in only: a `/panel <question>` command, or a `panel: true`
139+
flag on a `Task` call. Never automatic or default — it's billed as the sum
140+
of every panel completion plus the judge, same cost shape OpenRouter
141+
itself documents.
142+
- Primary use: unverifiable, high-stakes, or ambiguous judgment calls —
143+
architecture decisions, "should I use X or Y," irreversible/security-
144+
sensitive changes — where being wrong is expensive enough to justify
145+
paying for multiple opinions up front.
146+
147+
### 4.4 `auto` mode — the combination
148+
149+
A policy, not a fourth execution mechanism. Sequences the three above:
150+
151+
1. **Route** picks the cheapest-sufficient model for the task's role.
152+
2. Mutating edits run through **Refine**'s existing verify-loop unchanged.
153+
3. On `OutcomeAborted`: if verifiable, escalate **wider** (stronger model,
154+
or verifier-arbitrated mini-panel). If unverifiable and either
155+
low-confidence or explicitly flagged high-stakes, escalate straight to
156+
**Panel**.
157+
4. Otherwise, stay on the cheap path.
158+
159+
Net effect: the cheap model absorbs the routine majority of turns (where
160+
most token volume actually sits); Panel's multi-model cost is only paid on
161+
the minority of cases that demonstrably need it. This is the concrete
162+
mechanism behind "both at once" — cost savings and quality aren't competing
163+
settings, they're different rungs of one cascade.
164+
165+
## 5. Build sequencing
166+
167+
Route and Panel do not depend on each other and can be built in parallel;
168+
`auto` depends on all three existing first.
169+
170+
1. **Route** — foundational, lowest risk, mostly config + defaults on
171+
already-wired plumbing. Also required as `auto`'s first tier regardless
172+
of sequencing.
173+
2. **Wire the discarded `Outcome`** at `loop.go:602` and define the
174+
escalation-policy interface — small, mechanical, unlocks the extension
175+
point cleanly before the escalation mechanics themselves exist.
176+
3. **Refine's "wider" escalation** for verifiable tasks — ship the
177+
stronger-single-model fallback first; verifier-arbitrated mini-panel is a
178+
follow-up enhancement, not required for first ship.
179+
4. **Panel** — new fan-out dispatch + judge synthesis, its own opt-in
180+
command; independent of step 3, can proceed in parallel.
181+
5. **`auto` mode** — composes 1+3+4 once all three exist. Pure policy/config,
182+
no new execution machinery of its own.
183+
184+
## 6. Risks
185+
186+
- **Silent quality regression on Route.** A cheap model can silently
187+
underperform on a harder-than-expected task. Mitigated by Refine's
188+
escalate-wider path once §4.2/step 2-3 exist — don't ship Route's defaults
189+
as a dead end with no escalation path for long.
190+
- **Surprise.** A session should not silently feel like multiple different
191+
models without the user knowing. Surface "ran on {model}" per specialist
192+
call in the TUI (`internal/tui` already renders specialist tool-call
193+
dispatch — this is a display change, not new infra).
194+
- **Provider/quota sprawl.** Routing across more of the 25+ supported
195+
providers within one session raises concurrent auth/rate-limit surface.
196+
Not a new problem class (credstore already handles multi-provider auth),
197+
but exercise it under real routing load before shipping.
198+
- **Panel cost blowup if scope creeps.** The single biggest risk to the
199+
whole feature paying off is letting Panel's per-query cost multiplier leak
200+
into default behavior. Keep it opt-in-only, and keep `auto`'s escalation
201+
criteria to Panel deliberately narrow (unverifiable + low-confidence, or
202+
explicitly flagged high-stakes) — not "whenever unsure."
203+
- **`OutcomeAborted` today has no escalation consumer.** Until step 2 of the
204+
build sequence lands, `SelfCorrector` exhausting its budget is a dead end
205+
that surfaces `abortedSelfCorrectNotice` and stops. Refine's value only
206+
materializes once wider-escalation is wired — don't consider Refine
207+
"shipped" with only the deeper half.

0 commit comments

Comments
 (0)