Token economics in genesis v0.3.6: real-telemetry validation across 3 scenarios × 4 cells

[danielmeppiel]

Headline

This PR makes token economics a first-class design dimension in the genesis skill (v0.2 → v0.3.6). A new vocabulary of patterns lets the architect reason about which model each sub-agent runs on, which tools it can call, and how big each prompt needs to be — not just whether the workflow is correct.

We validated this on three workloads against four execution shapes each (twelve cells total), with every dollar harvested from the Copilot cloud events table — no estimates, no models. The headline is honest:

v0.3.6 is insurance, not optimisation. It pays back on workloads where missing a finding is expensive — supply-chain security, anti-pattern rejection, verifier-confirmed audits. It does not pay back on simple workloads where a single Sonnet pass already produces good output.

v0.3.7 update (this PR's headline addition). The B14b CAVEMAN BRIEF + B14c AUDIENCE BOUNDARY substrate is now empirically load-bearing. A cold-start fidelity test (Opus architect → fresh Sonnet executor, packet-only, no skill loaded) produces 5/5 CAVEMAN_FULL spawn briefs on the wire for S1's 5-lens panel (vs v0.3.6's 0/9), and the cost effect at quality parity is −24.6% on S1, −27.5% on S2N with real telemetry. S3 substrate path is byte-identical to v0.3.6; its +37.5% cell number is documented fixture-rebuild overhead. Detail in § v0.3.7 cold-start packet fidelity test.

The next three sections explain why that conclusion is the right one, what the experiment actually measured, and how to read the per-cell numbers.

---

TL;DR — when to pick which

If you need...	Pick	Why (one-line evidence)
A deterministic batch op (rename, codemod, regex refactor)	zero-sonnet with shell access	Sonnet + one sed call cost $4.81 on S3 — architecture is dead weight here
A one-off doc audit, mistakes are cheap to recover	zero-sonnet	$6.20 for a 7/10 audit on S2; architected cells charge $3.60 more for one quality point
Verifier-confirmed precision on a structured-finding job	v0.3.6	S2-v0.3.6 is the only cell with an A7 verifier pass — 6/6 HIGH confirmed, 0 downgrades, $9.80
To catch supply-chain BLOCKERs on a security-sensitive PR review	v0.3.6	S1-v0.3.6 caught both arbitrary-cmd-exec and validation-bypass BLOCKERs at $24.59; zero-sonnet missed both at $8.89
A workflow you can run nightly in CI, audit by reviewer, hand to a junior operator	v0.3.6	The architected packet is a committed artifact with per-lens trace; zero-x is per-run improvisation
To avoid the worst-case cost trap (over $30 on a small job)	v0.3.6, never v0.2	v0.2's per-file loop cost $33.79 on the same rename v0.3.6 did for $10.40 — bounded variance is the real win

Per-cell scorecard (raw $, quality, weighted ROI):

Cell	Cost	Quality	ROIraw	ROIweighted	What it bought
S3-v0.3.6	$10.40	10/10	0.96	0.96	Bounded-variance rename via S7 BRIDGE
S3-zero-sonnet	$4.81	10/10	2.08	2.08	Cheapest correct path on a deterministic batch
S3-v0.2	$33.79	9/10	0.27	0.27	Cautionary tale — per-file loop
S3-zero-opus	$41.01	2/10	0.05	0.02	Fuel-burned-on-a-wrong-symbol failure
S2-v0.3.6	$9.80	8/10	0.82	2.45	Verifier-confirmed precision (only cell)
S2-zero-sonnet	$6.20	7/10	1.13	2.90	Cheapest reasonable audit
S2-v0.2	$9.42	7/10	0.74	2.34	Five lenses, no verifier
S2-zero-opus	$33.01	8/10	0.24	0.73	Pays Opus premium without measurable return
S1-v0.3.6	$24.59	9/10	0.37	2.28	Only cell that caught both supply-chain BLOCKERs
S1-zero-sonnet	$8.89	6/10	0.67	2.59	Reasonable review, missed BLOCKERs
S1-v0.2	$3.94	3/10	0.76	2.79	Under-completed (27-line stub)
S1-zero-opus	$19.82	7/10	0.35	1.36	Opus premium does not pay back vs. Sonnet
S3-v0.3.7	$14.30	10/10	0.70	0.70	Substrate path identical to v0.3.6; cell overhead from fixture rebuild
S2-v0.3.7	$7.10	8/10	1.13	3.38	Best ROI in the matrix; 0 spawns (AUDIENCE BOUNDARY refusal)
S1-v0.3.7	$18.53	8/10	0.43	2.27	5/5 CAVEMAN_FULL on the wire; −24.6% vs v0.3.6 at near-parity

Higher ROI is better. ROI_weighted uses BLOCKER × 5, HIGH × 3, MEDIUM × 1, LOW × 0.3. Full ROI definition in REPORT.md § The ROI function.

---

Why this experiment exists

Genesis is a skill an LLM-driven agent loads to architect other agentic workflows. When a team says "design us a PR-review panel" or "audit our docs corpus", the genesis architect decides how many sub-agents to spawn, which model each runs on, what tools each can call, and how to wire their outputs.

Before this PR, the architect optimised for correctness only. Cost was implicit, and that made the architect prone to two failure modes we kept observing in practice:

1. Naive fan-out — spawning a 5-lens panel of Opus sub-agents for a task that a single Sonnet shell call could finish. Quality unchanged, cost 10–50× higher.
2. Wrong-primitive binding — designing a per-file view + edit loop for a 19-file rename instead of a single perl -i shell invocation. The agent does the work; it just does it through the wrong tool surface.

The patterns added in v0.3.6 give the architect explicit vocabulary for both: B12 SELECTION RULE (how to pick the right primitive), B15 TOOL SUBSET + S7 DETERMINISTIC TOOL BRIDGE (when a shell call replaces N file edits), A12 GRADIENT WORKFLOW + benchmark-grounded model catalog (Sonnet for the bulk, Opus only for narrow arbiter roles), and A1 PANEL + A7 ADVERSARIAL VERIFIER (multi-stream reviewability with precision evidence). The question this PR answers is: do those patterns actually move the needle in production-grade conditions, on real workloads, against honest baselines?

---

How we measured

Three scenarios, each run four ways:

• S3 — Bulk rename (19 JS files, 60–90 reference sites, npm test must pass)
• S2 — Doc audit (11-page CLI documentation corpus, drift + link + schema lenses)
• S1 — PR review (feat(lsp): add first-class LSP server support to install pipeline  microsoft/apm#1424, +2363/-114 across 24 files)

The four cells per scenario are zero-opus and zero-sonnet (single-shot prompt to a frontier model, no architecture), v0.2 architected (workflow designed by the previous-version architect), and v0.3.6 architected (workflow designed by this PR's architect with the new pattern vocabulary).

Cost is the real-telemetry sum of usage_input_tokens, usage_output_tokens, usage_cache_read_tokens, and usage_cache_write_tokens for the executor session and any sub-agents it spawned, priced at Anthropic public rates. Per-cell cost-report.json files are committed alongside chat-session ids for replay. The architect session is not counted — architecting is amortised, infrequent infrastructure.

Quality was graded by the orchestrator against scenario-specific rubrics: binary correctness plus tool-mechanic cleanliness for S3, real structural drift caught + verifier precision for S2, actionable bugs caught + severity calibration for S1. Full grading detail with the per-finding evidence is in REAL-TELEMETRY-RESULTS.md.

A buyer's decision is return on each dollar, not absolute spend, so we score every cell on three ROI axes — raw QualityScore / Cost, severity-weighted (BLOCKER × 5, HIGH × 3, MEDIUM × 1, LOW × 0.3), and tail-risk-adjusted (QualityScore / (Cost + P(failure) × C_failure)). The first answers "what does the dollar buy?"; the second weights findings by impact-class; the third dominates when missing one finding costs orders of magnitude more than the run itself. Higher is better on all three. The full definition with inputs and caveats is in REPORT.md § The ROI function.

---

What we found — three findings, in plain language

1. Bad architecture costs more than no architecture

The S3 rename cell where the v0.2 architect designed a per-file view + edit loop cost $33.79 in real telemetry. The same rename, executed by Sonnet directly with one sed shell call, cost $4.81. Same outcome, 7× the cost, because every tool call round-trips the entire conversation context back to the model and a 19-file rename done one file at a time pays that overhead 19 times.

The v0.3.6 architect, given the same brief, rejected the per-file design and chose a single grep | xargs perl -i shell call (the S7 DETERMINISTIC TOOL BRIDGE pattern). It cost $10.40 — 3.2× cheaper than v0.2 and within 2× of zero-sonnet — and the architecture overhead bought bounded variance: no risk of the per-file loop trap, no risk of substituting the wrong symbol the way zero-opus did. This is the load-bearing claim of the PR: the new pattern vocabulary makes the architect reject wrong-tool-surface designs, and the gap is measured, not modelled.

2. On simple workloads, single-shot Sonnet is the cost-rational baseline

On S2 doc-audit and S1 PR review, zero-sonnet wins on raw $/quality. Sonnet at $6.20 produced a 7/10 audit. The v0.3.6 architecture at $9.80 produced an 8/10 audit. Sonnet at $8.89 produced a 6/10 PR review. The v0.3.6 architecture at $24.59 produced a 9/10 review.

Architecture has a fixed overhead. Every sub-agent dispatch reloads the Copilot CLI tool descriptions, the system prompt, and the genesis skill bundle — roughly 80K input tokens per turn that the single-shot path does not pay. On workloads where one Sonnet pass already produces good output, that overhead is dead weight. The v0.3.6 architect knows this: its S2 doc-audit handoff explicitly chose monolithic single-session over fan-out, citing the B12 SELECTION RULE wrong-primitive-binding warning.

What architecture buys on these workloads is not lower mean cost. It is multi-stream reviewability (five independent lens verdicts instead of one prose blob), class-routed quality (Sonnet on reviewer lenses, Haiku on trivial ones), and — uniquely on v0.3.6 — adversarial verification. The S2-v0.3.6 cell is the only one with a verifier pass that confirmed 6/6 HIGH findings with zero downgrades. That is the kind of precision evidence a buyer cannot get from a single-shot prompt at any cost.

3. On production-critical workloads, severity weighting flips the verdict

The S1 PR review is the test case where ROI_raw and ROI_tail diverge. Three of the four cells produced reasonable reviews. Only v0.3.6 caught the two supply-chain security BLOCKERs — arbitrary command execution via plugin-controlled LSP config, and validation bypass via the raw-dict install path. zero-opus and zero-sonnet both missed both. v0.2 under-completed the workflow entirely (a 27-line YAML stub with three bullet points).

On raw $/quality, zero-sonnet still wins (0.67 vs v0.3.6's 0.37). On severity-weighted ROI, the gap closes (zero-sonnet 2.59 vs v0.3.6 2.28 weighted-points-per-dollar) — the BLOCKER × 5 weighting moves v0.3.6's 56 weighted points to nearly twice the next cell's. On tail-risk-adjusted ROI, v0.3.6 dominates: the cost of merging a supply-chain CVE into a published CLI is at minimum thousands of dollars in remediation and reputation, against an architecture premium of ~$15. The math says: pay the premium when C_failure ≫ Cost, do not pay it when it does not. That is precisely the conclusion the v0.3.6 architect's catalog now encodes.

The full per-cell numbers, including ROI_raw, ROI_weighted, and the failure-mode column for each cell, are in the scorecard at the top of REPORT.md. Cost methodology is in PROFILING-PROTOCOL.md. Full grading detail with the per-finding evidence is in REAL-TELEMETRY-RESULTS.md.

---

---

Per-pattern attribution — which patterns moved which dollars

Pattern	Empirical effect	Confidence
S7 DETERMINISTIC TOOL BRIDGE	$23.39 saved on S3 (v0.2 $33.79 → v0.3.6 $10.40) — same brief, same fixture	Strong
B12 SELECTION RULE + explicit model: declarations	~$3–5 saved on S1 by routing 63 trivial-lens calls to Haiku ($2.01) instead of Sonnet	Strong — per-model breakdown in cost-report.json
A12 GRADIENT WORKFLOW	Applied in S1; refused by the architect in S2 and S3 (gradient-free workflows). The refusal is itself the win on simple workloads	Strong-by-omission
B15 TOOL SUBSET	Visible at the Haiku tier (6K input floor vs. 35K Sonnet sub-agent)	Partial — only effective on tier-aware agent types
A1 PANEL + A7 ADVERSARIAL VERIFIER	Quality patterns, not cost patterns. Spend $3.60 (S2) and $15.70 (S1) over zero-sonnet for verifier-confirmed precision and BLOCKER-catching	Strong — only S2-v0.3.6 has a verifier pass; only S1-v0.3.6 caught the BLOCKERs
B13 CACHE-AWARE PREFIX	No measurable differentiation — all cells already cache at 93–99%	None — currently a guideline, not a lever
B14b CAVEMAN BRIEF	Empirically load-bearing in v0.3.7 (this PR). 5/5 CAVEMAN_FULL on the wire on S1; cost delta −24.6% on S1, −27.5% on S2N at quality parity. Receipt audit at S1-v037-real/caveman-classification.md	Strong — receipts queryable, cost delta in real telemetry

The honest summary: S7 + B12 + B15 + A12 + B14b (new in v0.3.7) are doing the work; B13 + B16 are catalogued but not yet pulling weight. Full attribution and v0.4 frontier (5 quick wins) in REAL-TELEMETRY-RESULTS.md § Per-pattern attribution and § v0.3.6 flaws and quick wins.

---

Reliability and predictability — the dimension ROI does not capture

Raw $/quality scores a single cell on a single run. It misses three properties that matter once the question shifts from "which is cheapest for this one task?" to "which workflow do I want in a CI pipeline run nightly, or handed to a junior engineer?":

• Variance bounds. Single-shot Sonnet has no worst-case floor. The architected workflow does — every lens runs, every verifier confirms, the worst case is the worst lens. n=1 here so we cannot measure variance directly, but the structural argument holds.
• Auditability. S1-v0.3.6 produced 5 lens artifacts + 1 synthesizer + 1 verifier, all committed. Zero-sonnet produced one prose blob. For workloads that gate production decisions (security review, compliance audit, release sign-off), the multi-stream artifact is a working audit trail; the blob is not.
• Repeatability. The architected workflow is a committed packet plus N committed sub-agents. A different operator, on a different day, running the same packet against the same brief, gets a comparable workflow shape. The single-shot prompt is improvisational — the prompt typed once in a chat session is not a reusable artifact unless a human turns it into a .agent.md and adds tests, which is exactly what the architect already does.

This shifts the buyer's calculation. For one-off use, zero-sonnet is fine. For productionised, repeatable, automatable use, v0.3.6 is the only thing on the table — because it is the only one that produces a workflow artifact at all. Detail in REAL-TELEMETRY-RESULTS.md § Reliability and predictability.

What changed in the corpus

• New patterns in skills/genesis/assets/design-patterns.md: B12 SELECTION RULE, B13 CACHE-AWARE PREFIX, B14b CAVEMAN BRIEF, B15 TOOL SUBSET, B16 EFFORT GOVERNOR, S7 DETERMINISTIC TOOL BRIDGE
• New architectural pattern A12 GRADIENT WORKFLOW
• Model catalog with benchmark-grounding reference (vals.ai SWE-bench data)
• Empirical-proof corpus at dev/empirical-proof/ — twelve runtime cells with real-telemetry validation, formal ROI definition, per-cell artifacts and chat-session ids for replay

What this PR does NOT prove

This is a small, opportunistic study. We have n=3 scenarios and n=1 per cell. The conclusions above are about the shape of the cost-quality curve under v0.3.6 patterns, not a SWE-bench-grade benchmark. Specifically:

• It is not a claim that v0.3.6 is always cheaper. It is not, on raw $, for S1 and S2.
• It is not a claim the catalogue of patterns is complete. Telemetry shows every sub-agent dispatch pays a fixed entry tax of roughly 6K tokens (Haiku/explore), 35K (Sonnet sub-agent), or 54K (full orchestrator) before the prompt is read, and naive per-file loops amplify it via rolling re-send — S3-v0.2 burned a 290:1 input/output ratio for that reason. No pattern in this PR compresses that surface; it is the visible frontier for the next PR. The full breakdown, with per-bucket turn counts and per-cell I/O ratios, is in REAL-TELEMETRY-RESULTS.md.
• Quality grading is orchestrator-judged, not held against a human gold standard. We are confident about the ordinal rankings and the BLOCKER findings on S1; the absolute /10 scores are best-effort.
• P(failure|cell) in ROI_tail is the empirical 0/1 of this single run, not a calibrated probability. The tail-risk argument relies on the structural claim that C_failure ≫ Cost for security-critical workloads, not on a fitted failure-rate model.