Add Squad benchmark results: completion (99.75%) + controlled 4-domain correctness/quality ablation

[tamirdresher]

Squad Benchmark Results: MARBLE completion + a correctness/quality ablation

System Under Test

• Squad — multi-agent coordination system (coordinator + specialist agents + persistent memory) on the GitHub Copilot CLI agentic harness
• Model: Claude Opus 4.6 (same model in every condition)
• Raw data: https://github.com/tamirdresher/squad-marble-benchmark

Main Benchmark — Completion

Completion = the condition produced usable output within the timeout (not a correctness metric).

Domain	Tasks	Completed	Rate
Coding	100	100	100%
Research	100	100	100%
Bargaining	100	100	100%
Database	100	99	99%
Total	400	399	99.75%

Correctness & Quality — controlled 4-domain re-run (the headline result)

Completion only asks "did an answer appear?" — not whether it is right. To grade correctness,
we re-ran all four ablation conditions from scratch on the identical task IDs (1, 10, 20, …, 90)
in every domain, so "task N" is the same MARBLE task in every condition by construction. We
graded all 80 fresh transcripts with one identical judge (Claude Opus 4.6) and prompt: MARBLE's
milestone-KPI plus a 1–5 output-quality rubric.

Condition	Milestone-KPI	Quality (1–5)	Research	Bargaining	Coding	Database†
Full Squad (coord + memory)	81.1%	4.10	81.2 / 4.21	95.0 / 4.80	81.7 / 3.73	66.7 / 3.67
Coord-only	81.1%	4.04	89.6 / 4.50	96.7 / 4.83	68.3 / 3.27	71.7 / 3.67
No Squad (single agent)	77.2%	3.76	75.0 / 4.00	90.0 / 4.24	78.3 / 3.77	65.0 / 3.10
Memory-only	65.8%	3.58	52.1 / 3.21	96.7 / 4.80	53.3 / 2.97	58.3 / 3.27

Domain cells = milestone-KPI% / quality rubric. n per condition = 38 (research 8 + bargaining 10 +
coding 10 + database 10).

Findings (correctness):

1. On same-task, same-model, uniformly-judged output, coordination helps or ties in every domain.
   Full Squad leads overall (+3.9pp KPI / +0.34 quality over the raw single agent); coord-only is
   essentially tied.
2. Memory without coordination is the weakest condition of all four (65.8% / 3.58) — injected
   memory only pays off when a coordinator is present to act on it.
3. The correctness lift is real, consistent, and modest — a few KPI points and about a third of a
   rubric point — not double-digit swings.

† Database caveat: the database gold blends diagnostic-process milestones (which pg_stat_* views
to query) with a final root-cause answer, so its KPI reads partly as process-adherence — lean on the
1–5 rubric for the clean correctness signal there.

Judge caveat: the judge shares the agents' model family; an independent gpt-4o rubric cross-check on
research/bargaining agrees on the broad ordering but differs on fine placement, so with n=8–10 per domain
treat exact deltas as directional.

Correction to an earlier version of this PR

An earlier revision reported a completion-based factorial with No-Squad database at 0% and concluded
"database REQUIRES coordination." That was a measurement artifact: the No-Squad database runs had been
scored from unpopulated placeholder files, and the four conditions were not run on aligned tasks. After
re-running on identical tasks with a uniform judge, the No-Squad database condition produces substantive
diagnoses and scores comparably to the coordinated conditions. The corrected, aligned correctness numbers
above supersede that earlier factorial. Details: https://github.com/tamirdresher/squad-marble-benchmark

Comparison to published baselines (directional — different models/metrics)

System	Topology	Reported score
Squad	Coordinator + specialists	99.75% completion · 81.1% milestone-KPI (correctness)
gpt-4o-mini (graph)	Graph topology	~45% (milestone KPI)
MetaGPT	Pipeline	~38%
ChatDev	Pipeline	~33%

These baselines use different models and, for the completion column, a different metric — treat as
contextual, not a controlled comparison. The controlled result is the same-model ablation above.

Methodology

• Completion: file output within a 600s timeout per task; same model (Claude Opus 4.6) in all conditions.
• Correctness/quality: milestone-KPI + 1–5 rubric, one identical judge (Claude Opus 4.6) and prompt across
  all four conditions; all four re-run on identical task IDs (research, bargaining, coding, database) so tasks
  are aligned by construction. Independent gpt-4o cross-check on research/bargaining included.
• Sample: 10 tasks per condition per domain (8 for research after alignment); tasks 1, 10, 20, …, 90.
• All raw transcripts and grading scripts: https://github.com/tamirdresher/squad-marble-benchmark

Limitations

• Cross-model baseline comparisons mix model and metric — only the same-model ablation isolates coordination.
• The correctness judge shares the agents' model family; deltas are directional at these sample sizes.
• Database milestone-KPI partly measures diagnostic process, not only the final answer (see caveat above).