Benchmarks

Benchmarks

What a room of agents buys over one agent answering alone, what it costs, and which settings decide the difference.

Everything here is produced by:

cargo run --release -p tinyhivemind-hive --example bench

The harness itself is documented next to the code, in crates/tinyhivemind-hive/examples/bench/. Episode policy is the tuning guide these numbers produced, and Further reading has the literature they sit against.

Summary

Five agents, four options, 5000 seeded rooms:

arm                quality     speed       thru    conc    tok/ep     tok/s
ladder               57.6%     5.1s        701     1.0      1.9k       361
vote                 78.5%     2.6s       1.4k    15.0     12.3k      4.8k
hive                 73.3%    15.8s        228     1.0      5.3k       336
hive+                82.1%    17.3s        208     1.0      6.0k       345
hive+wide            77.1%     8.7s        415     2.3      6.9k       798
hive+pooled          91.5%    15.6s        231     1.0      5.2k       335

A tuned deliberation is right 82.1% of the time. The matched-budget control, given all fifteen turns, reaches 78.5%. One responder off the ladder, which is how the system behaves today, reaches 57.6%.

And the deliberation takes six times as long to get there. vote answers in one round; hive+ takes seven. That trade — three and a half points of quality for 14 extra seconds and half the tokens — is the actual finding, and it was invisible for as long as the table reported the library's nanoseconds instead of the system's seconds. Whether it is worth taking depends entirely on what the room is deciding, which is what the grid is for.

Three things are worth reading off those six rows:

• A poll is fast and expensive; a deliberation is slow and cheap. vote spends 12.3k tokens in 2.6 seconds and hive+ spends 6.0k in 17.3. They sit at opposite corners, and which corner is affordable is a property of the deployment rather than of the protocol.
• Concurrency buys most of the latency back. hive+wide gives up 5 points against hive+ and halves the wall clock, doubling throughput. See ADR 0014.
• The ceiling is a long way above all of them. hive+pooled hands every member every peer's reading for free and reaches 91.5% — for fewer tokens than hive+, because it needs fewer turns. Nobody can deploy it; the gap to it is how much the protocols are still losing in the moving.

The margin over the control is a few points, and it is meant to be. Independent sampling plus a plurality is most of what a room is for, and a protocol that could not clear that bar would not be worth its budget.

What the columns are

column	what it is
quality	share of episodes that decided the genuinely best option
speed	mean wall clock, brief to decision — a sum over rounds
thru	episodes one seat pool finishes per hour at that latency
conc	mean turns in flight; 1.0 is strictly sequential
tok/ep	prompt and completion tokens one episode spends
tok/s	the rate those tokens are drawn at

Five of the six are computed from a model of what a turn costs — 42 tokens a transcript row, 180 a turn, a 600-token base prompt, 450 ms to first token, 85 tokens a second decoded. Every run prints that point with the flags that reproduce it, and --calibrate measures all four against a real endpoint. Nothing below should be believed at one setting of those constants that does not survive moving them; the constants that can actually reorder an arm are named in COST.md.

The library's own cost is reported separately, and is about 2.3 microseconds per step — six orders of magnitude below a model turn. It used to share this table, where vote's ns/step 0 and episodes/s inf read as "this arm is free" rather than "this arm never calls the library".

The grid

The four questions a host actually asks — what does this do on my kind of problem, at my size, at my difficulty, at the concurrency I can afford — are one question with four parameters. --grid walks the cross product:

cargo run --release -p tinyhivemind-hive --example bench -- \
  --grid --topic all --scale 5,11 --complexity 1,3,5 --concurrency 1,3

Every cell runs the same five systems and reports the same six columns, so a row and the row beneath it differ in exactly one axis value. Two results the single-point table could not have shown:

A deliberation is not uniformly better than a poll — it depends on the topic. On a hidden profile, where one member alone holds the fact that rules out the decoy everybody else prefers, hive+ scores 15.3% against the poll's 15.0% while pooled reaches 92.3% (3000 rooms). The protocol is not moving the deciding fact at all on that topic; the information is in the room and stays there.

Difficulty inverts the ordering. At complexity 1 every arm but the ladder is at or near 100%. At complexity 5 — eight options at ±150 — hive+ falls to 43.2% and is beaten by the 47.5% poll (3000 rooms). Deliberation helps in the middle of the difficulty range and hurts at the end of it.

What is measured

A room chooses between several options, exactly one of which is genuinely best. Every member holds a private, noisy evaluation of every option: the true quality plus a uniform error of half-width --noise. No member is individually reliable, so the room's only route to the right answer is to pool what its members separately believe.

The participants are arithmetic, deliberately. A language model would make the numbers unreproducible and would confound protocol quality with model quality. What is being measured is whether a policy aggregates information or throws it away, which is the question a host has to answer when it configures a desk. Real models appear in live rooms, where the claim is only that they can hold the protocol.

Everything is seeded. The same --seed produces the same rooms, the same private evaluations, and the same transcripts, so a change in a reported number is a change in the library rather than in the weather.

The arms

Every arm decides the same rooms from the same private evaluations.

arm	what it is	turns
ladder	responder_plan selects one responder off the real ladder, selector rung included and validated through accept_selection, and that agent answers alone	1
vote	independent answers decided by plurality, nobody seeing anybody: self-consistency at a matched budget	the whole budget
hive	a deliberation episode at EpisodePolicy::DEFAULT	up to the budget
hive+	the same, at the tuned policy	up to the budget
hive+ref	the tuned policy with refutation_cap: Some(2)	up to the budget
hive+ev	the same, plus require_evidential	up to the budget

vote is the honest control, and Condorcet's jury theorem is the reason it is a strong one. A multi-agent result without such a control is close to meaningless, because the multi-agent arm has usually just spent more compute. It is given the whole budget, which is more turns than the deliberation actually spends, though with deterministic participants it saturates at one distinct answer per member, which is exactly what self-consistency does with a deterministic sampler.

Correctness is scored over the whole sample, including episodes that decided nothing. An arm cannot buy accuracy by declining to answer.

Results

The two bounds on quorum

The quorum threshold is the single most consequential setting, and it has a bound on each side. Five members, 5000 rooms, everything else held at the tuned policy:

quorum	deadlocked	exhausted	decided %	correct %
2 of 5, below a majority	514	0	89.7	73.3
3 of 5, smallest majority	0	29	99.4	82.1
5 of 5, unanimity	0	2092	58.2	55.2

Below a majority, rooms deadlock. Five members can put two grounded supporters behind each of two options, and an episode in which two options both carry is deadlocked by definition: no amount of further support resolves it, because both stay above the line. Requiring a majority makes that state unreachable and the deadlock rate falls to zero.

At unanimity, rooms cannot finish. Cross-inhibition removes a silenced advocate from a topic's supporter set and does not put them back, so a single grounded !object makes quorum unreachable for the rest of the episode. Two in five episodes then spend their whole budget without deciding. A live three-member room hit exactly this, described in live rooms.

So: a majority of the desk, and never the whole of it. The benchmark's tuned policy computes threshold = min(agents / 2 + 1, agents - 1).

Refutation and evidential grounds lose

Both are opt-in and both are off in QuorumPolicy::DEFAULT, and this table is why. 5000 rooms, five members, --noise 90, everything else at the tuned policy:

arm	quality	speed	tok/ep	decided %
vote	78.5%	2.6s	12.3k	100.0
hive+	82.1%	17.3s	6.0k	99.4
hive+ref	75.0%	23.1s	8.7k	88.6
hive+ev	55.9%	26.4s	10.4k	60.8

!refute costs seven points and drops the arm below even the vote control. require_evidential costs twenty-six and fails to decide two episodes in five, because a room that has not deposited facts cannot carry anything at all.

Both also cost what they lose twice over: hive+ref is a third slower and 45% more expensive than hive+, and hive+ev is half again slower and 73% more expensive. An arm that is worse, slower and dearer is an easy call, and until the table carried tokens and seconds only the first third of that was visible.

The damage scales with how noisy each member's private evaluation is — nothing at ±30, six points at ±60, seven at ±90, fifteen at ±120 — and that is the diagnosis. A refutation is global where an objection is local. An !object removes one advocate from one topic; a !refute caps the topic for the whole room, so a member firing one on a noisy read removes an option for everybody. This is the same neutrality the live rooms showed when their one observed !object fired against the correct option, with a much larger blast radius.

The grid search agrees without being asked: --sweep scores 864 policies over the same rooms, and every policy in the top twelve has both knobs off.

What this does not test is the case the mechanism was built for. The simulated task gives every member a noisy estimate of every option, so there is no decoy that accumulates support no individual's private read contradicts and no fact held by one member that overturns it — which is what a hidden profile is, and what the live checkout-503 scenario has. On a task where every member can already evaluate every option, weighing evidence against support has nothing to win and a real cost to pay. The full record, including the open items, is in docs/experiments/2026-09-01-refutation-and-grounds.md.

The budget has to scale with the desk

A fixed budget makes a larger room look worse than a smaller one, and the effect is entirely an artifact of the cap. An eight-member room, 1500 rooms each:

budget	decided %	correct %	turns actually spent
12	65.3	63.1	10.36
16	89.6	82.9	10.96
20	94.5	86.3	11.24
24	96.4	87.7	11.42

A blind opening round costs one turn per member before anybody has seen anybody, a majority then has to assemble on one option, and the decision has to be recorded. Three turns per member covers that, and it is a cap rather than a cost: the eight-member room finishes in 11.4 turns of the 24 it is allowed. At five members, budgets of 15, 20 and 25 score 82.0, 82.1 and 82.1. Past the point where the room can finish, extra budget buys nothing.

The blind round is not decoration

Turning it off, five members, 5000 rooms:

opening round	decided %	correct %
blind	99.4	82.1
full visibility	100.0	58.0

With full visibility from the first turn the room cascades onto whatever was proposed first and lands level with a single agent. That is an information cascade, and Visibility::Blind is what prevents it, bought as a filter on the projection rather than as concurrency. See ADR 0002.

Across desk sizes

2000 rooms per size, threshold and budget scaled as above:

agents	quorum	budget	ladder %	vote %	hive+ %	turns/ep	decided %
3	2	9	57.3	68.4	71.6	4.42	99.8
4	3	12	57.3	74.2	76.8	6.68	94.6
5	3	15	57.6	78.8	81.5	6.79	99.3
6	4	18	57.1	82.2	83.4	9.04	95.6
8	5	24	58.4	87.3	88.6	11.32	96.9

The deliberation beats the matched-budget control at every size, by 1.2 to 3.2 points, while spending roughly half the turns. Deadlocks are zero throughout — but the margin is decaying, and past eight members it crosses. On a hidden profile every floor-bound mechanism reaches 0% by thirty-two members, and only a channel costing no turn survives. See Benchmarks: scale.

What the library costs

ns/step is one call to tinyhivemind_hive::step over a live transcript, with the participants' own time excluded. That is about 2.3 microseconds, or roughly 57,000 whole episodes per second on one core. An episode of nine steps costs about 20 microseconds of library time. A model turn is six orders of magnitude more expensive, so the protocol is free in any real deployment.

Three changes made during this work cut that cost by about a fifth, measured before and after under identical settings (2816 to 2222 ns/step). None of them changes behaviour, and every arm's outcome was byte-identical across them:

episode::step folds a borrowed Vec<&SessionMessage> rather than cloning the filtered transcript on every step, and computes consensus once instead of twice. trace::extract returns early on a body containing no !, which is most of a real transcript, before scanning for fences. quorum::standings folds on borrowed topic and agent keys and allocates owned strings only for what survives, and attention::bids hoists saturation and the reader-independent half of salience out of its per-member loop.

The remaining cost is dominated by re-reading the transcript on every step, which is inherent. An episode is a pure fold with no state cached between calls, so a step over a transcript of n messages parses n messages.

Live rooms

--agent-cmd swaps the simulated participants for a real agent CLI, one process per authorized turn. Those runs assert nothing about accuracy — a handful of live episodes could not measure one — but they establish that real models hold the trace grammar, and they surfaced a set of host-side obligations the simulation cannot see: one idea proposed twice under two topic names, a !commit written while the room was still deliberating, dropped sigils, and a question restated four turns running. Each fix is one a host owes its agents rather than something the library can impose.

--scenario then gives the room a problem with a recorded answer. On checkout-503, a hidden profile whose answer needs four facts held by four different members, deliberation reached it in six episodes of ten while the independent poll reached it in none of sixteen. Every failed room failed the same way — the fact that killed the decoy was in the transcript and changed nothing — which is the argument for a !refute that debits a topic rather than an author.

See Benchmarks: live rooms for the rooms, the two scenario designs thrown away before one could separate the arms, and the harness defects the runs exposed.

A federation of desks

--swarm measures something the arms above cannot: several desks that cannot read each other's transcripts, deciding one question. See Cross-desk referral for the mechanism.

The task changes shape to make the boundary cost something. Each desk overrates one option — a different one per desk — because its members read the same transcript and are wrong about the same thing. Within a desk that bias is invisible and averaging correlated error does not remove it; across desks the biases cancel. Three desks of four, 400 seeded federations:

arm        correct  decided     turns  crossings
siloed        0.2%        1      15.9        0.0
swarm        77.5%      389      32.3       12.0
pooled       74.5%      371      16.7        0.0
merged       10.5%       96      33.8          —
vote          4.0%      141      12.0          —

siloed is the same desks with referrals off, and it is not merely worse — it is destroyed. 1,199 of its 1,200 desk episodes reach a confident decision, and three confident desks disagreeing three ways produce no plurality at all. A federation of well-run rooms that cannot talk does not degrade gracefully.

merged puts all twelve members on one desk with the whole budget, and scores 10.5%: a larger room with three factions cannot assemble a majority quorum, so most episodes exhaust. Removing the boundary is not the fix, and costs the same turns as crossing it. pooled is the ceiling control — every desk handed every other's readings for free — and swarm matches it, so the protocol delivers essentially all of what the information is worth and what it costs is turns. At --bias 0, where no desk has a blind spot, every arm scores 100% and crossing buys nothing at twice the turns; that is why every knob in ReferralPolicy::DEFAULT is off.

The largest effect measured is not in the library. A desk whose members share a bias reaches quorum inside its own blind opening round, so a fact arriving after that is one the desk has already voted past: asking before backing anything rather than after is the difference between failing outright and 77.5%.

Five live runs against claude -p --model sonnet add what the simulation cannot. The mechanism works end to end with real agents, and in the best run a desk reached an answer no member of it could have reached alone by asking another desk for a number. But agents ignored the move entirely until it was placed in the marker list rather than above it, and one desk answered a question with its hypothesis instead of its evidence and exported the error intact. A protocol that moves messages does not by itself move evidence. Full write-up, sweeps and transcripts: docs/experiments/2026-09-02-federated-hidden-profile.md.

Delegation and expertise

A second family of arms asks what changes when the room's members are not interchangeable: --specialists, --hidden-profile, --blind-evidence, --defer-cap and --cost-tiers. The short version is that the deciding fact reaches the floor in 1.5% of hidden-profile episodes under the ordinary opening and 96.8% when members deposit before they argue — 15% against 66% correct — while the directory itself moves nothing outside the interval and a directed router loses to a uniform draw. Both knobs ship off.

Twenty-four live rounds across nine backend rows say the same from the other side. The matched-budget poll found the answer in none of them; the fact-holder spoke before the commit in every room that had one and twelve of those twenty rooms were still wrong; no turn was awarded on BidReason::Knows; and !defer was used on none of 240 turns although it sat in every prompt's move list. Putting a reasoning model on the expert seat cost eight times as much and scored 0 of 8. See Benchmarks: delegation.

Private exchange, and what it is worth

A third family of arms asks whether letting two members say something the desk cannot read makes the room decide better. The short version: a private exchange that spends a floor turn costs -16.9 on a hidden profile, and the reason is not privacy — an arm that writes the identical words and discards the answer loses more. The cost is the turn. Charge it differently and the sign flips: an aside riding alongside the turn that authored it is +0.5, and exchange rounds held off the floor reach +3.2 at forty-five model calls an episode. Free pooling bounds the whole question at +30.8 — the largest effect anywhere in this benchmark, and in fewer turns. Uniform rooms are a null throughout.

See Benchmarks: private exchange, and Private asides for the mechanism.

Reproducing

cargo run --release -p tinyhivemind-hive --example bench                      # the table above
cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000
cargo run --release -p tinyhivemind-hive --example bench -- --agents 8
cargo run --release -p tinyhivemind-hive --example bench -- --hidden-profile --blind-evidence --budget 40 --episodes 2000
cargo run --release -p tinyhivemind-hive --example bench -- --quorum 5        # unanimity
cargo run --release -p tinyhivemind-hive --example bench -- --no-blind        # the cascade
cargo run --release -p tinyhivemind-hive --example bench -- --sweep           # the policy grid
cargo run --release -p tinyhivemind-hive --example bench -- --trace           # one episode
cargo run --release -p tinyhivemind-hive --example bench -- --swarm           # a federation
cargo run --release -p tinyhivemind-hive --example bench -- --swarm --bias 0  # no blind spots
cargo run --release -p tinyhivemind-hive --example bench -- \
  --agent-cmd "opencode run --pure -m openrouter/openai/gpt-5-mini" \
  --scenario crates/tinyhivemind-hive/examples/bench/scenarios/checkout-503.txt \
  --repeat 5                                                                  # a real problem

CI runs cargo run -p tinyhivemind-hive --example bench -- --episodes 25, so the harness cannot rot. Live mode is not in CI and needs a configured agent CLI.

What this does not show

Nothing about model quality. The deterministic participants are arithmetic. A room of language models may aggregate better or worse than this, and these numbers cannot tell you which.

Nothing about real tasks, in the deterministic arms. One synthetic task with a known best option and independent errors is the friendliest possible case for aggregation. Real disagreements are correlated, and correlated errors are exactly what pooling cannot fix — within the group that shares them. The federated arms above are the one place that limitation is measured rather than assumed, and they say what to do about it: pool across a boundary the correlation does not cross. They also say what not to expect, because two desks sharing a blind spot would confirm each other rather than correct each other, and that case is not measured anywhere here.

The scenario runs in live rooms are a first step off that synthetic task and are not a substitute for it. Four live rooms are an existence proof that a hidden profile separates deliberation from a matched-budget poll; they are not a rate, and the two rooms that failed did so by pooling their information under the wrong hypothesis, which is a failure mode the deterministic arms cannot produce at all.

Nothing about long rooms. Conformity in a group of language models rises with interaction time. The budgets here are small on purpose, and a longer episode should be expected to buy correlated error rather than better judgement.

tinyhivemind-hive is a protocol for bounded deliberation with an auditable termination reason. That is the whole claim.