Benchmarks scale

Benchmarks: scale

What a room's channel is worth as the room grows. The main benchmark page reports a room of five; this one sweeps room size against channel topology and reports where the five-member result stops holding, which is sooner than the five-member table suggests.

Produced by --scale-sweep; see Reproducing at the foot.

The short version

No. The harness does not improve with scale — it degrades, and on the adversarial task it degrades to nothing. Two separate reasons, and only one of them is a defect.

On an easy task the protocol's advantage disappears because the control gets good. On a hard task every floor-bound mechanism reaches zero, because the thing that has to travel cannot get through a floor that more members are competing for. Only a channel that costs no turn survives, and even that decays.

Why the default task stops being informative

Extend the size ladder past eight on uniform noise and both arms saturate:

agents      8       16       32       64
vote     83.3     96.7     99.3    100.0
hive+    83.3     95.0     99.3    100.0

A plurality of independent noisy readings converges on the truth by itself as the room grows. By sixteen members there is nothing left for a protocol to add, and a benchmark run there measures the law of large numbers rather than the library. The margin the main page reports — +3.2 at three members decaying to +1.2 at eight — does not level off. It crosses.

So the regime worth sweeping is the one where pooling does not trivially win.

Hidden profile, by room size

A decoy above every member's own argmax except one, who alone holds the fact that rules it out. 1000 seeded rooms per size, --noise 50.

arm                 3        5        8       16       32       64
ladder           44.6     33.8     28.4     27.0     22.5     19.4
vote             31.2     15.9      7.4      1.1      0.0      0.0
hive+            31.3     16.0      8.7      1.0      0.0      0.0
hive+fact        24.1     15.8      9.8      0.9      0.0      0.0
hive+rounds      31.3     16.1      9.1      1.1      0.0      0.0
hive+fact°       56.7     44.7     32.9     17.6      6.4      1.1
hive+pooled      95.7     91.9     87.0     85.2     87.4     89.8

Every floor-bound channel dies. The matched-budget vote, broadcast deliberation, and on-floor pairwise checking all reach 0.0% by thirty-two members. Growing the room makes a hidden profile monotonically worse for any majority mechanism: the lone fact-holder is one voice among more and more.

The information never leaves the room. hive+pooled — every reading and every fact in every member's hands, free — is flat at 85–96% across the whole ladder. Nothing is lost as the room grows. What fails is every mechanism for moving it.

Only an off-floor channel survives. hive+fact° — the aimed check carrying the fact rather than a number, held before the episode opens so it spends no turn — is the one arm above zero at every size: 44.7 against hive+'s 16.0 at five members, 6.4 against 0.0 at thirty-two. Its absolute lead decays, and it is still the only thing that is not nothing.

This is the law ADR 0011 and ADR 0012 established at five members, with room size turning the volume up: an exchange that costs a turn cannot pay for itself, and the larger the room the more turns the floor is already spending.

Volume is not the answer either

contacts/ep      3        5        8       16       32       64
hive+rounds   13.5     32.0     76.5    275.3   1059.1   4173.7

hive+rounds — continuous off-floor exchange, everybody contacting somebody every turn — costs model calls growing roughly with the square of the room and buys nothing: accuracy identical to hive+ at every size. What works is the aimed, bounded, fact-carrying single check. A host should size exchange_cap against who needs reaching, never against how many peers exist.

A federation exhausts itself answering

Three-desk federations are the shape the main page reports, where swarm reaches 77.5% against a vote at 4.0%. Grow the federation and that collapses. 200 federations per size, four members per desk:

desks          3        6       12       24
swarm       78.0     69.0      0.0      0.0
pooled      75.5     85.0     94.5     99.5

At twelve desks the swarm arm decides nothing at all, and the diagnostic says why rather than leaving it to be inferred:

arm        correct  decided     turns  crossings  stranded
swarm         0.0%        0     370.0      246.0      20.0
siloed desk endings: converged 2399 · deadlocked 0 · exhausted 1 · idle 0
swarm  desk endings: converged 0 · deadlocked 0 · exhausted 2400 · idle 0

Every desk exhausted its turn budget, while the same desks with referrals off converge in 2399 of 2400 episodes. A referral costs the answering desk a floor turn, and a desk in a federation of twelve is asked by eleven peers while its budget stays what it was for a federation of three. Incoming load scales with the federation; the budget does not. Twenty answers are stranded on top.

That is the on-floor exchange law one level up: inside a room the exchange costs the asker a turn, and between desks it costs the answerer one. The fix implied is the same — a referral should be answerable off the floor — and it is not implemented.

What this does not show

• One hard task. The hidden profile is the adversarial case on purpose. It says nothing about a room whose members are merely noisy, where the table above shows every mechanism saturating instead.
• Simulated members. The participants are arithmetic, as everywhere in this benchmark. What is measured is whether a policy moves information, not whether a model writes a useful row.
• hive+fact° is given oracle targeting. It picks its peer from private room state rather than from the transcript, so it is an upper bound on what an aimed off-floor check is worth, not an implementable arm. The gap to hive+fact is what aiming and scheduling are jointly worth; the gap to hive+pooled is what is still on the table.
• The federation result is a diagnosis, not a fix. Nothing here makes a referral answerable off the floor.
• No live models. Every number on this page is simulated.

Reproducing

cargo run --release -p tinyhivemind-hive --example bench -- \
  --scale-sweep --hidden-profile --episodes 1000 --sizes 3,5,8,16,32,64

cargo run --release -p tinyhivemind-hive --example bench -- \
  --scale-sweep --episodes 1000 --sizes 8,16,32,64        # uniform noise

cargo run --release -p tinyhivemind-hive --example bench -- \
  --swarm --desks 12 --per-desk 4 --episodes 200

Everything is seeded: the same flags produce the same rooms and the same transcripts, so a change in a number here is a change in the library.

The working record behind this page, including what had to change in the harness before the question could be asked at all, is in docs/experiments/2026-09-09-topology-at-scale.md.