b_compare costs 1.3-1.65x more per comparison under the #139 carry on soil 16S; unexplained, and paid for many times over

[cjfields]

What

On NovaSeq soil 16S only, enabling the #139 shuffle carry
(DADA2RS_SHUFFLE_CARRY=1) makes the k-mer screen 2.2-2.4x slower on
byte-identical work, giving back roughly a third of the shuffle saving.

	rebuilt	carry	delta
screen R1	495 ns/comp	1109 ns/comp	+124%
screen R2	512 ns/comp	1231 ns/comp	+140%
DP kernel R1 / R2	16361 / 15104	16588 / 15321	+1.4%

Comparison counts are identical in both arms (11,605,515,622 / 14,485,250,256),
as are screen-pass counts. Outputs are byte-identical. So this is the same work
running slower, not different work.

Cost: b_compare wall 706.8 -> 950.7 s (R1) and 749.8 -> 1096.7 s (R2), against
a shuffle saving of 764.9 / 869.2 s. Net still -32.0% / -28.8% of run_dada,
but without the give-back it would be roughly -47%.

Not a regression the change "causes"

ITS2 from the same instrument shows no screen effect at all (1856 -> 1820,
1669 -> 1697 ns/comp; -1.9% / +1.7%), across two reps. So this is 16S-specific.

Note the direction: 16S's screen runs at 495 ns/comp natively, against ITS2's
1669-1856. Even degraded, 16S's carry-arm screen (1109) is still faster than
ITS2's baseline. 16S is not being penalised so much as losing an unusually good
operating point. That framing is probably the way in.

Falsified - do not retry

1. Cache residency of the carried map (compmax 29 MB + emax 10 MB, now
   permanently resident). Predicts the bandwidth-bound DP kernel suffers. The DP
   moved +1.4%; only the screen changed. Wrong selectivity.
2. Peak parallel load. Both arms run ~57 of 64 threads during the parallel
   map (busy thread-seconds / map wall). Instantaneous load is identical.
3. Sustained-clock throttling. Duty cycle does change a lot (occupancy
   12-16% -> 31-34% of 64 cores, since the carry deletes 765-869 s of
   single-threaded shuffle). But on the EPYC 7713 the whole range between light
   and all-core load is ~1.35x (base 2.0 GHz, boost 3.72, all-core ~2.7-3.0).
   It cannot arithmetically produce 2.2-2.4x. At most a third of the effect.
4. Anything intrinsic to carrying the map - see the ITS2 control above.

Suggested next step

Not another whole-run A/B - those have now cost four wrong hypotheses. Isolate
the screen: a microbenchmark that streams the 16S k-mer vectors with and without
a ~39 MB scattered-access working set alive alongside, at matched thread counts.
If that does not reproduce it, the effect is not in the screen kernel and the
next place to look is allocator/page behaviour (the rebuilt arm allocates and
frees compmax/emax 9,700 times; the carry allocates once).

Also worth trying first because it is nearly free: rerun the 16S A/B at 24 and
48 threads. Every falsified hypothesis above is thread-count-sensitive in a
different way, so the shape of the curve discriminates between what is left.

Priority

Headroom, not correctness. #139 is byte-identical on 180 files and is a
large win with the give-back already priced in. This is about turning -30% into
~-47% on the largest pooled workload we have.

Data: tmp/new_shuffle/full-pooling-novaseq-16S-carry (16S, 1 rep both arms),
tmp/new_shuffle/full-pooling-novaseq-ITS-carry (ITS2 control, 2 reps).
See docs/findings/shuffle-build-scan.md Result 4.

[cjfields]

Three more controls, from the MiSeq and PacBio A/Bs run for #139. The 16S-only framing is now much better supported.

dataset	pool size	screen (rebuilt)	carry effect
soil 16S	1.23 M raws	495 ns/comp	+124–140%
MiSeq	272 K	1104	−1.2%
ITS2	825 K	1669–1856	−1.9% / +1.7%
PacBio	547 K	3780–3872	+1.3%

Three independent controls spanning 3.4× in screen rate, across both platforms and a 4.5× range in pool size, all indifferent to the carry. Any explanation in which the carry imposes a cost now has to say why three pools do not notice.

Two things this adds:

The predicted 'large working set' mechanism did not pan out either. I expected MiSeq (272 K raws) to be safe and had no strong prediction for PacBio at 547 K — but ITS2 at 825 K is already indifferent, so the boundary is not simply pool size. 16S is 1.5× ITS2, which is a weak basis for a threshold.

The direction is confirmed: 16S is the fast outlier, not the penalised one. Its 495 ns/comp is 2.2× faster than the next-fastest pool, and under the carry it degrades to 1109 — essentially exactly MiSeq's 1104 baseline. It is not being pushed below the field; it is being pulled back into it. Whatever 16S is doing to get 495 is the thing to characterise, and the carry removing it is a symptom rather than the phenomenon.

Suggested reframing of the investigation: instead of asking why does the carry slow 16S's screen, ask what lets 16S's screen run 2.2× faster than any other pool in the first place, then check which of those conditions the carry disturbs. That is a question answerable by profiling one arm, without an A/B.

Data: tmp/new_shuffle/full-pooling-miseq-carry and tmp/new_shuffle/full-pooling-pacbio-carry (2 reps each, both arms).

[cjfields]

Re-measured: the effect is real, smaller than reported, and not screen-specific

Two reps per arm, soil 16S, both reads, exclusive node (compute-5-6), 64 threads,
numactl --interleave=all, one binary at 5b589e7 with the gate flipped by
DADA2RS_SHUFFLE_NO_CARRY=1. 120/120 per-sample outputs byte-identical across
arms. Positive control on the flag itself: shuffle scan split reads
1 builds with the carry on and 9700/11282 builds (97-98% of scanned) with
it off, so both arms demonstrably ran the intended configuration.

screen ns/comp	rep1	rep2	mean	carry penalty
R1 carry	1039	937	988
R1 no-carry	1009	741	875	1.13x (see below)
R2 carry	1196	1143	1170
R2 no-carry	712	707	710	1.65x

R2 is tight -- 0.7% between reps in both arms -- and gives a clean, reproducible
1.65x. R1's no-carry arm swings 36% between reps (1009 vs 741) and the slow rep
is slow across the board (map 390 vs 334s), so I read it as contention rather
than signal; against rep2 the R1 penalty is ~1.3x.

Revised magnitude: 1.3-1.65x, not the 2.2-2.4x this issue was opened on.

Two corrections to the record

1. The 495 ns/comp arm is gone. The original 2.24x was 495 (no-carry) vs
1109 (carry) at build 8d4ca0c. That was a legitimate same-build, same-series
A/B -- an earlier draft of this comment claimed it was a cross-series artifact,
and that was wrong. But no arm in the current code reaches 495: the no-carry
floor is now ~710-740. Something between 8d4ca0c and 5b589e7 raised the
no-carry baseline ~1.45x, which shrinks the ratio without removing the effect.

2. It is not a k-mer screen problem. The whole compare map slows under the
carry, not just the screen:

R2, rep-mean	carry	no-carry	ratio
b_compare map	425s	306s	1.39x
k-mer screen	16939s (thread-sum)	10275s	1.65x

The screen takes a somewhat larger share of the increase, but b_compare as a
whole is more expensive per unit of identical work. The issue title's framing
should be widened accordingly.

It is paid for

run_dada	carry	no-carry
R1	781.5 / 751.6s	1525.3 / 1443.2s	-48%
R2	891.2 / 879.7s	1591.6 / 1597.4s	-45%

The carry costs ~130s of extra compare time and saves ~890s of shuffle. At the
worst screen ratio measured it still wins by roughly sevenfold. (These are
larger than #139's original -28.8 to -32.0% because #143's fold removed ~290s of
serial work from the denominator; the ITS2 and 16S figures in
docs/findings/shuffle-build-scan.md are understated post-fold.)

ITS2 remains a clean control

Two reps per arm, same session: screen +1.5% (R1) / -2.7% (R2), sign flipping
between reads, inside the 3.7% same-config spread. The carry costs ITS2 nothing
at the screen while still returning -15.4 to -15.9% of run_dada. Consistent
with the three indifferent controls in 4409fb1.

Mechanism: still unexplained, and the obvious guess has the wrong sign

The no-carry arm performs ~1e11 comparisons of build scan -- two orders of
magnitude more memory traffic than the carry arm -- and has the faster screen.
Any cache- or residency-based account has to survive that inversion.

Also falsified since this issue opened: 16S doing less work; pool/working-set
size (at one thread the rate is flat, 512.9 / 521.0 / 518.7 ns across a 4.5x
range); thread spacing; and reduction-contention (I predicted ITS2's screen
would move once #143 folded away the serial reduction's six passes -- it did not
move at all).

The ns/comp metric is also confounded independently of any of this: the screen
runs interleaved with the alignments that pass it, and screen_bandwidth shows
that interleaving alone shifts the measured rate with no contention present
(418.9 / 517.5 / 675.3 ns at --align-frac 0.008 / 0.045 / 0.25 at one
thread). See docs/diagnostics.md.

Proposal: retitle and downgrade, do not close

The effect reproduces, so closing it would be wrong. But it is a bounded,
paid-for cost on a -45 to -48% win, not a regression to fix -- and the remaining
serial store in b_compare (#143 step 2, ~27-30% of run_dada) is a much
larger target.

Suggested retitle: "b_compare costs 1.3-1.65x more per comparison under the
#139 carry on soil 16S; unexplained, and paid for many times over".

If it is picked up again, the instrument should be hardware counters (LLC
misses, dTLB misses, stall cycles) across two arms of one binary on one pool.
Every hypothesis above was killed by a timer that could say that but never
why.