Investigate ~7% perf regression after mojo 1.0.0b1 upgrade

[msaelices]

Background

Upgrading from mojo 0.26.2 to mojo 1.0.0b1 (#152) introduced a measurable performance regression on the bench_engine.mojo suite, despite all source-level adaptations being either mechanical (fn -> def, len -> byte_length) or documented as zero-overhead (UnsafePointer -> Optional[UnsafePointer] via the None-niche layout).

Measured numbers

Full stable bench_engine comparison, same hardware, back-to-back in one session:

• Geomean: 1.07x slower across 80 benches (W/L/within +/-5%: 20/38/22)
• 28 benches >10% slower, 10 benches 5-10% slower
• 20 benches faster (some by 20-50%)

Persistent regressions (confirmed across re-runs with --filter):

benchmark	baseline (0.26.2)	new (1.0.0b1)	ratio
optimize_extreme_quantifiers	59-71 ns	95-120 ns	1.60-1.69x
toll_free_simple	8.38 us	12.06 us	1.44x
nanpa_findall	5.15 us	7.04 us	1.37x
complex_number	6.72 us	8.95 us	1.33x
dfa_paren_phone	4.07 us	5.37 us	1.32x
sub_whitespace	11.91 us	15.09 us	1.27x
dfa_digits_only	109.97 us	137.51 us	1.25x
flexible_phone	54.32 us	67.36 us	1.24x
optimize_phone_quantifiers	26.12 us	31.97 us	1.22x
dual_quantifiers	17.6-24.4 us	26.8-29.1 us	1.20-1.52x
mixed_range_quantifiers	3.56-4.56 us	5.40-5.85 us	1.28-1.52x

Notable wins (mojo 1.0.0b1 made these faster):

benchmark	baseline	new	ratio
range_lowercase	277 ns	183 ns	0.66x (1.5x faster)
match_all_digits	3.83 us	2.89 us	0.76x
sub_group_phone_fmt	19.95 us	15.40 us	0.77x
range_alphanumeric	557 ns	432 ns	0.78x
flexible_datetime	16.29 us	13.10 us	0.80x
predefined_word	17.49 us	14.07 us	0.80x
simple_phone	27.92 us	22.91 us	0.82x

Single-run noise observed

A --filter=quantifiers re-run flipped several big regressions: ultra_dense_quantifiers went from 1.60x slower to 1.19x faster; optimize_multiple_quantifiers from 1.47x slower to 1.13x faster; grouped_quantifiers from 1.21x slower to 1.25x faster. So the headline 7% geomean is partly inflated by single-run noise. A 3x-median full pass would be the next step before fully trusting the numbers.

Hypothesis

The regression pattern doesn't correlate with files touched by the migration (e.g. range_lowercase is among the biggest wins despite SIMD char-class code being relatively unchanged; optimize_extreme_quantifiers is a hard regression but routes through DFA whose pointer storage just moved to Optional). That points to compiler-level codegen shifts in 1.0.0b1 rather than this PR's adaptations as the dominant cause.

Suggested investigation

1. Run a 3x-median full bench pass on both toolchains to filter single-run noise. The persistent ~25-70% regressions should narrow down to a smaller, more reliable set.
2. For each persistent regression, identify the dispatch path (DFA vs lazy DFA vs OnePass vs backtracking NFA). Pattern: most look NFA-routed (*_phone, *_quantifiers, flexible_phone).
3. Compare LLVM IR / asm output for one or two representative hot loops between 0.26.2 and 1.0.0b1.
4. Confirm by reverting the Optional[UnsafePointer] migration locally on top of 1.0.0b1 and re-benching - if the perf doesn't recover, it's purely compiler.
5. If the regression is in a small set of compiler-sensitive hot loops, try targeted rewrites (e.g. manual hoist + alias annotations) before falling back to filing upstream.

Workaround for users

Anyone on the perf-critical path can stay on mojo-regex 0.13.0 + mojo 0.26.x until this is resolved.

[msaelices]

Attribution analysis (3-median, 3 runs each side)

Repeated the comparison with 3 full bench runs on each of HEAD / revopt / baseline and took the per-bench median. Per-bench coefficient of variation across the 3 runs is ~11% median, ~20% p90, with a few outliers up to ~28%, so single-run pairings can mislead in either direction. The revised numbers:

comparison	geomean ratio	meaning
HEAD (mojo 1.0.0b1 + PR #152) vs baseline (mojo 0.26.2 / main)	1.13	HEAD is ~13% slower
revopt (mojo 1.0.0b1, Optional migration reverted) vs baseline	1.26	revert is ~26% slower
HEAD vs revopt (same compiler)	0.90	HEAD is ~10% faster

Optional[UnsafePointer] migration is not the cause — it's actually a net positive on 1.0.0b1, recovering ~half of the compiler-side regression. With it reverted, the regression deepens from ~13% to ~26%.

The "revopt" branch (revert-optional-for-perf in the worktree, not pushed) was constructed by: checking out the PR HEAD, git revert b6083b6 (the Optional migration commit), and patching the three remaining Bool(self.engine_ptr) / Bool(self._lazy_dfa_ptr) / Bool(self._onepass_ptr) sites with self.engine_ptr.__bool__() calls so it would build under 1.0.0b1 with deprecation warnings instead of hard errors. Everything else - byte_length conversions, the LiteralInfo split, pikevm dangling pointer, fn->def - was left intact, so the comparison isolates only the Optional vs raw-UnsafePointer choice for the matcher / MatchList pointers.

Top persistent regressions vs baseline (3-median, sorted by ratio):

benchmark	baseline	HEAD	ratio	engine
pure_dfa_paren	71 ns	156 ns	2.18x	DFA (literal fast path)
deep_nested_groups_depth4	8.4 ns	14.2 ns	1.70x	(sub-ns, noisy)
optimize_extreme_quantifiers	60 ns	99 ns	1.64x	DFA
dual_quantifiers	21.8 us	32.2 us	1.48x	DFA
single_quantifier_alpha	11.7 us	17.1 us	1.47x	DFA
dfa_digits_only	110 us	159 us	1.45x	DFA
mixed_range_quantifiers	4.6 us	6.6 us	1.43x	DFA
ultra_dense_quantifiers	7.8 us	10.5 us	1.35x	DFA
optimize_multiple_quantifiers	11.4 us	15.3 us	1.34x	DFA
single_quantifier_digits	20.1 us	26.0 us	1.29x	DFA
triple_quantifiers	4.1 us	5.2 us	1.28x	DFA
toll_free_complex	8.2 us	10.3 us	1.26x	DFA
grouped_quantifiers	4.0 us	5.0 us	1.25x	DFA
optimize_phone_quantifiers	24.4 us	30.4 us	1.24x	DFA
nanpa_findall	5.7 us	7.0 us	1.24x	NFA
complex_number	7.2 us	9.0 us	1.24x	DFA

Most of the worst regressions are DFA-routed simple patterns. The DFA hot loop (dfa.mojo match_all + simd_search) hasn't changed semantically: only len(s) -> s.byte_length() (the methods return the same byte count) and engine_ptr[] -> engine_ptr.value()[]. Reverting Optional doesn't recover these (still 1.4-2.2x slower than baseline in the revopt vs baseline comparison), confirming the regression is in mojo 1.0.0b1's codegen of the existing DFA execution code, not in anything PR #152 changed.

Top wins vs baseline:

benchmark	baseline	HEAD	ratio
literal_match_long	700 ns	616 ns	0.88x
range_digits	298 ns	266 ns	0.89x
nanpa_search	28 ns	25 ns	0.90x
range_alphanumeric	540 ns	484 ns	0.90x
national_phone_validation	37.4 us	33.7 us	0.90x
sub_group_phone_fmt	20.0 us	18.0 us	0.90x

So mojo 1.0.0b1 makes some paths faster (mostly NFA-routed and range-class scans) and others much slower (mostly DFA fixed-quantifier paths).

Hypothesis

The DFA inner loops in match_all / simd_search / _try_match_at_position got worse codegen in 1.0.0b1, possibly from:

• different inlining decisions for the chained MatchList.append / simd_search calls
• different scalar tail handling on the two-byte SIMD scan
• different vectorization on the per-position state transition

Worth diffing LLVM IR for DFAEngine.match_all between toolchains on a representative pattern (e.g. pure_dfa_paren) before filing upstream.

Recommendation

• Ship PR Upgrade to mojo 1.0.0b1 #152 as v0.20.0 with the regression caveat in the changelog (already updated).
• File an upstream issue at modular/modular pointing here once the IR diff is captured.
• Users on the perf-critical path stay on v0.13.0 / mojo 0.26.x until the next compiler release.

[msaelices]

Optimization attempt — paused due to bench-environment noise

Started a profile-driven pass on branch perf-debug-1.0.0b1 (forked from upgrade-mojo-1.0.0b1, not pushed since no clean delta to commit). Quick summary so we can resume cleanly:

What was tried

1. Trampoline audit: DFAMatcher.match_all was missing @always_inline (peers is_match/match_first/match_next and NFAMatcher.match_all all have it). Per the mojo-optimizations skill ("every level of the dispatch chain"), added the decorator. Result: too noisy to tell — three full-bench runs spread over 12 minutes showed massive inter-run variance (e.g. pure_dfa_paren ranged 218-602 ns within the same binary), with one filtered re-run showing 27% wins on some benches and 1.5-2.7x losses on others. No conclusion possible.

2. -D ASSERT=none (the 1.0.0b1 release notes call out "Bounds checking is now on by default for all collections on CPU"; this disables it): three full-bench runs came back 2.6x slower than baseline geomean. That's physically impossible — disabling bounds checks can't slow code down, so this is purely measurement contamination.

3. Diagnosis: uptime showed load average 11.7 with chrome at 100% CPU, two more chromes 30-50%, MainThread 55%, slack 25%, spotify 5%, plus this Claude session 47%. The system is heavily loaded; microsecond-scale wall-clock measurements are not reliable in this state.

What to resume with

1. Close browsers/Slack/Spotify; verify cat /proc/loadavg is < 2.0 for at least a few minutes before starting.
2. Re-run a clean 3-median pass on upgrade-mojo-1.0.0b1 vs main to re-establish the actual regression magnitude (the existing ~13% number was from a session that may have also been noisy in retrospect).
3. Test -D ASSERT=none first — if 1.0.0b1's new CPU bounds checking is the cause, this single flag fixes most of the regression. If so, the fix is to set ASSERT=none for release builds in pixi.toml, not source changes.
4. If ASSERT isn't the culprit, the next thing to check is the DFA hot loop in _try_match_at_position (dfa.mojo:1934): states_ptr[current_state].get_transition(Int(text_ptr[pos])). The 1024-byte DFAState.transitions: SIMD[DType.int32, 256] field is huge; whether RegisterPassable actually passes it in registers or via memory may have shifted between 0.26.2 and 1.0.0b1.

Hypotheses ranked by likelihood

• CPU bounds-checking default flip (release notes flag this explicitly) — easy to verify, easy to fix.
• RegisterPassable codegen change on the 1024-byte DFAState — would explain why all DFA-heavy benches are slow but range_lowercase / match_all_digits (which both skip the per-state walk) are faster.
• Optional[UnsafePointer].value() not inlining through the dispatch chain — already partially mitigated by @always_inline on most dispatchers; adding it to DFAMatcher.match_all is a one-line change worth re-testing in a clean environment.

[msaelices]

Root cause found + ~2pp recovered (PR #154)

After re-baselining on a quiet system (the earlier session was contaminated by browser load, see the prior note), the 1.0.0b1 regression on bench_engine is ~7% geomean (3-median, interleaved) rather than the ~13% from the earlier loaded run. Same shape on individual benches: a handful of patterns 40-75% slower, a handful 20-50% faster.

Root cause is the 1.0.0b1 default-on CPU bounds check on stdlib collections. The release notes mention it briefly ("Bounds checking is now on by default for all collections on CPU") but the impact on tight regex hot loops is substantial: every pattern[i] in verify_match runs check_bounds(i, len) per byte; same for the per-candidate position scan in simd_search and the _try_match_simd start-state probe.

Attribution confirmation

-D ASSERT=none (which globally disables bounds checks) selectively recovers the worst regressions:

benchmark	baseline	1.0.0b1 default	1.0.0b1 ASSERT=none
pure_dfa_paren	63 ns	110 ns	57 ns
optimize_extreme_quantifiers	57 ns	86 ns	58 ns
literal_match_short	68 ns	84 ns	64 ns
dfa_digits_only	110 us	120 us	105 us

The recovery is selective — patterns that don't go through verify_match or the per-candidate scan are unaffected by ASSERT=none, which is why the geomean barely budges with the global flag. This pointed directly at the per-callsite bounds-check sites as the culprit, not a uniform compiler regression.

Per-callsite fix → PR #154

Rather than ship the library with ASSERT=none baked in (which would lose the safety net for non-hot code), PR #154 applies unsafe_ptr() at the specific identified hot sites in simd_ops.mojo and dfa.mojo, plus a short-circuit for _try_match_simd calls that are guaranteed to return None on bounded quantifier patterns.

Geomean drops from 1.073 to 1.053 vs baseline. The previously-worst sub-µs benches return to parity or better:

benchmark	baseline	PR #152 head	PR #154
pure_dfa_paren	63 ns	110 ns	69 ns
optimize_extreme_quantifiers	61 ns	86 ns	69 ns
literal_match_short	68 ns	84 ns	56 ns (faster than baseline)

Residual ~5% regression

After PR #154, the remaining gap is on DFA bounded-quantifier patterns (single_quantifier_alpha 1.55x, dual_quantifiers 1.16x, dfa_digits_only 1.13x, etc.). The hot loop for these is _try_match_at_position's DFA state walk:

while pos < text_len:
    var next_state = states_ptr[current_state].get_transition(Int(text_ptr[pos]))
    ...

This already uses unsafe_ptr and has no bounds-check sites. The regression appears compiler-side — likely codegen differences for the states_ptr[...].transitions[...] access pattern (1024-byte RegisterPassable SIMD field). Capturing an LLVM IR diff between 0.26.2 and 1.0.0b1 for this exact loop is the next step if anyone wants to push the remaining 5%.

Documentation

Added a CLAUDE.md note about the new default-on CPU bounds checks so future contributors don't rediscover it from a benchmark regression.