Skip to content

Add performance regression CI guardrails - #52

Merged
sergioald merged 1 commit into
mainfrom
ci/performance-regression-guardrails
Oct 5, 2026
Merged

sergioald merged 1 commit into
mainfrom
ci/performance-regression-guardrails

Conversation

@sergioald

Copy link
Copy Markdown
Owner

Summary

Adds dedicated performance-regression CI guardrails for the validated accelerated LDSFL-Meander backends.

The new checks use warmed, repeated, same-runner reference/candidate comparisons rather than absolute whole-solver wall-time thresholds.

No solver equations, production defaults, or scientific/numerical behavior are changed.

What is protected

This PR adds CI protection for:

  • Numba vertical coefficient acceleration
  • Recursive finite-window SEMIANA routing and performance
  • Numba spatial-grid neck detector
  • Exact Numba geometry unwrap
  • Numba kernel configuration, including:
    • fastmath=False
    • parallel=False
    • no Numba disk caching where required
  • Existing optional-dependency and backend-routing behavior

The existing Python 3.12 Numba-equivalence job is retained and extended to include recursive SEMIANA tests.

Performance guardrails

The CI benchmark uses:

  • deterministic inputs
  • Numba warm-up before timing
  • alternating reference/candidate execution order
  • five measured samples
  • median timing as the primary statistic
  • numerical parity checks before accepting timing results
  • machine-readable JSON output
  • one repeat measurement if a timing floor is marginally missed

Current relative performance floors are:

Metric CI floor
Vertical k0123 3×
Recursive SEMIANA, N=1000 2×
Recursive SEMIANA, N=2500 2×
Numba neck grid, no-hit case 2×
Numba neck grid, later-hit case 2×
Geometry unwrap correctness only

Absolute whole-solver wall time is intentionally not used as a blocking gate because repeated 5,000-step profiling showed substantial host/runtime variability.

New CI workflow

Adds:

  • .github/workflows/performance.yml
  • benchmarks/ci_performance_guardrails.py
  • docs/performance_regression_ci.md

The performance workflow runs on:

  • pull requests
  • pushes to main
  • manual workflow dispatch

It uses Ubuntu with Python 3.12 and installs .[dev,numba].

The workflow uploads:

  • performance_guardrails.json

as a diagnostic artifact.

Deterministic hardening

Additional tests now protect:

  • recursive SEMIANA kernel options
  • recursive strategy selection for the public serial Numba path
  • legacy fallback for unsupported SEMIANA execution modes
  • Numba vertical routing into the accelerated coefficient cache
  • Numba neck-grid kernel options
  • explicit verification that required Numba kernels are not disk-cached using kernel.stats.cache_path

The cache check was intentionally implemented using the dispatcher cache state rather than targetoptions, because Numba does not expose the cache=True/False setting reliably through targetoptions.

Local validation

Focused Numba tests:

  • 65 passed

Quality checks:

  • Ruff: passed
  • py_compile: passed
  • git diff --check: passed

Final performance guardrail run:

  • total runtime: 9.56 s
  • all metrics passed
Metric Measured ratio Floor
Vertical k0123 284.67× 3×
SEMIANA N=1000 7.51× 2×
SEMIANA N=2500 27.77× 2×
Neck grid, no hit 59.16× 2×
Neck grid, later hit 50.06× 2×
Geometry unwrap 109.54× correctness only

The SEMIANA and neck guards both reported:

  • cache_enabled: false

and the run would fail if caching were enabled for those required kernels.

Files changed

  • .github/workflows/performance.yml
  • .github/workflows/tests.yml
  • benchmarks/ci_performance_guardrails.py
  • docs/performance_regression_ci.md
  • tests/test_neck_numba.py
  • tests/test_semiana_recursive_numba.py
  • tests/test_vertical_numba.py

Notes

  • No solver-core implementation files are changed.
  • No equations, tolerances, or scientific defaults are changed.
  • No whole-solver timing threshold is introduced.
  • Base-vs-head benchmarking is intentionally deferred to avoid a fragile dual-checkout CI setup.
  • Performance CI is a regression guardrail, not a substitute for scientific validation.

@sergioald
sergioald merged commit 82436a5 into main Oct 5, 2026
5 checks passed
@sergioald
sergioald deleted the ci/performance-regression-guardrails branch October 5, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant