Skip to content

Improve benchmark lifecycle, accuracy gates and B200 profiling - #27

Merged
cfregly merged 19 commits into
mainfrom
codex/b200-followthrough-20260908
Sep 8, 2026
Merged

Improve benchmark lifecycle, accuracy gates and B200 profiling#27
cfregly merged 19 commits into
mainfrom
codex/b200-followthrough-20260908

Conversation

@cfregly

@cfregly cfregly commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Serving examples included engine startup in repeated request measurements, approximate examples lacked consistent independent acceptance gates, and minimal Nsight Compute captures expanded into extra replay work. This change reuses serving engines with explicit lifecycle timings, adds fixed full-output KV/Ozaki accuracy requirements, honors profiler warmup counts, limits minimal captures to five metrics, and adds standalone execution-placement and output-write audits.

Direct B200 and 2×B200 follow-through measured a 1.574× cache-aware result, repeated 1.46–1.47× dual-pool steady-state results, and a 1.279× matched FP8 result at batch 4096. Full outputs and matched traces support those observations. DDP, pipeline, small-batch FP8, dynamic routing, KV, and Ozaki retain their no-win dispositions. Results are portable development evidence on the recorded virtualized host.

Final-head CI passes all four checks: full CPU validation, the four CUDA architecture builds, static analysis, and dashboard checks. The full CPU suite reports 5,529 passed and 527 explicit skips, with zero failures. The benchmark-contract linter checks 936 files with zero errors or warnings. The full-suite budget is 35 minutes; updated regression guards retain the contiguous pipeline capture boundaries and the FP8 default-workload versus batch-4096 distinction.

Target validation also includes 27 passing B200 execution-audit tests with no skips, a real CUDA audit CLI run, 20 arithmetic qualification cases, fresh ordinary KV/Ozaki correctness checks, and the paired benchmark/profile runs. All 645 retained result files match the remote SHA-256 inventory. Nsight Compute 2026.2.1 selected-kernel captures pass; collective/application-range replay still times out and remains an explicit limitation. Shared CUDA, PyTorch, and drivers were not changed.

See the follow-through report for source checkpoints, timings, numerical policies, retained failed attempts, and claim boundaries.

@cfregly
cfregly merged commit 0299307 into main Sep 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant