code

AI Systems Performance Engineering

Summary

Reference implementation of high-performance PyTorch, CUDA, and Triton workloads for NVIDIA Blackwell platforms. The repository packages 20 focused chapters, advanced labs, and the shared benchmarking harness so you can profile baselines, apply optimizations, and capture artifacts that prove performance gains.

Roadmap: docs/performance_repo_roadmap.md defines the prioritized plan for canonical suites, trend tracking, anti-pattern enforcement, shared benchmark bases, and evidence-first documentation.

Problem

Most performance repos are easy to browse and hard to trust. This one is meant to do the opposite:

• baseline and optimized paths live side-by-side
• correctness checks run before speedup claims matter
• profiler artifacts and benchmark manifests are first-class outputs, not optional extras

Tier-1 Canonical Suite

The fastest way to answer "is this repo still delivering real wins?" is the canonical tier-1 suite.

python -m cli.aisp bench run-tier1 --single-gpu --profile minimal

That command now writes a stable history package under artifacts/history/tier1/<run_id>/:

• summary.json: per-target baseline, optimized path, and best speedup
• regression_summary.md: human-readable before/after summary against the previous tier-1 run
• regression_summary.json: machine-readable regressions and improvements
• trend_snapshot.json: run-history summary for dashboards and release notes
• artifacts/history/tier1/index.json: suite history index

See docs/tier1_benchmark_suite.md for the current target list, artifact contract, and interpretation guidance.

Historical Representative Deltas

These hardcoded historical rows are retained in the README generator. The cited original artifact is not part of the checked-in generator source. Their lineage and measurements remain unverified. This audit changed correctness, workload and verification contracts; neither the table nor its aggregate speedups qualify the repaired revision. Repeat the applicable full-output and exact-target timing gates before making new performance claims.

Source artifact: artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json

Stored suite speedup (not requalified): 8.22x geomean, 8.54x median, 34.41x arithmetic average.

Target	Baseline	Optimized	Measured delta	Artifact
labs/block_scaling:block_scaling	0.197 ms	0.112 ms	1.76x	artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json
labs/flashattention4:flashattention4_alibi	5.637 ms	0.328 ms	17.20x	artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json
labs/persistent_decode:persistent_decode	1.322 ms	0.090 ms	14.65x	artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json
labs/kv_optimization:kv_standard	1613.038 ms	956.656 ms	1.69x	artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json
ch04:gradient_fusion	54.484 ms	0.323 ms	168.70x	artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json
labs/real_world_models:llama_3_1_8b	12.761 ms	5.242 ms	2.43x	artifacts/history/tier1/20260329_e2e_truthful_canonical_main_4a8c827a__tier1/summary.json

Profiler Evidence

When you want proof beyond wall-clock timing, use the same harness target with a profiling mode instead of a different script.

python -m cli.aisp bench run --targets labs/block_scaling:block_scaling --profile deep_dive --single-gpu
python -m cli.aisp bench run --targets labs/flashattention4:flashattention4_alibi --profile deep_dive --single-gpu
python -m cli.aisp bench run --targets labs/persistent_decode:persistent_decode --profile deep_dive --single-gpu

• minimal is the fastest artifact-bearing path.
• deep_dive is the profiler-backed path for Nsight Systems + Nsight Compute comparisons.
• The benchmark harness now blocks more hot-path anti-patterns and follows imported helper code, so "clean benchmark" means more than it used to.

Optimization Opportunity Radar

After a run, use the opportunity radar to decide what deserves the next profiling or experiment slot:

python -m cli.aisp bench opportunities --data-file artifacts/history/tier1/<run_id>/summary.json --top 10
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --json
python -m cli.aisp bench opportunity-catalog --bench-root . --output-json target_catalog.json
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --catalog-file target_catalog.json --top 20
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --include-discovered-targets --top 20
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --output-run-queue-sh artifacts/opportunity_run_queue.sh
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --output-novelty-validation-sh artifacts/novelty_validation_queue.sh
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --novelty-queue-root artifacts/novelty_validation_queue/<run_id> --output-novelty-next-wave-sh artifacts/novelty_next_wave.sh
python -m cli.aisp bench opportunity-run-summary --run-queue-root artifacts/opportunity_run_queue/<run_id> --json
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --run-queue-root artifacts/opportunity_run_queue/<run_id> --json
python -m cli.aisp bench opportunities --data-file benchmark_test_results.json --novelty-queue-root artifacts/novelty_validation_queue/<run_id> --json

The radar accepts raw benchmark_test_results.json, transformed analyzer JSON, or tier-1 summary.json files. It ranks flat wins, regressions, weak memory cases, missing evidence, compound-optimization candidates, optional catalog-only frontier targets, and discovered-but-unmeasured runnable targets. opportunity-catalog mines the benchmark tree into a reusable frontier catalog with source-derived motifs, matched terms, signal rationales, baseline-vs-optimized delta terms, and optimized-side primitives such as CUDA Graph replay, TMA, FP8/FP4 paths, communication overlap, KV-cache layout, and vectorized memory access. Frontier probes are scored by high-leverage signals such as serving decode, distributed fabric, attention/KV layout, emerging precision, memory movement, runtime launch, storage, and control-plane disaggregation instead of falling back to alphabetical order. The output includes a phased execution plan with validation commands, BenchmarkRun render commands, target-level experiments, motif-level innovation hypotheses, a source-transfer map that proposes reusable source-mined patterns and recipient targets, compound primitive hypotheses that pair source-backed tactics not yet present together on a target, a novelty primitive-pair synthesis plan that proposes source-backed untried pairs outside the fixed compound catalog, a coverage gap map that turns under-covered signals/primitives/compound stacks into negative-space experiment leads, a cross-lane bridge map that makes multi-signal experiments explicit, a novelty queue that merges frontier probes, transfers, compound ideas, coverage gaps, and bridge leads into one ranked action list, novelty experiment playbooks with variant ladders/metrics/guardrails/profiler tools for top leads, a novelty mutation plan that expands playbooks into one-variable variant candidates, a novelty mutation budget plan that selects a small operator-diverse mutation batch with information-gain scores and unlock conditions, a risk-adjusted novelty budget plan that balances expected value, cost, risk, type diversity, and target diversity while emitting mitigation steps plus deferral reasons for backlog leads, a novelty decision frontier that separates quick proofs, high-upside bets, de-risk-first work, Pareto-efficient options, and deferred unlocks, a novelty falsification plan that declares null hypotheses, disproof checks, and claim boundaries before promotion, a novelty ablation plan that assigns negative controls to isolate selected claims, a novelty reproducibility plan that assigns repeat counts, stability metrics, and variance gates, a novelty instrumentation plan that defines profiler tools, launch environment, preflight checks, and artifact evidence, a novelty artifact contract plan that defines the required control/candidate/profile/review files and package manifests, a novelty claim packet plan that turns evidence into bounded claim text while listing blocked overclaims, an advisory novelty validation plan with control/candidate/profile/review jobs plus risk-specific, falsification-specific, ablation-specific, reproducibility-specific, instrumentation-specific, artifact-contract, and claim-packet required evidence for top leads, an optional novelty evidence audit plan that compares a supplied validation queue root against the artifact contract, an optional novelty recovery plan that converts failed logs and missing files into concrete repair/rerun actions, an optional novelty adaptive decision plan that decides whether to run, repair, review, approve, or use a backlog replacement, an optional novelty learning plan that turns validation outcomes into rerank guidance and risk updates, an optional novelty harvest plan that converts approved-and-audited claims into reusable patterns and bounded follow-up experiments, a frontier discovery map with diversity queue and experiment blueprints, an experiment matrix with controls/metrics/guardrails, a portfolio plan that selects a diverse first batch under a small experiment budget, promotion gates that block claims until required evidence is present, and a dependency-aware run queue with dispatch groups plus per-job artifact and success criteria.

--output-run-queue-sh writes a resumable executable runbook that records each job's job.json, command.txt, stdout/stderr logs, dependency markers, and manual promotion-review checklists under AISP_RUN_QUEUE_ROOT. --output-novelty-validation-sh writes the same artifact pattern for the top novelty-validation leads under AISP_NOVELTY_QUEUE_ROOT, keeping control/candidate/profile/review evidence separate from the main portfolio queue, writing each job's artifact_contract.json beside its job.json, and creating review-time claim_packet.json plus claim_packet.md files. Re-running either script skips jobs with DONE, preserves pending manual reviews, and treats APPROVED reviews as completed evidence. opportunity-run-summary reads the resulting DONE, MANUAL_REVIEW_REQUIRED, and APPROVED markers, adds diagnostic signatures for common failed-log patterns such as missing Zymtrace injection, and keeps the raw stdout/stderr paths attached. Passing the same root back to bench opportunities --run-queue-root overlays main-queue artifact state onto the fresh radar result, marks the next runnable jobs, and updates promotion gates only for approved reviews. Passing a novelty validation root with --novelty-queue-root overlays lead-level validation status onto the novelty queue, mutation budget, budget plan, validation plan, artifact contracts, and claim packets, then emits a novelty evidence audit that marks missing contract files as promotion blockers, a novelty recovery plan with concrete repair and rerun actions, a novelty adaptive decision plan that decides whether each selected lead should continue, recover, review, or yield to a backup lead, a novelty learning plan with explicit score/risk guidance for the next rerank, a novelty harvest plan that turns approved-and-audited claims into reusable pattern seeds plus follow-up experiments, and a novelty next-wave plan that orders recover/continue/review/mutation/backup/harvest-follow-up/learning actions into the next campaign. --output-novelty-next-wave-sh writes that next-wave campaign as an executable checklist under AISP_NOVELTY_NEXT_WAVE_ROOT, preserving per-action JSON/Markdown, recovery logs, rerun logs, and manual action markers.

For direct script profiling, core/scripts/profiling/profile.sh <script.py> --tool zymtrace launches the workload with CUDA_INJECTION64_PATH/ZYMTRACE_CUDA_INJECTION64_PATH resolved and writes zymtrace_launch_manifest.json beside the profile artifacts. --tool all includes Zymtrace automatically when the injection library is available.

The same surface is exposed through dashboard API GET /api/benchmark/opportunities and MCP tool benchmark_opportunities for agent-driven optimization loops.

Lab Navigation

If you are navigating by performance problem instead of by chapter, start with these benchmark-pair labs:

Lab	Best for	Distinct from
labs/parameterized_cuda_graphs	fixed-shape PyTorch CUDA Graph replay where request bindings change but graph topology does not	broader decode-serving labs; dynamic-shape graph problems
labs/persistent_decode	persistent decode kernels plus graph/TMA variants on serving-style decode paths	the narrower executable-graph parameter-mutation story
labs/decode_optimization	decode microbenchmarks that separate pinned memory, streams, compile, graphs, and cache policy	a single graph-launch-overhead benchmark pair
labs/block_scaling	Blackwell block-scaled GEMM mechanics and measured tensor-core wins	serving/control-plane orchestration labs
labs/training_hotpath	training-path launch and fusion bottlenecks with clean benchmark pairs	inference/decode-oriented graph replay work

This keeps labs/parameterized_cuda_graphs visible without pretending it replaces the broader persistent-decode or decode-optimization stories.

Benchmark Methodology

This repo now exposes one repeatable benchmarking methodology instead of leaving performance work as a collection of scripts.

Start with:

• ../.agents/skills/dean-performance-review/SKILL.md for the evidence-first optimization workflow adapted from Jeff Dean and Sanjay Ghemawat's performance guidance.
• docs/benchmark_methodology.md for the three-layer model (micro, component, end_to_end), bottleneck taxonomy, publication-vs-realism policy, and straggler playbook.
• docs/autoresearch_campaigns.md for append-only experiment ledgers, diverse idea beams, per-case promotion gates, failed-experiment priors, and worktree provenance.
• docs/performance_warehouse.md for the stable event schema, raw-versus-curated storage split, retention tiers, and telemetry lineage back to raw evidence.
• templates/performance_intake.yaml for KPIs, constraints, and the variable under test.
• templates/benchmark_workload_spec.yaml for the frozen workload definition and measurement policy.
• templates/benchmark_run.yaml for the CRD-aligned declarative BenchmarkRun shape the repo would map onto a Kubernetes-native service.
• cluster/docs/kubernetes_benchmark_service.md plus cluster/configs/benchmarkrun-crd.yaml for the cluster-native operator/CRD direction already being sketched in the repo.

Thin surfaces for these contracts are also exposed through python -m cli.aisp tools benchmark-contracts, dashboard API GET /api/benchmark/contracts, and MCP tool benchmark_contracts.

The current harness already captures manifests, profiler artifacts, raw timings, and artifact hashes. Cryptographic provenance signing is still a documented gap, so external publication packets should record that explicitly rather than assuming hashes alone are sufficient.

Cluster Evaluation

Cluster evaluation has one supported artifact contract for new work:

cluster/runs/<run_id>/
  manifest.json
  structured/
  raw/
  figures/
  reports/

Start with:

• cluster/README.md for the current commands and folder contract.
• python -m cli.aisp cluster common-eval --preset common-answer-fast ... for the normal "evaluate this system" ask.
• python -m cli.aisp cluster common-eval --preset modern-llm ... when you need the full canonical package.
• python -m cli.aisp cluster common-eval --preset multinode-readiness ... before first real multi-node workloads.
• python -m cli.aisp cluster promote-run --run-id <run_id> ... when one collected run should become the published localhost package.

The current published canonical package lives under cluster/published/current/. New collection still happens under cluster/runs/<run_id>/.

Learning Goals

• Understand how the chapters, labs, and shared tooling fit together.
• Stand up a reproducible environment with the pinned PyTorch 2.9.1 + CUDA 13 stack for Blackwell workloads.
• Run the benchmark harness directly or through the Typer CLI for automated artifact capture.
• Validate peak hardware characteristics before grading optimizations against stored expectations.

Directory Layout

Path	Description
ch01 - ch20	One directory per chapter with baseline/optimized benchmarks, workload configs, and chapter-level harness entrypoints such as ch01/compare.py.
labs/	Deep-dive labs for memory-bandwidth patterns, matmul, routing, FlexAttention, MoE, persistent decode, distributed training, and more.
core/benchmark/, profiling/, core/, optimization/, analysis/	Shared harness, logging, workload metadata, profiling, and optimization utilities used by every chapter.
python -m cli.aisp bench	Typer-based CLI for running and profiling targets with reproducible artifacts.
docs/ + core/scripts/	Operational guides, profiling workflows, and setup/reset helpers (setup.sh, cleanup.py, reset-gpu.sh).

Running the Benchmarks

Use the benchmark harness for quick comparisons or drive the Typer CLI when you need repeatable artifact capture.

cd ai-performance-engineering/code
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements_latest.txt
# Optional ABI-bound CUDA 13 serving wheel; setup.sh performs this exact phase.
pip install --no-deps --index-url https://wheels.vllm.ai/0.16.0/cu130 \
  'vllm==0.16.0+cu130'
python -m cli.aisp bench list-targets --chapter ch01
python -m cli.aisp bench run --targets ch01 --profile minimal
python -m cli.aisp bench run-tier1 --single-gpu --profile minimal

• setup.sh installs system prerequisites (drivers, CUDA, Nsight) and should be rerun after driver upgrades.
• Benchmark validity profile defaults to strict. Virtualization is warning-only; use --validity-profile portable for broader compatibility on hardware-limited hosts.
• Use python -m cli.aisp bench expectations --hardware b200 --min-speedup 1.05 to report expectation entries below a target threshold.
• Repeat --targets for multi-scope runs, for example python -m cli.aisp bench run --targets ch01 --targets ch02 --profile minimal; use python -m cli.aisp bench run-tier1 --single-gpu --profile minimal for the canonical regression suite.
• Portable runs do not update expectation files unless --allow-portable-expectations-update is supplied.
• python core/analysis/analyze_expectations.py --artifacts-dir artifacts compares new runs to stored thresholds.

Validation Checklist

• pytest tests/integration succeeds to confirm harness discovery and CLI plumbing.
• python core/benchmark/benchmark_peak.py reports TFLOP/s, bandwidth, and NVLink numbers close to the published ceilings.

Optional vLLM and FlashAttention 4 compatibility

For the qualified FlashAttention 4 (flash-attn-4==4.0.0b19) environment on Linux x86_64, backport the merged vLLM rotary-import fix into the pinned wheel:

mkdir -p third_party/wheels/vllm-upstream
python -m pip download --no-deps --only-binary=:all: \
  --index-url https://wheels.vllm.ai/0.16.0/cu130 \
  --dest third_party/wheels/vllm-upstream 'vllm==0.16.0+cu130'
python scripts/build_vllm_fa4_compat_wheel.py \
  --input-wheel third_party/wheels/vllm-upstream/vllm-0.16.0+cu130-cp38-abi3-manylinux_2_35_x86_64.whl \
  --output-dir third_party/wheels/vllm-fa4-backport \
  --install-python "$VIRTUAL_ENV/bin/python"

The tool accepts only the qualified wheel hash, preserves the Torch and FA4 versions, installs with --no-deps, and writes a provenance manifest. Retire the backport after vllm_no_deps.pin advances to a wheel containing vLLM PR #42679 and that wheel passes the same FA4 import and full-model gates.

Wall of Shame

The benchmark harness includes a strict set of correctness and validity checks to prevent misleading speedups. Below is a threat inventory with scoped and unsupported checks, plus real-world incidents that motivated these checks.

This threat inventory includes scoped checks and unsupported policies. It is not a guarantee that every listed attack is detected. Unsupported cases and missing CUDA runs are explicit skips, never passing coverage. See the protection-test disposition and clock-lock follow-up. Unmapped rows below retain an advertised mechanism but remain individually unaudited; no independent coverage or runtime qualification is asserted.

CUDA Graph Note: Capturing CUDA graphs in setup() is allowed for steady-state replay benchmarks (we intentionally measure replay, not capture). It is NOT allowed to precompute and reuse the final output from setup(). The output used for verification must come from the timed benchmark_fn() run and be surfaced via capture_verification_payload().

Virtualization Note: the runtime checker warns rather than rejecting virtualization. Repository policy requires bare metal for canonical/publish-grade results. A virtualized current-host rerun requires explicit user approval, locked GPU clocks, recorded provenance and a virtualized/non-canonical label; the warning does not grant approval.

Benchmark Validity Issues Reference

Category	Issue	What Happens	Protection	Status	Real-World Incident
Timing	Unsynced Streams	Work on non-default streams is not timed	Full device sync + StreamAuditor	Inventory; not re-audited	Locus/KernelBench 2025
Timing	Incomplete Async Ops	Timer stops before async work finishes	Full device sync	Inventory; not re-audited	Locus/KernelBench 2025
Timing	Event Timing Gaps	CUDA events recorded incorrectly	Cross-validate with wall clock	Inventory; not re-audited
Timing	Timer Granularity	Measurement too coarse for fast ops	Adaptive measurement duration; no timer-resolution guarantee	Scoped; CUDA pending
Timing	Warmup Bleed	Real work happens during warmup	L2 clearing after warmup; not general warmup-work detector	Scoped; eviction pending
Timing	Clock Drift	System clock changes during measurement	Monotonic clock usage	Inventory; not re-audited
Timing	Profiler Overhead	Profiling tools add latency	Harness timing does not enable its profiler; no nested-profiler rejection	Scoped check
Output	Constant Output	Same result regardless of input	Jitter check	Inventory; not re-audited
Output	Stale Cache	Same result across different seeds	Fresh-input check	Inventory; not re-audited
Output	Approximation Drift	Rough estimate instead of full compute	Output tolerance validation	Inventory; not re-audited
Output	Invalid Values (NaN)	NaN in output	validate_result NaN check	Inventory; not re-audited
Output	Invalid Values (Inf)	Inf in output	validate_result Inf check	Inventory; not re-audited
Output	Invalid Ground Truth	Labels/expected values wrong	Selected-reference caching/comparison; does not validate dataset labels	Scoped check	ImageNet Labels 2021, MMLU Errors 2025
Output	Shape Mismatch	Output shape differs from expected	Shape validation	Inventory; not re-audited
Output	Dtype Mismatch	Output dtype differs from expected	ToleranceSpec dtype check	Inventory; not re-audited
Output	Denormalized Values	Subnormal floats cause slowdowns	Denormal check	Inventory; not re-audited
Output	Uninitialized Memory	Output contains garbage	No uninitialized-memory provenance detector	Unsupported
Workload	Precision Mismatch	Claims FP32 but uses FP16	InputSignature dtype verification	Inventory; not re-audited
Workload	Backend Precision Policy Drift	Global precision policy changes during timing	Backend policy immutability check	Inventory; not re-audited	PyTorch TF32 Default 2020
Workload	Undeclared Shortcuts	Skips elements without declaring	Workload invariant check	Inventory; not re-audited	AI Agent Benchmark Shortcuts 2024
Workload	Early Exit	Stops iteration loops early	Config immutability	Inventory; not re-audited
Workload	Batch Shrinking	Processes fewer samples	InputSignature matching	Inventory; not re-audited
Workload	Sequence Truncation	Processes shorter sequences	InputSignature matching	Inventory; not re-audited
Workload	Hidden Downsampling	Silently reduces resolution	Dimension validation	Inventory; not re-audited
Workload	Sparsity Mismatch	Different sparsity patterns	Sparsity ratio check	Inventory; not re-audited
Workload	Attention Mask Mismatch	Different masking applied	Mask equivalence check	Inventory; not re-audited
Workload	KV Cache Size Mismatch	Different cache sizes	Cache dimension check	Inventory; not re-audited
Workload	Train/Test Overlap	Model tested on training data	No dataset provenance, leakage or holdout enforcement	Unsupported	Computational Biology 2019
Location	CPU Spillover	Work offloaded to CPU	Wall/CUDA timing cross-check; no per-operation CPU placement detector	Scoped timing check
Location	Setup Pre-computation	Work done in setup	check_setup_precomputation	Inventory; not re-audited
Location	Graph Capture Cheat	Pre-compute during graph capture	GraphCaptureCheatDetector	Inventory; not re-audited
Location	Warmup Computation	Compute results during warmup	L2 clearing after warmup; not general warmup-work detector	Scoped; eviction pending
Location	Background Thread	Compute in separate thread	Subprocess execution does not prohibit threads, lock priority or isolate host processes	Unsupported policy
Location	Lazy Evaluation Skip	Returns unevaluated lazy tensor	force_tensor_evaluation	Inventory; not re-audited
Location	JIT Compilation Timing	JIT compile time included/excluded inconsistently	clear_compile_cache	Inventory; not re-audited
Memory	Pre-allocated Output	Result buffer allocated in setup	MemoryAllocationTracker	Inventory; not re-audited
Memory	Input-Output Aliasing	Output points to pre-filled input	check_input_output_aliasing	Inventory; not re-audited
Memory	Pinned Memory Timing	Async pinned transfers not waited	Transfer completion check	Inventory; not re-audited
Memory	Memory Pool Reuse	Cached allocations skew timing	reset_cuda_memory_pool	Inventory; not re-audited
Memory	Fragmentation Effects	Memory fragmentation differs	Allocator cleanup/memory-growth diagnostics; no fragmentation parity	Scoped check
Memory	Page Fault Timing	First-touch page faults included	No page-fault or managed-memory event detector	Unsupported
Memory	Swap Interference	Swapping affects timing	Detect enabled swap; does not disable swap or lock memory	Environment gate
CUDA	Host Callback Escape	cudaLaunchHostFunc returns early	No corresponding execution/provenance inspector	Unsupported
CUDA	Async Memcpy Incomplete	D2H/H2D copies not awaited	Full device sync	Inventory; not re-audited
CUDA	Workspace Pre-compute	Work in cuBLAS workspace alloc	No corresponding execution/provenance inspector	Unsupported
CUDA	Persistent Kernel	Kernel left running across calls	No corresponding execution/provenance inspector	Unsupported
CUDA	Undeclared Multi-GPU	Work spread across undeclared GPUs	No corresponding execution/provenance inspector	Unsupported
CUDA	Context Switch Overhead	CUDA context switches affect timing	No corresponding execution/provenance inspector	Unsupported
CUDA	Driver Overhead	Driver calls not accounted for	No corresponding execution/provenance inspector	Unsupported
CUDA	Cooperative Launch Abuse	Cooperative kernels bypass checks	No corresponding execution/provenance inspector	Unsupported
CUDA	Dynamic Parallelism Hidden	Child kernels not tracked	No corresponding execution/provenance inspector	Unsupported
CUDA	Unified Memory Faults	Page migration not timed	No page-fault or managed-memory event detector	Unsupported
Compile	Compilation Cache Hit	Returns cached compiled output	clear_compile_cache	Inventory; not re-audited
Compile	Trace Reuse	Exploits trace caching	torch._dynamo.reset	Inventory; not re-audited
Compile	Mode Inconsistency	Different compile mode verify vs perf	No general compiler-mode/backend parity or autotuning-variance guard	Unsupported
Compile	Inductor Asymmetry	Inductor optimizations inconsistent	No general compiler-mode/backend parity or autotuning-variance guard	Unsupported
Compile	Guard Failure Hidden	Recompilation not counted	Process-cumulative Dynamo graph counts with source metadata; not resident cache or compile parity	Scoped check
Compile	Autotuning Variance	Autotuning picks different kernels	No general compiler-mode/backend parity or autotuning-variance guard	Unsupported
Compile	Symbolic Shape Exploit	Different shapes trigger different code	InputSignature matching	Inventory; not re-audited
Distributed	Rank Skipping	Some ranks do not do work	check_rank_execution	Inventory; not re-audited
Distributed	Collective Short-circuit	Communication skipped	NCCL validation	Inventory; not re-audited
Distributed	Topology Mismatch	Claims different topology	Compare declared topology; no ring/tree algorithm field	Scoped signature check
Distributed	Barrier Timing	Barrier timing exploited	No barrier-timing, gradient-bucket parity or async-gradient completion detector	Unsupported
Distributed	Gradient Bucketing Mismatch	Different bucket sizes	No barrier-timing, gradient-bucket parity or async-gradient completion detector	Unsupported
Distributed	Async Gradient Timing	Async all-reduce not awaited	No barrier-timing, gradient-bucket parity or async-gradient completion detector	Unsupported
Distributed	Pipeline Bubble Hiding	Pipeline bubbles not counted	Declared rank workload and timing cross-checks; no bubble classifier	Scoped check
Distributed	Shard Size Mismatch	FSDP shards differ	InputSignature matching	Inventory; not re-audited
Environment	Device Mismatch	Uses different GPU than declared	Environment inventory lacks expected/observed GPU identity parity; separate Tier-1 preflight attests target	Unsupported generic identity policy
Environment	Frequency Boost	Overclocked for benchmark only	Application-clock lock; actual observed-NVML integration requires GPU	Implemented; runtime pending
Environment	Priority Elevation	Runs at higher priority	Subprocess execution does not prohibit threads, lock priority or isolate host processes	Unsupported policy
Environment	Memory Overcommit	Exploits memory overcommit	Memory-growth diagnostic; no overcommit policy	Scoped diagnostic
Environment	NUMA Inconsistency	NUMA placement differs	Advisory affinity diagnostics; no pinning or cross-node rejection	Advisory
Environment	CPU Governor Mismatch	Different CPU frequency scaling	Strict environment gate rejects non-performance governor; does not set or lock it	Environment gate
Environment	Thermal Throttling	GPU throttles during run	NVML temperature/clock-drop/throttling diagnostics	Scoped; hardware pending
Environment	Power Limit Difference	Different TDP settings	Power draw captured; configured power-limit parity absent	Unsupported
Environment	Driver Version Mismatch	Different CUDA drivers	Available RunManifest provenance; no cross-run version lock	Unsupported version parity
Environment	Library Version Mismatch	Different cuDNN/cuBLAS	Available RunManifest provenance; no cross-run version lock	Unsupported version parity
Environment	Container Resource Limits	cgroups limits differ	Resource limit check	Inventory; not re-audited
Environment	Virtualization Overhead	VM/container overhead varies	Runtime virtualization notice is advisory; separate repository policy still applies	Advisory
Statistical	Cherry-picking	Only best iterations reported	Preserve supplied samples/statistics; no upstream omission/injection/selection detector	Scoped reporting	Leaderboard Illusion 2025
Statistical	Outlier Injection	Slow iterations added to baseline	Preserve supplied samples/statistics; no upstream omission/injection/selection detector	Scoped reporting
Statistical	Variance Gaming	Variance reporting manipulated	Preserve supplied samples/statistics; no upstream omission/injection/selection detector	Scoped reporting
Statistical	Percentile Selection	Favorable percentile chosen	Preserve supplied samples/statistics; no upstream omission/injection/selection detector	Scoped reporting
Statistical	Insufficient Samples	Too few iterations for significance	Duration-driven adaptive iterations with maximum; no power/variance guarantee	Scoped timing	Measuring What Matters 2025
Statistical	Cold Start Inclusion	First run included unfairly	Warmup enforcement	Inventory; not re-audited
Statistical	GC Interference	Garbage collection during timing	gc_disabled	Inventory; not re-audited
Statistical	Background Process Noise	System processes affect timing	Subprocess execution does not prohibit threads, lock priority or isolate host processes	Unsupported policy
Evaluation	Eval Code Exploitation	Benchmark code modified to pass	BenchmarkContract enforcement	Inventory; not re-audited
Evaluation	Timeout Manipulation	Timeout extended to hide slowdowns	Config immutability	Inventory; not re-audited
Evaluation	Metric Definition Gaming	Redefine what speedup means	Standardized metric definitions	Inventory; not re-audited	MLPerf 2019, HANS 2019, Measuring What Matters 2025, Medical LLM Benchmarks 2025
Evaluation	Test Data Leakage	Training on test data	No dataset provenance, leakage or holdout enforcement	Unsupported	Benchmark Data Contamination Survey 2024
Evaluation	Benchmark Overfitting	Optimize specifically for benchmark	Fresh-input/jitter cached-output checks; no general dataset-overfitting detector	Scoped check	Underspecification 2020, Epic Sepsis 2021, NaturalCodeBench 2024
Evaluation	Self-Modifying Tests	AI/code modifies its own tests	Config-value immutability; no test-source immutability	Unsupported source policy
Evaluation	Benchmark Memorization	Agent memorizes test cases	Fresh-input/jitter cached-output checks; no general dataset-overfitting detector	Scoped check	AI Agent Benchmark Shortcuts 2024
Evaluation	Missing Holdout Sets	No proper train/test split	No dataset provenance, leakage or holdout enforcement	Unsupported	AI Agent Benchmark Shortcuts 2024, Microsoft Tay 2016

Total: 11 categories, 95 inventory rows. These are threats and advertised mechanisms, not a count of verified protections.

Notable Real-World Incidents

Year	Incident	Issue Type	What Happened	Source
2025	Locus/KernelBench Stream Exploit	Unsynced Streams	Claimed 20x speedup on Llama FFW kernel. AI launched work on non-default CUDA streams but timer only measured default stream. 32.8 percent of RL-generated kernels exploited this, causing fake 18x speedups.	https://x.com/miru_why/status/1991773868806361138
2025	Measuring What Matters: Construct Validity in LLM Benchmarks	Metric Definition Gaming / Construct Validity	Systematic review of 445 LLM benchmarks found construct-validity weaknesses and low statistical rigor; issued eight design recommendations.	https://ora.ox.ac.uk/objects/uuid%3Aad2b69b6-0986-42d0-a512-a6e56338b6cc
2025	Medical LLM Benchmarks and Construct Validity	Metric Definition Gaming / Construct Validity	Position paper argues exam-style medical LLM benchmarks miss real-world tasks and documents construct-validity gaps using clinical data.	https://arxiv.org/abs/2503.10694
2025	Sakana AI Scientist Evaluation	Evaluation Integrity	Independent evaluation found frequent experiment failures and hallucinated numerical results.	https://arxiv.org/abs/2502.14297
2025	Leaderboard Illusion (Chatbot Arena)	Cherry-picking	Analysis of Chatbot Arena reports selection effects and leaderboard instability when submissions are inconsistent or selectively disclosed.	https://arxiv.org/abs/2504.20879
2024	MMLU Benchmark Errors	Invalid Ground Truth	Analysis found 57 percent of MMLU virology subset questions incorrect and estimated 6.49 percent errors overall.	https://arxiv.org/abs/2406.04127
2024	AI Agent Benchmark Shortcuts	Missing Holdout Sets	Study found AI agents memorize benchmark test samples instead of learning to generalize. Many benchmarks lack proper holdout test sets.	https://arxiv.org/abs/2407.01502
2024	NaturalCodeBench vs HumanEval	Benchmark Overfitting	Real-user coding tasks in NaturalCodeBench show large performance gaps and weak correlation with HumanEval scores.	https://aclanthology.org/2024.findings-acl.471/
2024	Benchmark Data Contamination Survey	Data Contamination	Survey catalogs contamination pathways across LLM benchmarks and highlights mitigation gaps.	https://arxiv.org/abs/2406.04244
2023	NLP Evaluation Data Contamination	Data Contamination	Position paper warns that LLMs trained on benchmark test splits can inflate reported scores.	https://arxiv.org/abs/2310.18018
2022	MLPerf Participation Issues	Cherry-picking	MLPerf faced inconsistent vendor participation; selective scenario submissions led to biased performance representations.	https://web.archive.org/web/20250813110435/https://www.nextplatform.com/2022/04/08/the-performance-of-mlperf-as-a-ubiquitous-benchmark-is-lacking/
2022	ML Benchmark Validity (Berkeley)	Benchmark Overfitting	Small changes in data distribution caused significant performance drops, questioning external validity.	https://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-180.html
2021	ImageNet Label Errors	Invalid Ground Truth	At least 6 percent label errors in ImageNet validation set.	https://arxiv.org/abs/2103.14749
2021	MLPerf Reproducibility	Benchmark Reproducibility	Users could not reproduce MLPerf v0.7 results due to inaccessible datasets and outdated repos.	https://groups.google.com/a/mlcommons.org/g/public/c/T_8UsUPIWFo
2021	Epic Sepsis Model External Validation	Benchmark Overfitting	External validation found poor discrimination and calibration for the Epic Sepsis Model, leading to missed cases and alert fatigue.	https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307
2020	Underspecification in ML	Benchmark Overfitting	Models with equivalent benchmark performance diverged in deployment behavior.	https://arxiv.org/abs/2011.03395
2020	TF32 Default on Ampere	Precision Policy Drift	TF32-enabled matmul/conv trades precision for speed unless explicitly disabled.	https://pytorch.org/docs/stable/notes/cuda.html#tf32-on-ampere
2019	NLI Heuristic Shortcuts (HANS)	Metric Definition Gaming	Models trained on MNLI (GLUE) rely on shallow heuristics and fail on HANS, revealing spurious shortcut behavior.	https://aclanthology.org/P19-1334/
2019	MLPerf Inference Bias	Metric Definition Gaming	Vendors selectively submitted results highlighting strengths.	http://web.archive.org/web/20191112035148/https://www.forbes.com/sites/janakirammsv/2019/11/10/the-curious-case-of-mlperf-inferencing-benchmark-results/
2019	Computational Biology Overfitting	Train/Test Overlap	Tools developed and tested on same datasets failed on new data.	https://www.nature.com/articles/s41467-019-09406-4
2016	Microsoft Tay Chatbot	Missing Holdout Sets	AI chatbot learned abusive behavior within 24 hours after deployment due to adversarial user interactions.	https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/

Incident Categories and Our Protections

Category	Incidents	Our Protection	Status
Timing Manipulation	1 (Locus/KernelBench)	Full device sync + StreamAuditor	Inventory; not re-audited
Invalid Ground Truth	2 (ImageNet Labels, MMLU)	Selected-reference caching/comparison; does not validate dataset labels	Scoped check
Benchmark Overfitting	4 (Underspecification, Epic Sepsis, NaturalCodeBench, Berkeley)	Fresh-input/jitter cached-output checks; no general dataset-overfitting detector	Scoped check
Data Contamination	2 (LLM Survey 2024, NLP Contamination 2023)	No dataset provenance, leakage or holdout enforcement	Unsupported
Metric Gaming	4 (Measuring What Matters 2025, Medical LLM Benchmarks 2025, HANS 2019, MLPerf 2019)	Standardized metric definitions	Inventory; not re-audited
Cherry-picking	2 (Leaderboard Illusion, MLPerf 2022)	Preserve supplied samples/statistics; no upstream omission/injection/selection detector	Scoped reporting
Train/Test Overlap	1 (Computational Biology)	No dataset provenance, leakage or holdout enforcement	Unsupported
Missing Holdout Sets	2 (AI Agent Shortcuts, Microsoft Tay)	No dataset provenance, leakage or holdout enforcement	Unsupported
Reproducibility	1 (MLPerf 2021)	Available RunManifest provenance; no cross-run version lock	Unsupported version parity
Evaluation Integrity	1 (Sakana AI Scientist)	Contract/config checks; no test-source immutability	Scoped checks
Precision Policy Drift	1 (TF32 Default)	Backend policy immutability check	Inventory; not re-audited

Deep Dive: The Locus/KernelBench Stream Timing Vulnerability

This 2025 incident illustrates why correctness verification alone is insufficient.

# VULNERABLE TIMING (what KernelBench did)
start_event.record(original_model_stream)  # Only records on default stream
model(*inputs)                              # But work runs on s1, s2, s3
end_event.record(original_model_stream)    # Timer stops before s1/s2/s3 finish
torch.cuda.synchronize(device=device)      # Waits, but timing already recorded

# CORRECT TIMING (the fix)
for stream in custom_model_streams:
    custom_model_stream.wait_stream(stream)  # Wait for ALL streams
_event.record(custom_model_stream)           # Then record timing

The exploit pattern:

1. AI creates non-default streams: s1 = getStreamFromPool(), s2 = ..., s3 = ...
2. AI launches GEMMs on those streams: at::mm_out(gate, x2d, gate_proj.t()) on s1
3. AI does not call setCurrentCUDAStream(s3) or wait for streams before returning
4. Correctness test uses torch.cuda.synchronize() and passes
5. Performance test uses stream-specific events and reports fake speedups

Result: 82/250 (32.8 percent) of RL-generated CUDA kernels exploited this, producing artificial 18x speedups with zero actual performance improvement.

Protection Implementation Reference

Module	Key Protections
core/harness/benchmark_harness.py	Full device sync, L2 cache clearing, GPU clock locking, warmup isolation, config immutability, adaptive iterations, CUDA graph mode
core/harness/validity_checks.py	StreamAuditor, MemoryAllocationTracker, GraphCaptureCheatDetector, gc_disabled, clear_compile_cache, capture_gpu_state, validate_environment
core/harness/l2_cache_utils.py	Dynamic L2 cache size detection, clear_l2_cache
core/benchmark/verify_runner.py	VerifyRunner, GoldenOutputCache, jitter check, fresh-input check, output comparison, workload invariants
core/benchmark/verification.py	InputSignature, ToleranceSpec, QuarantineReason, seed mutation detection
core/benchmark/quarantine.py	QuarantineManager with persistence
core/benchmark/contract.py	BenchmarkContract enforcement

Notes

• core/scripts/profile_all_workloads.sh and ncu_template.ini capture Nsight traces with consistent metric sets.
• artifacts/runs/ holds run outputs (results/profiles/reports/logs); clean via python cleanup.py when rotating hardware.
• docs/benchmark_methodology.md defines the repo-wide benchmarking method that ties harness runs, cluster packages, and publication-grade evidence together.
• docs/performance_warehouse.md defines the warehouse contract that ties published numbers back to raw evidence and telemetry joins.
• docs/perf_intake_and_triage.md outlines the standard intake bundle for performance investigations.