|
| 1 | +# Inline CUDA: persistence saves 19–28% of evaluation wall time |
| 2 | + |
| 3 | +Measured on 2026-09-04 using KernelBot's bundled inline-CUDA FP16 vector-add |
| 4 | +example, three benchmark shapes (1024², 2048², 4096²), and cold private build |
| 5 | +caches for every submission. GPU measurements remain sequential. |
| 6 | + |
| 7 | +## Turnaround |
| 8 | + |
| 9 | +Each batch contains four submissions and includes worker startup/teardown. |
| 10 | +Compilation and correctness checks are included; Modal image/container startup |
| 11 | +and the once-per-experiment machine probe are excluded. |
| 12 | + |
| 13 | +| Experiment | Fresh processes | Persistent worker | Less wall time | |
| 14 | +|---|---:|---:|---:| |
| 15 | +| Initial allocation, median of two counterbalanced rounds | 173.8 s | 125.4 s | 27.8% | |
| 16 | +| Complete repeat on another allocation, one fresh/persistent round | 151.1 s | 122.2 s | 19.1% | |
| 17 | + |
| 18 | +That is approximately **29–48 seconds saved per four-submission batch** on this |
| 19 | +workload. Each comparison uses the same assigned GPU and CPU resources for both |
| 20 | +modes. Absolute times vary across the two allocations; do not pool those times |
| 21 | +into a single baseline. This small experiment is not a production workload |
| 22 | +estimate or a confidence interval. |
| 23 | + |
| 24 | +## What still takes time |
| 25 | + |
| 26 | +Per-request phase medians from the complete repeat: |
| 27 | + |
| 28 | +| Phase | Fresh processes | Persistent worker | |
| 29 | +|---|---:|---:| |
| 30 | +| Compile and import submission | 31.74 s | 29.85 s | |
| 31 | +| Benchmark request, including validation and setup | 5.89 s | 0.41 s | |
| 32 | + |
| 33 | +Keeping Python/CUDA alive removes most benchmark-request startup. Rebuilding |
| 34 | +C++/CUDA dominates the remaining time. Every inline request produced its own |
| 35 | +native library; persistent requests used `add_cuda.so`, then `add_cuda_v1.so`, |
| 36 | +`add_cuda_v2.so`, and `add_cuda_v3.so` in separate cold build directories. |
| 37 | +The phase medians are individual-request statistics, not additive batch totals. |
| 38 | + |
| 39 | +## Kernel scores are not yet equivalent |
| 40 | + |
| 41 | +The same correctness and CUDA-event timing function bodies produced different |
| 42 | +reported kernel times. Below are medians of each submission's reported mean in |
| 43 | +the complete repeat. These are GPU microseconds, distinct from evaluation wall |
| 44 | +seconds above. |
| 45 | + |
| 46 | +| Shape | Fresh, µs | Persistent, µs | Persistent change | |
| 47 | +|---|---:|---:|---:| |
| 48 | +| 1024² | 36.02 | 40.28 | +11.8% | |
| 49 | +| 2048² | 108.37 | 123.65 | +14.1% | |
| 50 | +| 4096² | 407.96 | 468.47 | +14.8% | |
| 51 | + |
| 52 | +These 12–15% shifts require investigation before using the prototype for ranked |
| 53 | +scores. This one ordered repeat does not separate process-state effects from |
| 54 | +clock/thermal drift or measurement variability; GPU clocks were not pinned. |
| 55 | +There is no claim that the kernel itself became faster or that the two execution |
| 56 | +modes are score-equivalent. |
| 57 | + |
| 58 | +## Correctness and code replacement |
| 59 | + |
| 60 | +All 24 timed submissions passed across the two experiments. The complete repeat |
| 61 | +also ran **correct addition → incorrect subtraction → correct addition** in one |
| 62 | +persistent worker, retaining the requested extension name `add_cuda`. Observed |
| 63 | +results were **pass → fail → pass**. The incorrect variant compiled successfully |
| 64 | +and failed with numerical mismatches on all three shapes; this was not a compiler |
| 65 | +or import failure. Restored code produced correct results again. |
| 66 | + |
| 67 | +The worker still does not isolate arbitrary user code, unload native modules, |
| 68 | +implement production request timeouts, or recover from a poisoned CUDA context. |
| 69 | +This PR remains an experiment with no production runner changes. |
| 70 | + |
| 71 | +## Evidence and reproduction |
| 72 | + |
| 73 | +- [Initial timing observations](results/inline-initial-timings.json): two complete |
| 74 | + rounds, recovered from provider logs after a local DNS/connection failure. The |
| 75 | + final reload check was interrupted. Full phase/kernel records were not exported, |
| 76 | + so this file explicitly marks its limited detail. |
| 77 | +- [Complete repeat](results/inline-repeat.json): all per-submission/phase timings, |
| 78 | + correctness output, kernel statistics, native-library names/hashes, and a hash |
| 79 | + of the full local result. Compiler logs are omitted from this compact artifact. |
| 80 | +- [Repeat manifest](results/inline-repeat-manifest.json): runtime/source hashes, |
| 81 | + pinned image, resources, and stopped sandbox metadata. The uploaded source |
| 82 | + hashes match the PR runtime files at commit `4de31428`. |
| 83 | +- [Run instructions](README.md): reproducible fresh/persistent comparison and |
| 84 | + correct/incorrect/correct check. The default runs two counterbalanced rounds; |
| 85 | + this complete repeat used `--rounds 1` after the original two-round matrix. |
| 86 | + |
| 87 | +The complete repeat used one Tesla T4, eight reserved CPU cores, 32 GiB RAM, |
| 88 | +CUDA 12.8.1 compiler image, Torch 2.7.1, and `MAX_JOBS=2`. It took 406 seconds |
| 89 | +including local launch/admission overhead, with an estimated $0.067 GPU-only |
| 90 | +cost at $0.000164/T4-second; CPU/RAM are excluded. Both experiment sandboxes are |
| 91 | +stopped. No production deployment, submissions, or database writes were made. |
| 92 | + |
| 93 | +Validation: repository-wide Ruff and three CPU regression tests pass. The |
| 94 | +summarizer additionally audits the actual GPU result, requiring valid timing |
| 95 | +statistics, successful compilation, native-library artifacts, and numerical |
| 96 | +mismatch evidence for the negative reload test. |
0 commit comments