Skip to content

Commit f3295bb

Browse files
authored
Merge pull request #172 from gpu-mode/docs/hosted-ncu-setup
Document hosted NCU profiling without provider setup
2 parents 2d3899c + 095fa2e commit f3295bb

1 file changed

Lines changed: 22 additions & 26 deletions

File tree

‎docs/ncu-profiling.md‎

Lines changed: 22 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,15 @@ at least one representative entry under `benchmarks`. See the
2626
[QR v2 task definition](../problems/linalg/qr_v2/task.yml).
2727

2828
KernelBot profiles `benchmarks`, not `tests`: it launches a separate evaluator
29-
run for each benchmark entry. With the Modal profiling command, `--benchmark-index N`
29+
run for each benchmark entry. With the profiling command, `--benchmark-index N`
3030
selects the zero-based `benchmarks[N]`; omitting it profiles every benchmark
3131
entry. The evaluator reads the benchmark specification file supplied by the
3232
runner. Do not hardcode a shape or import a separate shape list for profiling.
3333

3434
Register the problem's leaderboard name, directory, and supported GPU names in
3535
its competition YAML (for example, [linalg.yaml](../problems/linalg.yaml)).
36-
The CLI resolves that mapping to find the task and rejects unsupported GPUs.
36+
KernelBot syncs that mapping into its leaderboard configuration and rejects
37+
unsupported GPUs.
3738

3839
## Mark the submitted work
3940

@@ -79,7 +80,7 @@ inputs, and synchronize before entering the range.
7980

8081
If the evaluator uses worker processes, define the worker at module scope and
8182
use the existing pool. The runner must use `--target-processes all` to capture
82-
kernels launched in those children; the Modal CLI runner already does this.
83+
kernels launched in those children; the hosted NCU runner already does this.
8384

8485
## Dispatch and report profile results
8586

@@ -127,14 +128,14 @@ resources. An NCU-only profile path, like QR v2's, can call its worker directly.
127128

128129
Use a popcorn-cli build that exposes `submit --profile`; check `popcorn submit
129130
--help` or see the [CLI profiling guide](https://github.com/gpu-mode/popcorn-cli/blob/main/docs/profiling.md).
130-
`--profile` runs in your own Modal account. `--profile-brev` explicitly selects
131-
the separate Brev service; there is no automatic provider switch on failure.
131+
`--profile` submits through the normal authenticated GPU Mode API. Users do
132+
not need a compute-provider SDK, account, or token. `--profile-brev` explicitly
133+
selects the separate Brev service; there is no automatic provider switch on failure.
132134

133-
Install and authenticate Modal, then profile a known-correct starter submission:
135+
Register with Popcorn, then profile a known-correct starter submission:
134136

135137
```bash
136-
pip install modal
137-
modal setup
138+
popcorn register discord
138139
popcorn submit submission.py --leaderboard YOUR_LEADERBOARD --gpu B200 \
139140
--profile --benchmark-index 0
140141
```
@@ -164,31 +165,26 @@ popcorn submit submission.py --leaderboard YOUR_LEADERBOARD --gpu B200 \
164165

165166
A filter that matches no launched kernels cannot produce a useful profile.
166167
`--set full` requests a broad metric set but does not guarantee that every
167-
metric exists or has data on every GPU. The Modal runner leaves GPU clocks
168+
metric exists or has data on every GPU. The hosted NCU runner leaves GPU clocks
168169
unchanged, so use the normal benchmark path for latency comparisons.
169170

170171
## Record source provenance
171172

172-
The profiler runs the evaluator from its active reference-kernels checkout.
173-
Editing a local `eval.py` does not change a remote run. The Modal CLI resolves
174-
current repository revisions when launched and saves the problem directory,
175-
selected shapes, capture options, and source refs in `manifest.json`.
176-
To validate a public problem branch before merging, push it and select its
177-
full commit SHA explicitly:
173+
The hosted profiler runs the task/evaluator synced into KernelBot's leaderboard
174+
configuration. Editing a local `eval.py` or setting a client-side repository ref
175+
does not update that configuration. Publish the problem revision and have an
176+
operator sync it through the normal KernelBot problem-update workflow before
177+
validating the hosted profile.
178178

179-
```bash
180-
export POPCORN_REFERENCE_KERNELS_REF=FULL_COMMIT_SHA
181-
popcorn submit submission.py --leaderboard YOUR_LEADERBOARD --gpu B200 \
182-
--profile --benchmark-index 0
183-
```
184-
185-
This override selects a ref in `gpu-mode/reference-kernels`; it does not upload
186-
a local working tree or select an arbitrary fork. Brev uses its own deployed
187-
checkout, so the Modal override does not update that service.
179+
The CLI saves the leaderboard, GPU, selected shapes, capture options, and a
180+
SHA-256 digest of the evaluation configuration in `manifest.json`. This digest
181+
identifies evaluated content, not a repository commit. Record the synced
182+
reference-kernels commit separately in the problem PR or operator validation
183+
record. Brev has its own deployed checkout and must be refreshed separately.
188184

189185
In the problem PR, record the resolved problem directory, reference-kernels
190186
commit, GPU, benchmark indices tested, kernel filter/launch limit, and observed
191-
artifacts. For example, the existing Modal integration was verified against
187+
artifacts. For example, the NCU GPU integration was verified against
192188
`problems/linalg/qr_v2`, benchmark 0, on B200 at reference-kernels
193189
`51e22db671d36c1c76091c43c36a44546ba324a1`. That is evidence for that run, not a
194190
claim that every problem in this repository already supports NCU.
@@ -201,5 +197,5 @@ claim that every problem in this repository already supports NCU.
201197
| No kernels captured | Check the exact NVTX push/pop range name, matching kernel filter, and child-process capture. |
202198
| Only setup kernels appear | Move generation, cloning, and reference work outside the range; select the intended kernel or expand the launch limit. |
203199
| Profiling initialization fails | Ensure the NCU path does not also start PyTorch's profiler; inspect the NCU version and driver/runtime errors. |
204-
| Wrong shape or missing recent evaluator changes | Inspect `manifest.json` and the remote source ref; local edits are not automatically uploaded. |
200+
| Wrong shape or missing recent evaluator changes | Inspect `manifest.json` and the operator's synced source revision; local edits are not automatically uploaded. |
205201
| Timeout | Start with one shape and a bounded kernel capture; NCU replay can be much slower than a normal benchmark. |

0 commit comments

Comments
 (0)