Add authenticated hosted NCU profiling with capture options (#530)

Author: msaroufim

docs/hosted-ncu-profiling.md

@@ -0,0 +1,79 @@
+# Hosted Nsight Compute profiling
+
+`popcorn submit submission.py --leaderboard qr_v2 --profile --benchmark-index 0`
+uses the normal Popcorn identity and API. Users do not install Modal, create a
+provider account, or supply provider credentials. KernelBot dispatches the job
+to its GPU runner and streams back reports. `--profile-brev` remains an explicit,
+separate CLI choice; failures never switch providers.
+
+## API and capture contract
+
+`POST /profile/{leaderboard_name}/{gpu_type}` accepts the usual
+`X-Popcorn-Cli-Id` header and a multipart `file`, plus optional fields:
+
+| Field | Meaning |
+| --- | --- |
+| `benchmark_index` | Zero-based index in the task's `benchmarks`; omit for all entries. |
+| `ncu_kernel_name` | NCU kernel filter, including `regex:` expressions. |
+| `ncu_kernel_name_base` | `function`, `demangled`, or `mangled`. |
+| `ncu_launch_count` | Positive capture limit per benchmark; default 10. |
+
+The route retains normal submission permissions, limits, and leaderboard GPU
+validation. It supports the configured single-GPU NVIDIA Modal runners. Invalid
+options and unsupported GPUs are rejected; unsupported evaluators fail with NCU
+diagnostics. The dedicated route prevents older API versions from silently
+ignoring capture options.
+
+The runner sets `POPCORN_NCU=1`, follows child processes, and captures the NVTX
+push/pop range `custom_kernel/`. It collects the full NCU section set with kernel
+replay and leaves GPU clocks unchanged. Each benchmark returns a zip containing
+`profile.ncu-rep`, `ncu-details.txt`, and `ncu-details.csv`. A missing capture is a
+failure, even when the profiler process exits successfully.
+
+`FullResult.profile_metadata` records selected benchmark specifications, capture
+options, and a SHA-256 fingerprint of source/configuration fields. This is not a
+git commit. Record the synced reference-kernels revision in operator validation
+records; the client cannot update the hosted task through a local checkout.
+
+## Rollout
+
+Deploy the updated GPU runners and API before releasing the companion
+[popcorn-cli change](https://github.com/gpu-mode/popcorn-cli/pull/80). The runner
+image pins NCU 2025.2.1 and checks `ncu --version` during its build. NCU 2026.2
+from CUDA 13.3 produced many NaN counters on the tested Modal B200; repeating
+the same capture with 2025.2.1 restored hardware counters. Use the existing Modal deployment
+workflow and API deployment process; no additional container privileges or
+user-facing provider setup is introduced.
+
+Sync an NCU-compatible task through the existing problem-update workflow, then
+validate one benchmark through the public API using an ordinary Popcorn user.
+Inspect the returned report and counter exports. The
+[reference-kernels guide](https://github.com/gpu-mode/reference-kernels/pull/171)
+describes the evaluator contract for new problems. Merely accepting `profile`
+and running `torch.profiler` does not satisfy that contract.
+
+## Validation
+
+CPU tests cover API authentication and options, benchmark selection, capture
+commands, exports, and empty captures. The GPU fixture is
+`problems/linalg/qr_v2`, benchmark 0 (`batch=20, n=32, cond=1, seed=43214`), at
+reference-kernels `51e22db671d36c1c76091c43c36a44546ba324a1`.
+Only this problem/shape is used for the GPU integration check; other tasks and
+GPU types require their own validation.
+
+The integration check used the real FastAPI route, submission preparation,
+KernelBackend, and GPU `run_config`, with a fake database and an ephemeral
+launcher instead of production infrastructure. The CLI used normal Popcorn
+header authentication with an empty `PATH` and no `MODAL_*` environment values.
+It successfully saved and extracted a 7,593,257-byte report and both detail
+exports. NCU reopened the report to produce those exports.
+
+NCU 2025.2.1 collected 39 passes on NVIDIA B200 with PyTorch 2.12.0+cu130,
+CUDA 13.3, and `regex:geqr2` / demangled / launch count 1. Observed counters:
+322.78 us duration, 12.51% achieved occupancy, 0.42% SM throughput, 70.09% L2
+hit rate, and 1,565,428 executed instructions. Six `ctc__*` metrics remained
+unavailable. This validates the isolated implementation; the production API
+has not been deployed as part of this change.
+
+[Ephemeral validation run](https://modal.com/apps/coreauto/main/ap-p8eodYfHkZAMp0ZT5btR1y)
+(access requires the operator's workspace permissions).

src/kernelbot/api/main.py

@@ -8,17 +8,18 @@
 from dataclasses import asdict
 from typing import Annotated, Any, Optional
 
-from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request, UploadFile
+from fastapi import Depends, FastAPI, Form, Header, HTTPException, Query, Request, UploadFile
 from fastapi.responses import JSONResponse, StreamingResponse
 
 from kernelbot.env import env
 from libkernelbot.application_validation import ApplicationValidationService
 from libkernelbot.backend import KernelBackend
 from libkernelbot.background_submission_manager import BackgroundSubmissionManager
-from libkernelbot.consts import SubmissionMode
+from libkernelbot.consts import SubmissionMode, get_gpu_by_name
 from libkernelbot.db_types import IdentityType
 from libkernelbot.leaderboard_db import LeaderboardDB, LeaderboardRankedEntry
 from libkernelbot.problem_sync import sync_problems
+from libkernelbot.profiling import ProfileOptions
 from libkernelbot.submission import (
     ProcessedSubmissionRequest,
     SubmissionRequest,
@@ -522,6 +523,43 @@ async def admin_run_application_validation(
     }
 
 
+@app.post("/profile/{leaderboard_name}/{gpu_type}")
+async def profile_submission(
+    leaderboard_name: str,
+    gpu_type: str,
+    file: UploadFile,
+    user_info: Annotated[dict, Depends(validate_cli_header)],
+    benchmark_index: Annotated[int | None, Form(ge=0)] = None,
+    ncu_kernel_name: Annotated[str | None, Form(min_length=1, max_length=1024)] = None,
+    ncu_kernel_name_base: Annotated[str | None, Form()] = None,
+    ncu_launch_count: Annotated[int | None, Form(ge=1)] = None,
+    db_context=Depends(get_db),
+) -> StreamingResponse:
+    """Profile through the normal authenticated submission pipeline.
+
+    A dedicated route prevents older API deployments from silently ignoring
+    capture options sent by a newer CLI. Compute-provider credentials stay here.
+    """
+    await simple_rate_limit()
+    try:
+        gpu = get_gpu_by_name(gpu_type)
+        if gpu is None or gpu.runner != "Modal" or gpu.name == "L4x4":
+            raise ValueError("Hosted NCU profiling requires a supported single NVIDIA GPU")
+        options = ProfileOptions(
+            benchmark_index, ncu_kernel_name, ncu_kernel_name_base, ncu_launch_count
+        )
+    except (ValueError, KeyError) as error:
+        raise HTTPException(status_code=400, detail=str(error)) from error
+    request, mode = await to_submit_info(
+        user_info, "profile", file, leaderboard_name, gpu_type, db_context
+    )
+    request.profile_options = options.to_dict()
+    return StreamingResponse(
+        _stream_submission_response(request, mode, backend_instance),
+        media_type="text/event-stream",
+    )
+
+
 @app.post("/{leaderboard_name}/{gpu_type}/{submission_mode}")
 async def run_submission(  # noqa: C901
     leaderboard_name: str,

src/libkernelbot/backend.py

@@ -136,6 +136,10 @@ async def submit_full(
                     req.task,
                     mode,
                     None,
+                    **(
+                        {"profile_options": req.profile_options}
+                        if req.profile_options is not None else {}
+                    ),
                 )
                 for gpu in selected_gpus
             ]
@@ -174,6 +178,7 @@ async def submit_leaderboard(  # noqa: C901
         task: LeaderboardTask,
         mode: SubmissionMode,
         seed: Optional[int],
+        profile_options: dict | None = None,
     ) -> Optional[FullResult]:
         """
         Function invoked by `leaderboard_cog` to handle a leaderboard run.
@@ -193,6 +198,7 @@ async def submit_leaderboard(  # noqa: C901
             task=task,
             mode=mode,
             submission_id=submission_id,
+            **({"profile_options": profile_options} if profile_options is not None else {}),
         )
 
         if result.success:
@@ -232,6 +238,7 @@ async def handle_submission(
         task: Optional[LeaderboardTask],
         mode: SubmissionMode,
         submission_id: int = -1,
+        profile_options: dict | None = None,
     ) -> Optional[FullResult]:
         """
         Generic function to handle code submissions.
@@ -250,6 +257,9 @@ async def handle_submission(
             task=task, submission_content=code, arch=self._get_arch(gpu_type), mode=mode
         )
 
+        if profile_options is not None:
+            config["profile_options"] = profile_options
+
         logger.info("submitting task to runner %s", launcher.name)
 
         result = await launcher.run_submission(config, gpu_type, reporter)

src/libkernelbot/profiling.py

@@ -0,0 +1,38 @@
+"""Validated, request-scoped profiling options shared by API and GPU runners."""
+
+from dataclasses import asdict, dataclass
+from typing import Literal
+
+
+@dataclass(frozen=True)
+class ProfileOptions:
+    benchmark_index: int | None = None
+    ncu_kernel_name: str | None = None
+    ncu_kernel_name_base: Literal["function", "demangled", "mangled"] | None = None
+    ncu_launch_count: int | None = None
+
+    def __post_init__(self):
+        for name, minimum in [("benchmark_index", 0), ("ncu_launch_count", 1)]:
+            value = getattr(self, name)
+            if value is not None and (type(value) is not int or value < minimum):
+                raise ValueError(f"{name} must be an integer >= {minimum}")
+        if self.ncu_kernel_name_base not in (None, "function", "demangled", "mangled"):
+            raise ValueError("ncu_kernel_name_base must be function, demangled, or mangled")
+        if self.ncu_kernel_name is not None and (
+            not isinstance(self.ncu_kernel_name, str)
+            or not self.ncu_kernel_name
+            or len(self.ncu_kernel_name) > 1024
+            or "\x00" in self.ncu_kernel_name
+        ):
+            raise ValueError("ncu_kernel_name must contain 1..1024 characters without NUL")
+
+    def validate_task(self, benchmarks: list, multi_gpu: bool):
+        if multi_gpu:
+            raise ValueError("NCU profiling requires a single GPU")
+        if not benchmarks:
+            raise ValueError("This task has no benchmarks to profile")
+        if self.benchmark_index is not None and self.benchmark_index >= len(benchmarks):
+            raise ValueError(f"benchmark_index must be between 0 and {len(benchmarks) - 1}")
+
+    def to_dict(self) -> dict:
+        return {name: value for name, value in asdict(self).items() if value is not None}