This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
nnInteractive is a 3D interactive medical image segmentation framework. It supports diverse prompt types (points, scribbles, bounding boxes, lasso) using 2D interactions to generate full 3D segmentations. This repository contains the inference-only package.
# Install from source (editable). This repo builds TWO distributions that share the
# `nnInteractive` namespace (see "Two-distribution layout" below): the torch-free client
# must be installed first because the full package depends on it.
pip install -e ./client # nninteractive-client (torch-free wire client)
pip install -e . # nnInteractive (full stack; depends on the client)
# Install with dev tools
pip install -e ".[dev]"
# Code formatting
black nnInteractive/
# Linting
ruff check nnInteractive/ --fix
# Spell checking
codespell --skip='.git,*.pdf,*.svg'
# Pre-commit hooks (after: pre-commit install)
pre-commit run --all-filesThere is no test suite in this repository.
The repo builds two pip distributions that share the single nnInteractive import namespace:
nninteractive-client— source underclient/. Ships onlynnInteractive.inference.remote(the torch-free remote client + the shared wire protocol/serialization). Depends on justnumpy,httpx,blosc2. Built fromclient/pyproject.toml.nnInteractive(full) — source at the repo root. Ships everything else (local engine, server, model management, …) and depends onnninteractive-client. Built from the rootpyproject.toml.
How the shared namespace works:
nnInteractiveandnnInteractive.inferenceare PEP 420 namespace packages — neither has an__init__.py; the two distributions populate disjoint files into the same directory. Both builds therefore usenamespaces = true, and the full buildexcludesnnInteractive.inference.remote*(client-owned) andnnInteractive.supervoxel*(separate package). Do not add an__init__.pyto either of those two dirs — it would make the two distributions ship the same file and break the clean split.- Because the layout is layered (disjoint files, full depends on client), the two coexist
cleanly:
pip install nnInteractivepulls the client; uninstalling the client leaves the full install intact; a client-only machine upgrades withpip install nnInteractive(no uninstall). - There is no
nnInteractive.__version__(namespace package has no module to carry it). Read the version viaimportlib.metadata.version("nnInteractive")/version("nninteractive-client").nnInteractiveInferenceSession.INFERENCE_SESSION_VERSIONdoes exactly this. - Full-only imports from a client-only install raise a friendly "
pip install nnInteractive" error via a last-resortsys.meta_pathfinder registered inclient/nnInteractive/inference/remote/_full_required.py(installed when the remote client is imported). It is a no-op when the full package is present. - The full server (
server/app.py) imports the wire serialization fromnnInteractive.inference.remote.serialization, which is now provided by the client dependency.
Session-based inference engine (~2000 lines) that manages state across multiple predictions. This is the main user-facing API.
Workflow: initialize_from_trained_model_folder() → set_image() → set_target_buffer() → add_*_interaction() (repeatable)
- Background threading: Image preprocessing and interaction initialization run in a ThreadPoolExecutor (max 2 workers) via futures
- AutoZoom: Adaptive patch selection with border change detection; zooms out up to 4x if predictions touch crop boundaries
- Refinement: After coarse prediction, difference maps identify regions needing fine-grained re-prediction
- Memory management: Selective CPU/GPU transfers, pre-allocated tensors, half-precision interactions (not pinned); non-contiguous GPU transfers stream through one fixed pinned staging buffer (
utils/staging.py, disabled on Linux kernel 6.11)
Important API constraints:
use_torch_compileis supported. The session default isFalse, but the server enables it by default (disable with--no-torch-compile). When enabled, the first prediction is slow (lazy compilation on the first forward pass) but subsequent ones are faster; the one-time compile cost is amortized across the long-lived server process.nnInteractiveInferenceSession.warmup()runs a single dummy forward pass at the network's only input shape ([1, num_input_channels + num_interaction_channels, *patch_size]— every prediction path uses this shape) to trigger compilation up front, and also runs the ops around the network once (staging-buffer transfer, argmax, border check) so their one-off costs (pinned buffer allocation, lazy CUDA kernel loading) don't hit the first prediction; the server calls it at startup so clients never see the first-prediction delaytarget_buffermust be 3D (shape[X, Y, Z]), not 4D- Scribble and lasso images must match
original_image_shape[1:](the original uncropped spatial shape) add_initial_seg_interaction()resets all existing interactions (see WARNING in its docstring)reset_interactions()also clears the target buffer- If multiple
add_*_interaction()calls are made without calling_predict(), only the last added interaction center is used for the initial prediction (all centers are queued, but only the last is consumed) - Images should not be preprocessed (no normalization, no level-window). The session handles all preprocessing internally
The interaction tensor has shape [7, D, H, W] in half precision (but can vary — num_interaction_channels is set from capability['interaction_channels']):
| Channel | Content |
|---|---|
| 0 | Initial segmentation |
| 1 | Positive bounding box / Lasso |
| 2 | Negative bounding box / Lasso |
| 3 | Positive points |
| 4 | Negative points |
| 5 | Positive scribble |
| 6 | Negative scribble |
initialize_from_trained_model_folder() reads model metadata from:
inference_info.json(new format) — containssupported_interactions,channel_mapping,interaction_channels,interaction_decay,point_radius, etc.inference_session_class.json(legacy format) — falls back to hardcoded defaults
The capability system (_apply_capability()) normalizes all channel indices to positive values at load time. Channel mappings use pairs [pos_channel, neg_channel] for interactions and a single index for prev_seg.
A trained model folder must contain:
inference_info.json(or legacyinference_session_class.json)dataset.jsonplans.jsonfold_{N}/checkpoint_final.pth(orfold_all/for ensemble)
A trained model folder may also contain:
LICENSE— the model's license. Only the first non-empty line is read and exposed assession.license(a short identifier, e.g.CC BY-NC-SA 4.0); any following lines (URL, full license text, …) are for human readers and are ignored. If the file is absent the loader falls back to the official-checkpoint heuristic (_is_official_checkpoint→CC BY-NC-SA 4.0), otherwise it reports!!MISSING!!. The license is printed on load and surfaced over the remote/capabilitiesendpoint so GUIs can display it.
Official weights are hosted on HuggingFace at MIC-DKFZ/nnInteractive. The selectable models are enumerated by a models.json manifest at the root of that HF repo (not of this git repo); discovery/download is handled by nnInteractive/model_management.py (list_models, ensure_model_available, get_default_model_id), which stores models under $NNINTERACTIVE_MODEL_DIR (default ~/.nninteractive). The default model is currently nnInteractive_v1.0.
nnInteractive/interaction/point.py: Point interaction with spherical structuring elements and distance transforms.build_point()useslru_cachefor structuring element reuse.nnInteractive/trainer/nnInteractiveTrainer.py: Minimal stub extending nnUNetv2. Used only for architecture reconstruction from checkpoints. Adds 7 extra input channels (num_input_channels + 7) on top of image channels.nnInteractive/utils/crop.py: Tensor cropping/padding with boundary handling (crop_and_pad_into_buffer,paste_tensor,crop_to_valid,pad_cropped)nnInteractive/utils/staging.py:PinnedStager— one 128 MB pinned buffer per process and device, double-buffered, streams crops of any size to the GPU in slabs (copy_to/copy_into;fill_todecompresses blosc2 straight into it).get_stager()returnsNonewhen staging is unavailable (non-CUDA device, Linux 6.11)nnInteractive/utils/bboxes.py: Greedy set cover algorithm for generating refinement patch bounding boxes from difference maps; falls back to random sampling when recursion depth exceedsmax_depthnnInteractive/utils/erosion_dilation.py:iterative_3x3_same_padding_pool3d— used for dilation of point/scribble channels before downsampling (zoom-out) and for morphological opening of the diff mapnnInteractive/utils/checkpoint_cleansing.py: Utility to strip optimizer state and trainer class from checkpoints before releasennInteractive/inference/cvpr2025_challenge_baseline/predict.py: Reference script for the CVPR 2025 challenge baseline (stateless per-call API wrapper aroundnnInteractiveInferenceSession)nnInteractive/supervoxel/: Optional separate module for SAM-based supervoxel generation (has its ownpyproject.tomland installation)
nnunetv2(>=2.7.0): Provides network architecture, ConfigurationManager, PlansManager, preprocessing pipelinetorch(>=2.1.2, !=2.9.*): PyTorch 2.9 is excluded due to OOM bugs with 3D convolutionsacvl-utils(>=0.2.3, <0.3): Spatial operations (cropping, padding)
Images are 4D numpy arrays [C, X, Y, Z]. Images are intentionally not cropped during preprocessing: interactions, predictions and the target buffer all live in the original image's coordinate space, so all add_*_interaction() coordinates are used as-is (the nonzero region is located only to compute normalization statistics).
Bounding box coordinates use [[x1, x2], [y1, y2], [z1, z2]] half-open intervals throughout. Current pretrained models only support 2D bounding boxes (one dimension must have size 1).
The _predict() method (decorated with @torch.inference_mode()) is highly optimized for minimal VRAM usage. Comments in the code note that the implementation has been extensively tuned and changes should only be made after fully understanding the VRAM/timing implications. The method:
- Runs an initial coarse prediction at
zoom_out_factor - Detects changes at prediction borders (
_detect_change_at_border) and iteratively zooms out (up to 4x, growth factor 1.5) if needed - If
zoom_out_factor == 1: directly pastes prediction into interactions and target buffer - If
zoom_out_factor > 1: computes a diff map, morphologically opens it, then runs refinement patches via_refine_coarse()
- Linux kernel 6.11 detection (
utils/os_shennanigans.py) disables pinned memory due to a kernel bug interaction_decay(default 0.98, legacy 0.9) downweights older interaction channels (all exceptprev_seg) on each prediction cycle