Skip to content

cluster.md

Latest commit

 

History

History
95 lines (74 loc) · 4.26 KB

cluster.md

File metadata and controls

95 lines (74 loc) · 4.26 KB
Raw

Running cuNCorr on a server / cluster

cuNCorr ships two paths from the same source tree:

  • CPU reference — portable, no GPU, no external deps. Runs anywhere; it is also the parity reference. Good for validation and small jobs.
  • CUDA — the GPU-accelerated path. Auto-used by the engine when a device is present.

1. Quick local sanity (any machine)

./build.sh cpu
ctest --test-dir build-cpu --output-on-failure        # 5 tests, all CPU
./build-cpu/make_synth /tmp/ds 256 256 3 0.7 -0.4     # synthetic frames
./build-cpu/proxyncorr_gpu --scalefactor 3 /tmp/ds /tmp/out
ls /tmp/out                                            # frame_0001.json + _strain.csv

2. Build the CUDA backend on the GPU node

module load cuda                      # or ensure nvcc is on PATH
./build.sh cuda -DCMAKE_CUDA_ARCHITECTURES=80   # 70=Volta 80=Ampere 89=Ada 90=Hopper
ctest --test-dir build-cuda -R cuda_parity --output-on-failure   # GPU-vs-CPU parity gate
./build-cuda/proxyncorr_gpu --backend             # should print "cuda:0 <name> (available)"

cuda_parity must pass before trusting GPU results — it diffs every batched GPU stage against the CPU reference (they share the same detail/* cores, so the diff should be ~0).

3. Containers

# GPU image
docker build -f deploy/Dockerfile -t cuncorr-gpu --build-arg CUDA_ARCH=80 .
docker run --gpus all -v $PWD/data:/data -v $PWD/out:/out cuncorr-gpu \
    proxyncorr_gpu --scalefactor 3 /data /out

# Portable CPU image (also runs the test suite during build)
docker build -f deploy/Dockerfile.cpu -t cuncorr-cpu .

# Apptainer/Singularity for HPC
apptainer build cuncorr.sif deploy/cuncorr.def
apptainer exec --nv cuncorr.sif proxyncorr_gpu --scalefactor 3 /data /out

4. SLURM

# Binary on PATH or via $PROXYNCORR, or an Apptainer image via $CUNCORR_SIF
export CUNCORR_SIF=$PWD/cuncorr.sif
sbatch deploy/run_slurm.sh /scratch/images /scratch/out --scalefactor 3 --subregion-radius 20

The script requests --gres=gpu:1, binds OpenMP threads to --cpus-per-task, and runs the CLI (directly, or apptainer exec --nv when CUNCORR_SIF is set).

5. Parity against CppNCorr (optional, recommended once)

CppNCorr is bundled as a submodule (Tools/CppNCorr) and built by cuNCorr's CMake:

git submodule update --init --recursive
cmake -S . -B build-oracle -DCUNCORR_BUILD_ORACLE=ON   # also writes lib/libncorr.a
cmake --build build-oracle -j
./build-oracle/test/dump_oracle Tools/CppNCorr/test/examples/ohtcfrp/images golden.json

Needs OpenCV/FFTW/SuiteSparse/BLAS (CppNCorr's deps). See parity.md.

Output & diagnostics

Each frame writes frame_NNNN.json with valid, points (non-null grid samples), message (why it failed, if it did), and the u/v/corrcoef arrays. The arrays always begin with a band of nulls — the margin (≈ subregion_radius+2 px) where a subset can't fit — so the first entries being null is normal; check points for the real count. Strain is written to frame_NNNN_strain.csv.

If you see all-null (points: 0), read message:

  • "no valid seed found" — displacement exceeds --seed-search (default 15 px), or the frame lacks texture/contrast, or --subregion-radius is too big for the image. Raise --seed-search, and for motion that grows across a sequence use --sequence.
  • "geometry does not match reference" — a frame's dimensions differ from the reference (often the ROI/ref file got mixed into the frame list; pass --ref/--roi explicitly).
  • "manual seed … failed" — the --seed X,Y point is off-grid/outside ROI/untextured.

Video overlay

--video renders a displacement-magnitude heatmap over each deformed frame (frame_NNNN_overlay.png) and, if ffmpeg is on PATH, muxes overlay.mp4 (--fps N, default 10). Without ffmpeg it prints the exact command to mux the PNGs yourself.

Tuning notes

  • --scalefactor sets the reduced-grid spacing (larger = fewer points, faster).
  • --subregion-radius is the correlation window; bigger is more robust, slower.
  • For large global displacements, raise --seed-search to cover the motion, or use --sequence (each frame warm-starts from the previous, so motion need not fit one search).
  • The default interpolation is bicubic-Keys (quintic B-spline is M1b).