Skip to content

Integrate cuNCorr as GPU/CPU DIC engine (cppxdic → cuncorr) - #38

Closed
johnaoga wants to merge 6 commits into
mainfrom
feat/cuncorr-integration
Closed

Integrate cuNCorr as GPU/CPU DIC engine (cppxdic → cuncorr)#38
johnaoga wants to merge 6 commits into
mainfrom
feat/cuncorr-integration

Conversation

@johnaoga

@johnaoga johnaoga commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds cuNCorr (GPU-accelerated DIC engine, new submodule Tools/cuNCorr) as a selectable DIC backend. cuNCorr runs on CUDA when a device is present and on its CPU backend otherwise — the two are numerically identical by construction (shared math cores; cuNCorr's own cuda_parity_test gates CUDA==CPU). CppNCorr remains the parity oracle underneath.

Dependency chain: cppxdic → cuncorr (engine) and cppxdic → ncorr (result types + legacy path).

What changed

  • Submodule Tools/cuNCorr (@ a9518ad); CMake builds the standalone cuncorr lib (its bundled CppNCorr, tests, and apps disabled — cppxdic keeps its own ncorr target). Configure -DCUNCORR_BACKEND=CUDA on a GPU node.
  • Adapter src/dic/cuncorr_dic.{h,cpp} (cppxdic::run_cuncorr_dic): runs cuNCorr in sequence mode (fixed reference = frame 0, each frame warm-started from the previous) and packages the result as ncorr::DIC_analysis_output — raw Lagrangian, pixel-unit, on the reduced grid. Grids align exactly with ncorr ((dim-1)/sf+1 == ceil(dim/sf); reduced node (r,c) ↔ full pixel (r·sf, c·sf)), so the .bin and StepE 3D reconstruction consume it unchanged.
  • Dispatch at both DIC call sites (single_dic_workflow.cpp, step_d_workflow.cpp::runNcorrAnalysis) on new config dic_engine (cuncorr default | ncorr legacy) + cuncorr_seed_search.
  • Test tests/cuncorr_adapter_test.cpp (opt-in -DBUILD_CUNCORR_ADAPTER_TEST=ON): synthetic translation recovery through the adapter.

⚠️ Reviewer notes

  1. Default dic_engine = cuncorr changes production DIC numbers. cuNCorr is a different algorithm from CppNCorr (bicubic IC-GN vs biquintic RG-DIC), so results differ. Validate on real fingertip data before merging/relying on it. Reverting is one config line (dic_engine = ncorr).
  2. Build-graph deviation: cuNCorr is added as a standalone cuncorr lib and cppxdic keeps its own ncorr (same commit 861b5ce), rather than re-sourcing ncorr through cuNCorr's nested submodule — avoids a duplicate ncorr target and include re-plumbing, with identical runtime behaviour.

Verification

  • cppxdic, singledic, and cuncorr_adapter_test all build & link against cuncorr + ncorr (CPU).
  • Grid alignment proven; ROI2D::form_union cannot throw (mask built at the reduced ROI's exact dims).
  • Adapter runtime test couldn't execute on the dev Mac (unrelated broken local Homebrew protobuf dylib blocks all OpenCV-linked binaries — same as dump_oracle); runs on the cluster / any working-OpenCV host.

Build

git submodule update --init --recursive
cmake -S . -B build                          # CPU engine (default)
cmake -S . -B build -DCUNCORR_BACKEND=CUDA   # GPU node (needs nvcc)
cmake --build build -j

🤖 Generated with Claude Code

johnaoga and others added 6 commits July 1, 2026 23:59
@johnaoga @claude
feat(dic): integrate cuNCorr as GPU/CPU DIC engine (cppxdic -> cuncorr)
8a5e0ca 
Adds cuNCorr (submodule Tools/cuNCorr) as a selectable DIC backend. cuNCorr runs
on CUDA when a device is present, else its CPU backend — numerically identical
(shared cores; cuNCorr's cuda_parity_test gates it).

- Tools/cuNCorr submodule; CMake builds the standalone `cuncorr` lib
  (tests/apps/its-own-CppNCorr disabled; cppxdic keeps its ncorr target).
  Configure -DCUNCORR_BACKEND=CUDA on a GPU node.
- Adapter src/dic/cuncorr_dic.{h,cpp}: runs cuNCorr sequence mode (fixed
  reference, warm-started frames) and packages results as ncorr::DIC_analysis_output
  (raw Lagrangian/pixel, reduced grid). Grids align exactly with ncorr
  ((dim-1)/sf+1 == ceil(dim/sf); reduced (r,c) <-> full (r*sf,c*sf)), so the
  .bin + StepE reconstruction consume it unchanged.
- Dispatch at both DIC call sites (single_dic_workflow.cpp, step_d_workflow.cpp)
  on Config::dic_engine ("cuncorr" default | "ncorr" legacy) + cuncorr_seed_search.
- tests/cuncorr_adapter_test.cpp (opt-in -DBUILD_CUNCORR_ADAPTER_TEST=ON):
  synthetic translation recovery through the adapter.

Verified: cppxdic + singledic + adapter test all build & link against cuncorr+ncorr
(CPU). Runtime of OpenCV-linked binaries blocked locally by a broken brew
protobuf dylib (env, not code); runs on the cluster.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@johnaoga @claude
fix(ci): make cuNCorr optional + clang-format the adapter files
ca00f1d 
CI failures on PR #38:
- build/test/coverage: add_subdirectory(Tools/cuNCorr) failed because the CI
  runner does not check out the private cuNCorr submodule. cuNCorr is now
  OPTIONAL: guarded on Tools/cuNCorr/CMakeLists.txt existing (+ new option
  CPPXDIC_USE_CUNCORR, default ON). Absent -> ncorr-only build; the adapter
  source, its link, and the dispatch branches are gated behind CPPXDIC_HAVE_CUNCORR
  in single_dic_workflow.cpp and step_d_workflow.cpp.
- lint: clang-format (18.1.8) the three added files.

Also switch the cuNCorr submodule URL to https (consistent with CppNCorr; works
with a token that has org access).

Verified locally: builds with cuNCorr ENABLED (adapter compiled) and with
-DCPPXDIC_USE_CUNCORR=OFF (ncorr-only, what CI does). clang-format --dry-run
--Werror clean on the added files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@johnaoga @claude
ci: build + adapter-test the cuNCorr integration (now that cuNCorr is…
ec92eb2 
… public)

cuNCorr is a public repo, so CI can fetch the submodule.

- build job: checks out Tools/cuNCorr, compiles cppxdic WITH the cuNCorr engine
  (CPPXDIC_HAVE_CUNCORR), and runs the adapter synthetic-recovery test
  (-DBUILD_CUNCORR_ADAPTER_TEST=ON) — validates run_cuncorr_dic end to end on CPU.
- test + coverage jobs: pinned ncorr-only (-DCPPXDIC_USE_CUNCORR=OFF) so the
  ohtcfrp golden regressions keep validating the CppNCorr reference engine
  (cuNCorr is a different algorithm with different numbers).

Verified locally: the build-job commands build cppxdic + cuncorr_adapter_test
with the engine ENABLED.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@johnaoga @claude
build(gpu): make the CUDA image buildable (cppxdic_gpu.def)
688ec01 
- deploy/cluster/cppxdic_gpu.def: single Apptainer image on
  nvidia/cuda:12.4.1 building cuNCorr with -DCUNCORR_BACKEND=CUDA for
  sm 70/75/80/86/89 (all Manneback GPU types) + hdf5-tools (h5diff) and
  gdb + a RelWithDebInfo cppxdic_dbg for cluster debugging. One image
  serves GPU and CPU runs (cuNCorr falls back to its CPU backend without
  --nv), keeping binaries identical for bit-comparisons.
- CMakeLists: turn CUDA_SEPARABLE_COMPILATION OFF on the cuncorr target —
  the non-CUDA executables (cppxdic, singledic) cannot generate the
  device-link step an rdc static lib requires (and CMake < 3.23 cannot
  resolve device symbols on static libraries); cuNCorr's kernels are
  TU-local so rdc was never needed. Without this the link fails with
  undefined __cudaRegisterLinkedBinary_* symbols.
- generate_ncorr_bin: call H5Rdereference2 explicitly behind a version
  guard — Ubuntu 22.04's hdf5 1.10 maps the H5Rdereference macro to the
  3-arg v1.8 signature and the build fails inside the container.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@johnaoga @claude
fix(dic): cuNCorr pair-results validity + keep camera matching on ncorr
5aba750 
Two fixes surfaced by the first end-to-end cuNCorr runs:

1. formatOutput used a "value == 0.0 -> invalid" sentinel when building
   myDIC2DpairResults Points/CorCoeffVec. ncorr zero-fills invalid points
   so it mostly worked, but a VALID point can hold an exact 0.0 — notably
   cuNCorr's whole first frame (self-match returns u=v=0 exactly), which
   turned Points{1} (the reference seed grid, stored by MATLAB and ncorr
   alike) into all-NaN. Validity now comes from each frame's disp ROI mask
   (the set of converged points) in fillDirectFrame, the cam2 interpolation
   sampler, and the matching check. Behavior-preserving for ncorr.

2. Inter-camera MATCHING calls were dispatched to cuNCorr like tracking.
   Camera disparities (~276 px here) far exceed cuNCorr's local coarse
   seed search (cuncorr_seed_search, 15 px) — only ncorr's GLOBAL seed
   optimization bridges them — so matching silently produced an all-zero
   field with an empty valid ROI, cascading into all-NaN cam-def Points,
   all-NaN 3D vertices, and a 0-face stitched mesh. Matching calls
   (<= 2 current images, not perf-critical) now stay on ncorr; cuNCorr
   keeps the frame-series tracking it was designed for.

Verified end-to-end on S09 trial 7: GPU run completes Steps D+E+F in
32 min with healthy cam-def Points (4% NaN, ROI edges only) and the same
stitched mesh (6726 points / 12071 faces) as the ncorr CPU run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@johnaoga @claude
feat(study): CPU-vs-GPU comparison harness + results (Manneback)
458f8a3 
deploy/cluster/cpugpu/: SLURM harness for the ncorr-CPU vs cuNCorr-GPU
study on S09 trials 7/12/25 — isolated per-run dic_paths, explicit
dic_engine (the compiled-in default is cuncorr!), engine/backend
verification, axes for threads/GPU/exact_matlab/updates/fixed-step/stages,
crash-safe meta on home (globalscratch is purgeable). Plus:
- compare_bits.sh: bit-level comparison (h5diff on .mat — never md5, the
  v7.3 userblock embeds timestamps — and cmp on .bin).
- compare_dic.cpp: matio-based semantic comparator for MATLAB (v5 +
  struct-array layout) vs cppxdic (v7.3 + cells) files — recursive walk,
  NaN-aware, point-matched trajectory mode, DUMP/LIST probes with deep
  loading of deferred HDF5 data.
- results/REPORT.md + summaries: GPU ~4x faster than 32-core CPU at ~15x
  less memory; GPU bit-deterministic across reps; cuncorr CPU-vs-GPU
  differences bounded by 1e-12 (all trials); MATLAB agreement sub-0.1 px
  for the bulk with the boundary band explained by MATLAB's FFT-bcoef
  border bias (fixed on our side by CppNCorr 4758ad2); full-pipeline
  Step-E/F bug chain documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
johnaoga added a commit that referenced this pull request Sep 3, 2026
@johnaoga
Fix/step ef pipeline (#41)
b9bbfad 
* feat(dic): integrate cuNCorr as GPU/CPU DIC engine (cppxdic -> cuncorr)

Adds cuNCorr (submodule Tools/cuNCorr) as a selectable DIC backend. cuNCorr runs
on CUDA when a device is present, else its CPU backend — numerically identical
(shared cores; cuNCorr's cuda_parity_test gates it).

- Tools/cuNCorr submodule; CMake builds the standalone `cuncorr` lib
  (tests/apps/its-own-CppNCorr disabled; cppxdic keeps its ncorr target).
  Configure -DCUNCORR_BACKEND=CUDA on a GPU node.
- Adapter src/dic/cuncorr_dic.{h,cpp}: runs cuNCorr sequence mode (fixed
  reference, warm-started frames) and packages results as ncorr::DIC_analysis_output
  (raw Lagrangian/pixel, reduced grid). Grids align exactly with ncorr
  ((dim-1)/sf+1 == ceil(dim/sf); reduced (r,c) <-> full (r*sf,c*sf)), so the
  .bin + StepE reconstruction consume it unchanged.
- Dispatch at both DIC call sites (single_dic_workflow.cpp, step_d_workflow.cpp)
  on Config::dic_engine ("cuncorr" default | "ncorr" legacy) + cuncorr_seed_search.
- tests/cuncorr_adapter_test.cpp (opt-in -DBUILD_CUNCORR_ADAPTER_TEST=ON):
  synthetic translation recovery through the adapter.

Verified: cppxdic + singledic + adapter test all build & link against cuncorr+ncorr
(CPU). Runtime of OpenCV-linked binaries blocked locally by a broken brew
protobuf dylib (env, not code); runs on the cluster.

* fix(ci): make cuNCorr optional + clang-format the adapter files

CI failures on PR #38:
- build/test/coverage: add_subdirectory(Tools/cuNCorr) failed because the CI
  runner does not check out the private cuNCorr submodule. cuNCorr is now
  OPTIONAL: guarded on Tools/cuNCorr/CMakeLists.txt existing (+ new option
  CPPXDIC_USE_CUNCORR, default ON). Absent -> ncorr-only build; the adapter
  source, its link, and the dispatch branches are gated behind CPPXDIC_HAVE_CUNCORR
  in single_dic_workflow.cpp and step_d_workflow.cpp.
- lint: clang-format (18.1.8) the three added files.

Also switch the cuNCorr submodule URL to https (consistent with CppNCorr; works
with a token that has org access).

Verified locally: builds with cuNCorr ENABLED (adapter compiled) and with
-DCPPXDIC_USE_CUNCORR=OFF (ncorr-only, what CI does). clang-format --dry-run
--Werror clean on the added files.



* ci: build + adapter-test the cuNCorr integration (now that cuNCorr is public)

cuNCorr is a public repo, so CI can fetch the submodule.

- build job: checks out Tools/cuNCorr, compiles cppxdic WITH the cuNCorr engine
  (CPPXDIC_HAVE_CUNCORR), and runs the adapter synthetic-recovery test
  (-DBUILD_CUNCORR_ADAPTER_TEST=ON) — validates run_cuncorr_dic end to end on CPU.
- test + coverage jobs: pinned ncorr-only (-DCPPXDIC_USE_CUNCORR=OFF) so the
  ohtcfrp golden regressions keep validating the CppNCorr reference engine
  (cuNCorr is a different algorithm with different numbers).

Verified locally: the build-job commands build cppxdic + cuncorr_adapter_test
with the engine ENABLED.

* build(gpu): make the CUDA image buildable (cppxdic_gpu.def)

- deploy/cluster/cppxdic_gpu.def: single Apptainer image on
  nvidia/cuda:12.4.1 building cuNCorr with -DCUNCORR_BACKEND=CUDA for
  sm 70/75/80/86/89 (all Manneback GPU types) + hdf5-tools (h5diff) and
  gdb + a RelWithDebInfo cppxdic_dbg for cluster debugging. One image
  serves GPU and CPU runs (cuNCorr falls back to its CPU backend without
  --nv), keeping binaries identical for bit-comparisons.
- CMakeLists: turn CUDA_SEPARABLE_COMPILATION OFF on the cuncorr target —
  the non-CUDA executables (cppxdic, singledic) cannot generate the
  device-link step an rdc static lib requires (and CMake < 3.23 cannot
  resolve device symbols on static libraries); cuNCorr's kernels are
  TU-local so rdc was never needed. Without this the link fails with
  undefined __cudaRegisterLinkedBinary_* symbols.
- generate_ncorr_bin: call H5Rdereference2 explicitly behind a version
  guard — Ubuntu 22.04's hdf5 1.10 maps the H5Rdereference macro to the
  3-arg v1.8 signature and the build fails inside the container.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dic): cuNCorr pair-results validity + keep camera matching on ncorr

Two fixes surfaced by the first end-to-end cuNCorr runs:

1. formatOutput used a "value == 0.0 -> invalid" sentinel when building
   myDIC2DpairResults Points/CorCoeffVec. ncorr zero-fills invalid points
   so it mostly worked, but a VALID point can hold an exact 0.0 — notably
   cuNCorr's whole first frame (self-match returns u=v=0 exactly), which
   turned Points{1} (the reference seed grid, stored by MATLAB and ncorr
   alike) into all-NaN. Validity now comes from each frame's disp ROI mask
   (the set of converged points) in fillDirectFrame, the cam2 interpolation
   sampler, and the matching check. Behavior-preserving for ncorr.

2. Inter-camera MATCHING calls were dispatched to cuNCorr like tracking.
   Camera disparities (~276 px here) far exceed cuNCorr's local coarse
   seed search (cuncorr_seed_search, 15 px) — only ncorr's GLOBAL seed
   optimization bridges them — so matching silently produced an all-zero
   field with an empty valid ROI, cascading into all-NaN cam-def Points,
   all-NaN 3D vertices, and a 0-face stitched mesh. Matching calls
   (<= 2 current images, not perf-critical) now stay on ncorr; cuNCorr
   keeps the frame-series tracking it was designed for.

Verified end-to-end on S09 trial 7: GPU run completes Steps D+E+F in
32 min with healthy cam-def Points (4% NaN, ROI edges only) and the same
stitched mesh (6726 points / 12071 faces) as the ncorr CPU run.

* feat(study): CPU-vs-GPU comparison harness + results (Manneback)

deploy/cluster/cpugpu/: SLURM harness for the ncorr-CPU vs cuNCorr-GPU
study on S09 trials 7/12/25 — isolated per-run dic_paths, explicit
dic_engine (the compiled-in default is cuncorr!), engine/backend
verification, axes for threads/GPU/exact_matlab/updates/fixed-step/stages,
crash-safe meta on home (globalscratch is purgeable). Plus:
- compare_bits.sh: bit-level comparison (h5diff on .mat — never md5, the
  v7.3 userblock embeds timestamps — and cmp on .bin).
- compare_dic.cpp: matio-based semantic comparator for MATLAB (v5 +
  struct-array layout) vs cppxdic (v7.3 + cells) files — recursive walk,
  NaN-aware, point-matched trajectory mode, DUMP/LIST probes with deep
  loading of deferred HDF5 data.
- results/REPORT.md + summaries: GPU ~4x faster than 32-core CPU at ~15x
  less memory; GPU bit-deterministic across reps; cuncorr CPU-vs-GPU
  differences bounded by 1e-12 (all trials); MATLAB agreement sub-0.1 px
  for the bulk with the boundary band explained by MATLAB's FFT-bcoef
  border bias (fixed on our side by CppNCorr 4758ad2); full-pipeline
  Step-E/F bug chain documented.

* feat(dic): fixed-step reference updates + no_update tracking-preset option

Wire two ncorr_params.txt options into Step-D tracking (both default to the
long-standing behavior):

- fixed_step_ref = N (>0): force a reference change every N frames — MATLAB
  ncorr step analysis semantics (stepanalysis.step / step_ref_change). Bumps
  Tools/CppNCorr to feat/fixed-step-ref, which caps seed segments at N frames
  in matlab_compute_seed_segment; the existing segment loop does the seed
  propagation and the exact_matlab_* path chains segments back to the global
  reference (ncorr_alg_addanalysis port). Verified on S09 t7 (150 frames,
  step 10): 60 segments = 150/10 x 4 tracking calls, matching MATLAB's
  recorded dispinfo.stepanalysis exactly.
- no_update = false: KEEP_MOST_POINTS preset (with cutoff_corrcoef=0.5 ->
  correlation-triggered reference updates); default true keeps NO_UPDATE.
  The tracking call site previously hardcoded true.

Study finding (deploy/cluster/cpugpu/results/REPORT.md): reproducing MATLAB's
fixed-step structure is numerically WORSE at the 0.1 px level than direct
fixed-reference tracking — per-boundary composition differences compound over
15 boundaries — so the production default stays fixed-reference; fixed_step_ref
remains available for data where a fixed reference genuinely decorrelates.

* fix(step-ef): resolve the Step-E stitching-save segfault chain

The long-standing "segfault at Step E/F" was three stacked bugs, each
hiding the next (gdb-diagnosed on Manneback, full chain in
deploy/cluster/cpugpu/results/REPORT.md):

1. mat_writer embedded AllPairsResults/DIC2Dinfo/ROImask into the
   DIC3Dcombined struct by WRITING them as top-level variables into the
   output file and Mat_VarRead-ing them back through the write handle —
   the returned matvars carry live HDF5-internal state and crash
   Mat_VarSetStructFieldByName. Replaced with pure in-memory builders
   (buildAllPairsResultsVar / buildDIC2DinfoVar; ROImask built directly
   from the cv::Mat). Also removes the duplicated top-level variables
   (~half the output size).

2. A crash mid-save left a truncated DIC3Dcombined that later runs
   accepted as a valid checkpoint (existence-only test) and then failed
   on. dic_analysis now writes to a .partial file and renames on success.

3. With E finally producing valid files, Step F aborted with "0 faces":
   v7.3 reads defer nested cell/field data and mat_reader's per-field
   ensure_loaded patchwork missed Faces (and five other fields). Replaced
   with one recursive deep_load of the whole struct tree.

Verified end-to-end (S09 t7, both engines): Steps D+E+F complete rc=0,
identical stitched meshes (6726 points / 12071 faces), 1.7 GB
DIC3DPPresults from Step F. GPU 32 min, ncorr-CPU 75 min.


* feat(dic): wire the dangling perspective_interp tag (biquintic Eulerian conversion)

The `perspective_interp` ncorr-params key has been parsed since March
(e53dc45) but consumed by nothing. It now selects the interpolation for the
Eulerian perspective change: false (default) keeps the bicubic CUBIC_KEYS
workaround — a relic of the FFT-bcoef border bias era, preserved so existing
outputs are unchanged — and true selects biquintic B-spline, matching MATLAB
ncorr, which is numerically safe again since CppNCorr replaced the FFT
circular deconvolution with the Unser recursive filter (CppNCorr 4758ad2).

Background: the "48 px / 20 px border padding" investigation
(deploy/cluster/cpugpu/results/REPORT.md) — MATLAB still carries the
FFT-bcoef border bias; boundary points are 6% of the grid but 23-45% of the
>1px MATLAB-vs-cppxdic disagreements, and cppxdic is the more correct side
there.

* docs: CLAUDE.md for the cuncorr-integration branch family

Cluster specifics (Manneback/Lemaitre path aliasing, purgeable
globalscratch, sacct job-id recycling trap), cpugpu harness usage,
config-file placement traps, and the July-2026 findings: CPU/GPU
perf + bit-parity, MATLAB agreement decomposition, the cuNCorr
matching limitation, and the Step-E/F bug chain.

* docs(study): definitive pipeline validation results

- t7/t12/t25 full pipeline (D+E+F) pass rc=0 on both engines
  (GPU 32-38 min, ncorr CPU-32t 58-75 min).
- MATLAB Step-D interop: cppxdic E+F on MATLAB's own 2D results
  reproduces MATLAB's E exactly (Points3D max 3.2e-14, corrComb 0,
  same 6730 points) and F per-point quantities to <0.05
  (faces +0.23% from the zipping tail).
@johnaoga johnaoga closed this Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@johnaoga