Feat/mirrored mode mng

deploy/cluster/cppxdic_mirrored.def

@@ -0,0 +1,133 @@
+Bootstrap: docker
+From: ubuntu:22.04
+
+%labels
+    Author CPPxDIC Team
+    Version 1.0-mirrored
+    Description CPPxDIC built with XDIC_MODE=mirrored (mirrored-camera xDIC reconstruction)
+
+%help
+    CPPxDIC container built for XDIC_MODE=mirrored (Section 2d mirrored-camera
+    reconstruction, src/xdic/mirrored/mirrored_mode.cpp) instead of the default
+    camerapairs mode. XDIC_MODE is a CMake compile-time cache variable, so a
+    single binary can only serve one mode — this image cannot also do
+    camerapairs reconstruction. Only the `cppxdic` binary is mode-sensitive:
+    `singledic` hardcodes XDIC_MODE_CAMERAPAIRS regardless of this build's
+    XDIC_MODE (see CMakeLists.txt), so it behaves identically to the plain
+    cppxdic.def image and is included here only for convenience.
+
+    Apps baked into the image (select by first argument; default cppxdic):
+        cppxdic            mirrored-camera 3D-DIC pipeline (XDIC_MODE=mirrored)
+        singledic          single-camera 2D DIC (always camerapairs internals)
+        proxyncorr         pass-through DIC driver over an image folder
+        gen_subject_trial  generate subject_trial.csv for SLURM arrays
+        apptainer run cppxdic_mirrored.sif --list-apps   # list what is built in
+
+    Build:
+        apptainer build --fakeroot cppxdic_mirrored.sif deploy/cluster/cppxdic_mirrored.def
+
+    Run (default cppxdic, mirrored mode):
+        apptainer run cppxdic_mirrored.sif --dic-params /path/to/dic_params.txt
+
+    Run (with SLURM):
+        See submit_job_mirrored.sh  (APP=singledic sbatch submit_job_mirrored.sh ...)
+
+%environment
+    export LC_ALL=C
+    export LANG=C
+    export PATH=/opt/cppxdic/bin:$PATH
+    export LD_LIBRARY_PATH=/usr/local/lib:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
+    export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-4}
+
+%post
+    # =========================================================================
+    # System packages (identical set to cppxdic.def — this is a CPU/ncorr-only
+    # image, no CUDA toolchain, so CPPXDIC_HAVE_CUNCORR auto-detects OFF)
+    # =========================================================================
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update && apt-get install -y --no-install-recommends \
+        build-essential \
+        cmake \
+        g++ \
+        git \
+        pkg-config \
+        ca-certificates \
+        wget \
+        libopencv-dev \
+        libfftw3-dev \
+        libmatio-dev \
+        libhdf5-dev \
+        libcgal-dev \
+        libeigen3-dev \
+        libsuitesparse-dev \
+        liblapack-dev \
+        libblas-dev \
+        libopenblas-dev \
+        libomp-dev \
+        nlohmann-json3-dev \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+    echo "=== Verifying dependencies ==="
+    pkg-config --modversion opencv4   || echo "WARN: opencv4 pkg-config missing (may still work)"
+    dpkg -s libmatio-dev              | grep Version
+    dpkg -s libhdf5-dev               | grep Version
+    echo "=== All dependencies verified ==="
+
+    # =========================================================================
+    # Clone and build CPPxDIC (XDIC_MODE=mirrored)
+    # =========================================================================
+    rm -rf /opt/cppxdic/src/build
+    rm -rf /opt/cppxdic/src/tests/build
+    rm -rf /opt/cppxdic/src/Tools/CppNCorr/lib
+    rm -f  /opt/cppxdic/src/compile_commands.json
+
+    cd /opt/cppxdic/src
+    mkdir -p build && cd build
+
+    # The one difference from cppxdic.def: -DXDIC_MODE=mirrored. This only
+    # changes which src/xdic/<mode>/ file compiles into `cppxdic`; singledic
+    # and proxyncorr are unaffected by this flag.
+    cmake .. \
+        -DCMAKE_BUILD_TYPE=Release \
+        -DCMAKE_INSTALL_PREFIX=/opt/cppxdic \
+        -DCMAKE_CXX_STANDARD=17 \
+        -DXDIC_MODE=mirrored \
+        -DBUILD_SINGLEDIC=ON \
+        -DBUILD_PROXYNCORR=ON \
+        -DBUILD_GEN_SUBJECT_TRIAL=ON
+
+    make -j$(nproc) cppxdic singledic proxyncorr gen_subject_trial
+    mkdir -p /opt/cppxdic/bin
+    cp cppxdic singledic proxyncorr gen_subject_trial /opt/cppxdic/bin/
+    install -m 0755 /opt/cppxdic/src/deploy/entrypoint.sh /opt/cppxdic/bin/entrypoint.sh
+
+    # Cleanup build artifacts to reduce image size
+    rm -rf /opt/cppxdic/src/build
+
+%files
+    # Copy the entire project source into the container at build time
+    # Run: apptainer build --fakeroot cppxdic_mirrored.sif deploy/cluster/cppxdic_mirrored.def
+    # from the CPPxDIC repo root.
+    . /opt/cppxdic/src
+
+%runscript
+    # App dispatcher: `apptainer run cppxdic_mirrored.sif <app> [args]` selects
+    # a binary (cppxdic, singledic, proxyncorr, gen_subject_trial); with no app
+    # name it falls through to cppxdic (mirrored mode) so existing invocations
+    # keep working. `apptainer run cppxdic_mirrored.sif --list-apps` lists
+    # what is baked in.
+    exec /opt/cppxdic/bin/entrypoint.sh "$@"
+
+%test
+    echo "=== Running container self-test ==="
+    for app in cppxdic singledic proxyncorr gen_subject_trial; do
+        bin="/opt/cppxdic/bin/${app}"
+        test -x "${bin}" || { echo "FAIL: ${app} binary not found"; exit 1; }
+        if ldd "${bin}" | grep -q "not found"; then
+            echo "FAIL: ${app} has missing shared libs"; ldd "${bin}" | grep "not found"; exit 1
+        fi
+        echo "OK: ${app}"
+    done
+    /opt/cppxdic/bin/entrypoint.sh --help >/dev/null || { echo "FAIL: cppxdic --help failed"; exit 1; }
+    echo "=== Self-test PASSED ==="

deploy/cluster/mirrored/REPORT.md

@@ -0,0 +1,180 @@
+# Mirrored-camera (MNG) mode — Manneback run report
+
+**Date:** 2026-09-10/11 · **Branch:** `feat/mirrored-mode-mng` (worktree `CPPxDIC-mirrored`,
+stacked on `fix/config-driven-pairs-mng-data`) · **Cluster:** Manneback
+
+Three rounds of the mirrored-camera pipeline (`XDIC_MODE=mirrored`, port of
+`stepD_2DDIC_MNG.m`) were run on the real 4-camera mirror rig dataset, on two
+different node classes each. This report records what now works, how fast it is,
+and what still blocks a complete 7-pair reconstruction.
+
+---
+
+## 1. Setup
+
+| Item | Value |
+|---|---|
+| Data | `mirrored_smoke/data_root` — subject `Zhaochong`, material `index_finger`, trial 001, phase `no_phase` |
+| Videos | 4 physical cameras, 1936×1216 @ 50 fps, 837 frames; each frame carries two mirror views side by side |
+| Geometry | `mirrored_cam_order = 2,1,4,3`, `mirrored_num_pair = 7` (8 logical views → 7 overlapping stereopairs) |
+| REF masks/seeds | present for pairs **1, 3, 5, 7 only** (odd pairs); masks are 1216×968, seeds are single points |
+| DIC parameters | radius 25 (tracking) / 30 (matching), spacing 10, `LIMIT_GRAYSCALE` 120, image filtering on, `NO_UPDATE`, `fixed_step_ref = 300` |
+| Seed gates | tracking `cutoff_max_corrcoef = 0.5`, matching `matching_cutoff_max_corrcoef = 1.0`, `cutoff_max_diffnorm = 0.1` |
+| Engine | CppNCorr (`dic_engine = ncorr`), 32 threads, `parallel_processing = true` |
+| Image | `cppxdic_mirrored*.sif`, CPU-only build of `cppxdic_mirrored.def` (`-DXDIC_MODE=mirrored`), ~612 MB |
+
+The two node classes are **keira** (Milan, 32 cores allocated) and the **gpu**
+partition (IceLake node, 32 cores allocated). The image contains no CUDA, so the
+gpu-partition run is a *node-class* comparison, not a CPU-vs-GPU comparison.
+
+---
+
+## 2. Rounds
+
+| Round | Frames | Change under test | Jobs (cpu / gpu-partition) |
+|---|---|---|---|
+| v1 | 1..800 | first run of the reworked mode | 9638921 / 9638922 |
+| v2 | 1..800 | + matching-seed retry from the ROI centroid, `fixed_step_ref = 300` | 9639293 / 9639294 |
+| v3 | 1..700 | shorter window | 9652173 / 9652174 |
+
+---
+
+## 3. Results per stereopair
+
+Identical on both node classes in every round.
+
+| Pair | Views | v1 (800 fr) | v2 (800 fr) | v3 (700 fr) | Why |
+|---|---|---|---|---|---|
+| 1 | 1 & 2 | matching OK, tracking lost at frame 774 | same | **complete** | late finger motion leaves the fixed reference |
+| 2 | 2 & 3 | fail | fail | fail | no REF mask/seed in the dataset |
+| 3 | 3 & 4 | fail | fail | fail | matching seed does not converge (user seed **and** ROI centroid) |
+| 4 | 4 & 5 | fail | fail | fail | no REF mask/seed |
+| 5 | 6 & 5 | **complete** | **complete** | **complete** | — |
+| 6 | 7 & 6 | fail | fail | fail | no REF mask/seed |
+| 7 | 8 & 7 | **complete** | **complete** | **complete** | — |
+
+**Score: 3 of the 4 pairs that actually have a mask now reconstruct end to end** (v3),
+against 0 of 7 in the July 2026 smoke test.
+
+Output of a complete pair (v3): `myDIC2DpairResults_C_<v1>_C_<v2>.mat` plus the three
+`ncorr*.bin` caches.
+
+| Pair | Reference points | Delaunay triangles | `myDIC2DpairResults` size |
+|---|---|---|---|
+| 1 (views 1&2) | 2 105 | 4 146 | 72 MB |
+| 5 (views 6&5) | 2 558 | 5 034 | 87 MB |
+| 7 (views 8&7) | 3 349 | 6 647 | 114 MB |
+
+---
+
+## 4. Performance (v3, 700 frames, 32 threads)
+
+| Metric | keira (mb-mil015) | gpu partition (mb-icg101) |
+|---|---|---|
+| Wall clock | 2 h 46 min | 2 h 56 min |
+| Peak RSS | 35.7 GB | 34.8 GB |
+
+Per-pair breakdown on keira:
+
+| Phase | Time |
+|---|---|
+| View extraction (2 views × 700 frames, first pair) | ~3 min |
+| view1→view2 matching (2 frames) | ~30 s |
+| Tracking view 1 (700 frames) | ~24 min |
+| Tracking view 2 (699 frames) | ~24 min |
+| **Complete pair total** | **~51 min** |
+| Failed pair (extraction + failed matching) | ~3 min |
+
+Round totals were 2 h 44 (v1) and 2 h 39 (v2) at 800 frames against 2 h 46 (v3) at
+700 frames: v3 is *slower* despite the shorter window because pair 1 now runs its
+tracking to completion instead of aborting at frame 774.
+
+**Determinism.** Every `ncorr*.bin` of the three complete pairs is byte-identical
+between the keira run and the gpu-partition run, so the mode is reproducible across
+node classes. Outputs are *not* comparable across rounds: an 800-frame and a
+700-frame tracking differ from the first reference update onwards.
+
+**Frame cache.** Views are shared between adjacent stereopairs (view 2 serves pairs
+1 and 2), and the per-`(trial, view)` PNG cache with its window-signature marker
+means each view is decoded once per run. Cost: 6.5 GB per output tree.
+
+---
+
+## 5. What was fixed to get here
+
+The July 2026 smoke test failed on the very first pair with
+`matlab_DIC_analysis_sequential could not seed any current image`, and the run died
+there. Diagnosis at the time (a missing cross-view seed algorithm) was **wrong**.
+
+1. **Seed-quality gate, not a missing algorithm.** The view1→view2 matching seed
+   *converges* (47 iterations, diffnorm 9e-7) at corrcoef **0.51**; CppNCorr's default
+   gate `cutoff_max_corrcoef = 0.5` rejected it. Verified offline with `proxyncorr`
+   on the extracted view halves. A horizontally flipped view 2 diverges (corrcoef
+   1.28), which also proves the mirror views are **not** reversed images.
+2. **The gates were never wired.** `cutoff_max_corrcoef`, `cutoff_max_diffnorm` and
+   `seeds_are_optimized` were parsed from `ncorr_params.txt` but never copied into
+   `DIC_analysis_parallel_input`. Inter-view matching now has its own, looser gate
+   (`matching_cutoff_max_corrcoef`, default 1.0) because two mirror views correlate
+   worse than two consecutive frames by nature.
+3. **One bad pair no longer kills the run.** `processPair()` is wrapped in try/catch,
+   so pairs 2, 3, 4 and 6 fail individually and the run continues to 5 and 7.
+4. **Engine parity with the camerapairs path.** `runViewDic()` now uses the parallel
+   engine when `parallel_processing` is set, honours `NO_UPDATE`, `fixed_step_ref`
+   and the seed gates, and reloads existing `.bin` checkpoints.
+5. **Trial selection, saturation flags, frame cache.** `run(config, trials)` honours
+   `--trial`/`--trials`; saturation honours `im_saturation_mode` / `limit_grayscale`;
+   the per-view frame cache avoids re-decoding shared views.
+6. **Matching-seed fallback.** If the drawn seed does not converge, the ROI centre of
+   mass is tried once (helps the camerapairs rig; does not rescue pair 3 here).
+
+---
+
+## 6. Remaining issues
+
+Ranked by what blocks a full 7-pair reconstruction.
+
+1. **Pairs 2, 4 and 6 have no REF mask or seed** (data, not code). Without them the
+   code falls back to a full-frame ROI and a centre seed, which cannot match. The
+   MATLAB script prompts the operator to draw them (`draw_ref_roi_MNG.m`,
+   `draw_ref_seed_MNG.m`); the C++ path has no interactive drawing, so the four
+   missing masks must be produced with the MATLAB tool or `xdic_stepsABC`.
+   **This is the single biggest gap: 4 of 7 pairs are simply not set up.**
+2. **Pair 3 (views 3 & 4) does not match** even with a converging-seed retry from the
+   ROI centroid. Both candidate seeds fail, so either the pair-3 mask covers a region
+   the two views do not share, or the seed sits on a poorly textured area. Needs a
+   look at the mask against both view halves before any further code change.
+3. **Pair 1 tracking is window-limited.** At 800 frames the fixed reference is lost at
+   frame 774; at 700 frames it completes. Neither `fixed_step_ref = 300` nor the
+   0.5 seed gate rescues the tail. Whoever analyses this trial should either cap the
+   window or accept a shorter pair 1.
+4. **No Step E for the MNG rig.** `stepE_3DReconstruction_MNG.m` (7 overlapping pairs,
+   different stitch topology from the 2-pair camerapairs rig) is not ported, so the
+   pipeline stops at 2D pair results. Note that the camerapairs geometric stitcher is
+   already pathologically slow on a 4-pair rig, so a 7-pair MNG stitch will need the
+   `step_e_stitch_mode = simple` path or an accelerated overlap removal.
+5. **Exit code.** The run returns 1 whenever any pair fails, which is correct but means
+   a partially successful run looks like a failure to SLURM. Consider a distinct code
+   (or a summary line) for "some pairs succeeded".
+
+---
+
+## 7. Reproducing
+
+```bash
+# on Manneback, from the synced worktree (no .git; .build_commit carries the SHA)
+cd /globalscratch/ucl/inma/jaoga/mirrored_run
+sbatch build_mirrored_v3.sbatch                       # ~8 min
+MODE=cpu sbatch -p keira        run_mirrored_v3.sbatch
+MODE=gpu sbatch -p gpu --gres=gpu:1 run_mirrored_v3.sbatch
+```
+
+Locations:
+
+| What | Where (Manneback) |
+|---|---|
+| Source | `/home/users/j/a/jaoga/devlab/MultiDIC/CPPxDIC-mirrored` |
+| Image | `mirrored_run/cppxdic_mirrored_v3.sif` |
+| Configs | `mirrored_run/configs/` |
+| Logs | `mirrored_run/logs/cppxdic_{cpu,gpu}_v3_<jobid>.log` |
+| Results | `mirrored_run/output_{cpu,gpu}_v3/Zhaochong/index_finger/001/no_phase/mirrored_pair<N>/` |
+| Raw data | `mirrored_smoke/raw/trial1_Zhaochong_index_finger/` |