adds adaptive bitpacking kernels from cuszp2

Author: skyler-ruiter

CHANGELOG.md

@@ -17,6 +17,7 @@ Version numbers follow [Semantic Versioning](https://semver.org/).
 - `tests/helpers/stage_harness.h` `pipeline_round_trip<T>()` was unconditionally calling `cudaFree(d_dec)` on the decompress output. When the pipeline uses `pool_managed_decomp_=true` (the default), `d_dec` is owned by `MemoryPool::graph_allocations_` and freed by `Pipeline::~Pipeline()` via `cudaFreeAsync`. The harness's extra `cudaFree` was a double-free that surfaced under compute-sanitizer as `cudaErrorInvalidValue` from `cudaFreeAsync` at teardown (one error per affected test), and the sticky error propagated to the next test's `cudaGetLastError()` checks in `CompressionDAG::addStage`. Harness now branches on `pipeline.isPoolManagedDecompOutput()` — only `cudaFree`s when the caller opted into caller-owned mode via `setPoolManagedDecompOutput(false)`. `pipeline_file_round_trip<T>()` unchanged (it uses the static `Pipeline::decompressFromFile`, which always returns a `cudaMalloc`-owned pointer). Also fixed two callers in `tests/stages/test_ginterp.cpp` (GI27, GI29) that replicated the same pattern inline. Result: `test_ginterp`, `test_lorenzo`, `test_quantizer`, `test_bitpack`, `test_ans`, `test_graph_capture` all report `ERROR SUMMARY: 0 errors` under `compute-sanitizer --tool=memcheck` (was 1+ pre-existing errors on most). `test_huffman` still has a pre-existing kernel-side issue unrelated to ownership.
 
 ### Added
+- `AdaptiveBitpackStage<T>` (`modules/coders/adaptive_bitpack/`, `StageType::ADAPTIVE_BITPACK = 24`) — a per-block adaptive fixed-rate bit-plane coder: the cuSZp lossless back-end's "plain" mode as a modular stage. For each block of `setBlockSize(n)` signed elements (`int16_t`/`int32_t`; default n=32, range [1,1024]) it emits a rate byte (bit width of the block's largest magnitude) and, when non-zero, a sign bitmap + that many bit-planes (`ceil(n/8)` bytes each). Per-block payload offsets come from an ordinary CUB `DeviceScan::ExclusiveSum` over per-block byte costs — **not** cuSZp's fused decoupled look-back scan, which is a fusion-time lowering left to the downstream compiler. Single signed-codes input → single `uint8` archive (rate region + scan-packed payloads); `block_size`+`num_elements` live in the 16-byte FZM header (the archive needs no internal header). Kernels are one-thread-per-block for verifiability; **independent reimplementation — no cuSZp source copied** (THIRD_PARTY cuSZp section added). Not graph-compatible (forward D2H for the scanned length, like `BitplaneRLE`). Completes the modular cuSZp front-to-back pipeline `QuantizerStage(linear) → LorenzoStage(setBlockSize) → AdaptiveBitpackStage`. Also supports cuSZp2's per-block **outlier selection** via `setOutlierSelection(true)`: each block independently picks the cheaper of plain packing vs. extracting element 0 as a raw 1..`sizeof(T)`-byte outlier and packing only the rest (targets non-sparse data where the block-local predictor's first element is a full-magnitude delta-vs-0). Per-block metadata widens from 1 to 2 bytes (`[rate][sel]`) so the full `int32` rate range stays representable — cuSZp bit-packs the same flags into one byte; we keep them separate. The mode is carried in the FZM header. Registered in `stage_factory.h`, `fzgpumodules.h`, and TOML (`type = "AdaptiveBitpack"`, `input_type`, `block_size`, `outlier_selection`). `tests/stages/test_adaptive_bitpack.cpp` — 19 tests (int16/int32 round-trips, all-zero blocks, every fixed-rate 0–31, partial final block, INT_MIN/sign extremes, header round-trip, `setBlockSize` bounds, port/type contract, graph-compat=false, compression-ratio sanity, the full `Quantizer(linear)→Lorenzo(block=32)→AdaptiveBitpack` float pipeline, and outlier-mode round-trips covering every outlier byte-count, outlier-beats-plain, partial/single blocks, INT_MIN outlier, header flag, and the outlier-mode pipeline); all pass and compute-sanitizer-clean (0 errors). Docs: `docs/stages/adaptive_bitpack.md`; design notes in `memory/cuszp_stages.md` §7.3–7.4.
 - `LorenzoStage::setBlockSize(uint32_t)` — an explicit, launch-decoupled 1-D block-local reset period (cuSZp-style; cuSZp uses 32). `n=0` (default) keeps the existing behavior (N-D inclusion-exclusion delta selected by `dims_`; the 1-D path already reset per 256-thread launch block). `n>0` forces the 1-D path over the flattened array and restarts the prediction chain every `n` elements regardless of launch config or `dims_`. Implemented by launching the existing `lorenzo_delta_1d_kernel` / `lorenzo_scan_1d_kernel` with `blockDim == n` (both already reset per CUDA block), so no new kernels — the 1-D launchers gained a trailing `unsigned block_threads = 256` parameter. `n` must be in `[1, 1024]` (`setBlockSize` throws otherwise; the inverse scans one segment per CUDA block of `n` threads). Graph-compatible. New `block_size` field in `LorenzoConfig` (struct 16→20 B; `deserializeHeader` accepts legacy 16-byte headers and defaults it to 0). `tests/stages/test_lorenzo.cpp` — added LZ10–LZ14 (block round-trip, ragged final block, header round-trip incl. legacy, `setBlockSize` bounds, and a `Quantizer(linear)→Lorenzo(block=32)` cuSZp front-end pipeline tying together §7.1+§7.2); suite 14/14. Docs: `docs/stages/lorenzo.md`. See `memory/cuszp_stages.md` §7.2.
 - `QuantizerStage::setLinearMode(bool)` — a cuSZp-style linear / no-outlier quantization mode (ABS/NOA only). Maps each value to `q = round(x / (2·eb))` stored two's-complement in `TCode` with **no radius clamp, no outlier ports, no zigzag**; the forward kernel `quantizer_linear_fwd_kernel` is a pure memory-bound map (the regular forward kernel's only `atomicAdd` and only divergent branch — the outlier path — are removed), and the inverse reuses `quantizer_abs_inv_kernel<…,false>` with no memset/scatter. Single `"codes"` output (forward) / single input (inverse); the codes are *declared* as the signed DataType (`UINT16→INT16`, `UINT32→INT32`) so they connect directly to `LorenzoStage<intN>` — no new signed-`TCode` instantiations needed. A bin overflowing `TCode` wraps (no outlier fallback), so size `TCode` wide enough (use `uint32_t`). Fully graph-compatible in ABS mode (no D2H, no count readback). Mutually exclusive with REL / inplace / zigzag — each throws at the first `compress()`. New `linear_mode` byte in `QuantizerConfig` (reuses former `_pad[0]`; struct size unchanged at 36 B, old headers default it to 0). Intended front-end for the modular cuSZp pipeline `Quantizer(linear) → LorenzoStage(blockSize=32) → AdaptiveBitpackStage` (see `memory/cuszp_stages.md` §7.1). `tests/stages/test_quantizer.cpp` — added QuantizerLinear suite (5 tests: ABS/NOA round-trips, port/type contract, header round-trip, incompatible-mode throws); full suite 34/34. Docs: `docs/stages/quantizer.md`.
 - `fzgmod-cli --report-json <file>` — writes a standalone, machine-readable JSON report for the benchmarking harness to parse (compress/decompress/benchmark modes). New `src/utils/cli/report_json.{h,cpp}` with a plain-data `ReportData` model decoupled from CLI internals; the writer is the single source of truth for derived values (ratio, bitrate, throughput, median/min/max) while raw counts (`original_bytes`, `compressed_bytes`, `num_elements`) are always emitted so the harness can recompute. Schema (`schema_version` 1.0): `tool` (name/version/git_sha, injected via CMake `FZGMOD_VERSION`/`FZGMOD_GIT_SHA` defines), `status`+`error_message` (a forced failure still emits valid JSON with a non-zero exit, via `emit_error_report` on the top-level catch), `config` echo, `size`, `timing` (per-rep `device_ms`/`host_wall_ms` arrays with `timing_method: cuda_events_dag`, labeled honestly now that `dag_elapsed_ms` is event-based), derived `throughput` (device-median basis), `memory`, optional `quality` (when an original is available), and per-stage `stages[]`. `--report-json` forces profiling on (without enabling the stdout perf table) so device timing + stage breakdown are populated even for single-shot compress/decompress. `tests/test_cli.cpp` — added `ReportJsonCompressEmitsValidFile` (asserts schema/status/raw-count fields + stage block) and `ReportJsonErrorStillEmitsValidJsonAndNonZeroExit`; full CLI suite 7/7. Spec: `memory/report_json_spec.md`.

CMakeLists.txt

@@ -243,6 +243,8 @@ add_library(fzgmod_modules
   modules/coders/rle/rle.cu
   modules/coders/rze/rze_stage.cu
   modules/coders/bitpack/bitpack_stage.cu
+  modules/coders/adaptive_bitpack/adaptive_bitpack_kernels.cu
+  modules/coders/adaptive_bitpack/adaptive_bitpack_stage.cu
   modules/shufflers/bitshuffle/bitshuffle_stage.cu
   modules/fused/lorenzo_quant/lorenzo_quant.cu
   modules/fused/lorenzo_quant/lorenzo_quant_nd.cu

THIRD_PARTY.md

@@ -164,6 +164,35 @@ the [cuSZ / PHF](#cusz--phf) section above.
 
 ---
 
+## cuSZp
+
+**Used by:** `AdaptiveBitpackStage` (`modules/coders/adaptive_bitpack/`),
+and the `linear` mode of `QuantizerStage` + the `setBlockSize` option of
+`LorenzoStage`.
+
+**Relationship:**
+- These are **independent reimplementations of published cuSZp schemes — no
+  cuSZp source code is copied.** `AdaptiveBitpackStage` implements cuSZp's
+  per-block adaptive fixed-rate bit-plane "plain" encoding with a byte-granular
+  layout, one-thread-per-block kernels, and an ordinary CUB `DeviceScan` for the
+  per-block offsets (cuSZp uses a fused single kernel with a decoupled look-back
+  scan; that fusion is left to a downstream compiler). `QuantizerStage`'s linear
+  mode reproduces cuSZp's `q = round(x / 2·eb)` mapping with no radius/outlier
+  fallback, and `LorenzoStage::setBlockSize` reproduces cuSZp's block-local 1-D
+  delta. The reference codebase (for cross-checking only, not vendored) lives at
+  `compressors/cuSZp2/`; design notes are in `memory/cuszp_stages.md`.
+
+  Original authors: Yafan Huang et al.
+  Papers: Yafan Huang, Sheng Di, Xiaodong Yu, Guanpeng Li, Franck Cappello,
+  "cuSZp: An Ultra-fast GPU Error-bounded Lossy Compression Framework with
+  Optimized End-to-End Performance", SC '23; and "cuSZp2: A GPU Lossy Compressor
+  with Extreme Throughput and Optimized Compression Ratio", SC '24.
+
+**License:** cuSZp is BSD-3-Clause (Argonne National Laboratory / University of
+Iowa). As no source is copied, this is an algorithmic attribution.
+
+---
+
 ## cuSZ-Hi
 
 **Used by:** `GInterpStage` (`modules/fused/ginterp/`)

docs/stages/adaptive_bitpack.md

@@ -0,0 +1,133 @@
+# AdaptiveBitpackStage {#stage_adaptive_bitpack}
+
+**Header:** `modules/coders/adaptive_bitpack/adaptive_bitpack_stage.h`
+**Class:** `fz::AdaptiveBitpackStage<T>`
+**Category:** Coder (lossless)
+
+---
+
+## What it does
+
+Per-block adaptive fixed-rate bit-plane coder — the cuSZp lossless back-end's
+"plain" mode, as a modular stage. It partitions the signed input into fixed-size
+blocks and, for each block, stores only as many bit-planes as the largest
+magnitude in that block requires:
+
+- **rate byte** `r` — the bit width of the largest `|value|` in the block (0 if
+  the block is all zeros).
+- when `r > 0`: a **sign bitmap** (`ceil(block_size/8)` bytes) followed by `r`
+  **bit-planes** of `ceil(block_size/8)` bytes each. Bit `j` of byte `k` of
+  plane `p` is bit `p` of `|element 8k+j|`.
+
+Per-block payloads are concatenated using a device-wide exclusive scan of the
+per-block byte costs, preceded by the array of rate bytes. The archive carries no
+internal header of its own — `block_size` and `num_elements` live in the FZM
+stage header.
+
+Unlike cuSZp's fused single kernel, the offset scan here is an ordinary CUB
+`DeviceScan` rather than the cuSZp decoupled look-back scan: fusing predictor +
+quantizer + coder into one kernel (and lowering the scan to a single pass) is a
+job for the downstream compiler, not the stage.
+
+---
+
+## Template parameter
+
+`T` — signed element type: `int16_t` or `int32_t` (quantizer codes or block
+deltas).
+
+---
+
+## Stage settings
+
+| Setting | Purpose | Notes |
+|---|---|---|
+| `setBlockSize(n)` | Elements per block (fixed-rate granularity) | `[1, 1024]`; default 32 (cuSZp) |
+| `setOutlierSelection(b)` | cuSZp2 per-block plain/outlier selection | off by default; see below |
+
+```cpp
+auto* ab = p.addStage<AdaptiveBitpackStage<int32_t>>();
+ab->setBlockSize(32);
+ab->setOutlierSelection(true);   // cuSZp2 mode (optional)
+```
+
+### Outlier selection (cuSZp2)
+
+With `setOutlierSelection(true)`, each block independently chooses the cheaper of
+two encodings:
+
+- **plain** — pack all elements (as above), or
+- **outlier** — store element 0 separately as a raw 1..`sizeof(T)`-byte magnitude
+  and pack only elements 1..n-1.
+
+This targets non-sparse, high-smoothness data: with a block-local predictor the
+first element of each block is a delta-vs-0 (a full magnitude) that would inflate
+the whole block's bit width. Per-block metadata grows from 1 to 2 bytes
+(`[rate][sel]`, where `sel` bit 0 = is-outlier and bits 1-2 = outlier byte count
+− 1). The mode is recorded in the FZM header, so a cold decompress selects the
+right path automatically. This is the cuSZp **outlier** mode (SC'24); cuSZp packs
+the same flags into a single rate byte, which we widen to two bytes so the full
+`int32` rate range stays representable.
+
+---
+
+## Ports
+
+Single input → single output.
+
+| Direction | Port | Type |
+|---|---|---|
+| Forward in / inverse out | `"output"` | `T[n]` (signed codes) |
+| Forward out / inverse in | `"output"` | `uint8_t[]` (archive) |
+
+---
+
+## Graph compatibility
+
+`isGraphCompatible()` is **false** — the forward path does a host-blocking D2H to
+read the scanned total payload length (same pattern as `BitplaneRLEStage`).
+
+---
+
+## Typical pipeline (cuSZp-style)
+
+```cpp
+auto* quant = p.addStage<QuantizerStage<float, uint32_t>>();
+quant->setErrorBound(1e-3f);
+quant->setErrorBoundMode(ErrorBoundMode::ABS);
+quant->setLinearMode(true);          // signed INT32 codes, no outliers
+
+auto* lrz = p.addStage<LorenzoStage<int32_t>>();
+lrz->setBlockSize(32);               // block-local 1-D delta
+
+auto* ab = p.addStage<AdaptiveBitpackStage<int32_t>>();
+ab->setBlockSize(32);
+
+p.connect(lrz, quant, "codes");
+p.connect(ab,  lrz);
+p.finalize();
+```
+
+`block_size` on the coder need not match the Lorenzo block, but for faithful
+cuSZp both are 32.
+
+---
+
+## TOML
+
+```toml
+[[stage]]
+type = "AdaptiveBitpack"
+input_type = "int32"   # or "int16"
+block_size = 32
+outlier_selection = false   # true = cuSZp2 per-block plain/outlier selection
+```
+
+---
+
+## Acknowledgements
+
+The per-block adaptive fixed-rate bit-plane scheme is from cuSZp (Yafan Huang et
+al., SC'23/SC'24). This is an independent reimplementation of the published
+scheme — no cuSZp source is copied. See `THIRD_PARTY.md` and
+`memory/cuszp_stages.md`.

docs/stages/coders.md

@@ -7,3 +7,4 @@
 | \subpage stage_rle | Run-length encoding |
 | \subpage stage_rze | Recursive zero-byte elimination |
 | \subpage stage_bitpack | Dense bit-packing of fixed-width integers |
+| \subpage stage_adaptive_bitpack | Per-block adaptive fixed-rate bit-plane coding (cuSZp plain mode) |

include/fzgpumodules.h

@@ -22,6 +22,7 @@
 #include "predictors/lorenzo/lorenzo_stage.h"
 #include "coders/rle/rle.h"
 #include "coders/bitpack/bitpack_stage.h"
+#include "coders/adaptive_bitpack/adaptive_bitpack_stage.h"
 #include "coders/rze/rze_stage.h"
 #include "fused/lorenzo_quant/lorenzo_quant.h"
 #include "quantizers/quantizer/quantizer.h"

include/fzm_format.h

@@ -94,6 +94,7 @@ enum class StageType : uint16_t {
     ADM        = 19,   ///< Adaptive Data Mapping transform (MANS)
     G_INTERP   = 22,   ///< Spline interpolation predictor + quantizer (cuSZ-Hi G-Interp)
     BITPLANE_RLE = 23, ///< Fused bitplane transpose + zero-byte RLE (FZ-GPU lossless encoder)
+    ADAPTIVE_BITPACK = 24, ///< Per-block adaptive fixed-rate bit-plane coder (cuSZp plain mode)
 };
 
 /**
@@ -319,6 +320,7 @@ inline std::string stageTypeToString(StageType type) {
         case StageType::ADM:  return "ADM";
         case StageType::G_INTERP:    return "GInterp";
         case StageType::BITPLANE_RLE: return "BitplaneRLE";
+        case StageType::ADAPTIVE_BITPACK: return "AdaptiveBitpack";
         default:                     return "Unknown";
     }
 }

include/stage/stage_factory.h

@@ -16,6 +16,7 @@
 #include "shufflers/bitshuffle/bitshuffle_stage.h"
 #include "coders/rze/rze_stage.h"
 #include "coders/bitpack/bitpack_stage.h"
+#include "coders/adaptive_bitpack/adaptive_bitpack_stage.h"
 #include "coders/huffman/huffman_stage.h"
 #include "coders/ans/ans_stage.h"
 #include "transforms/adm/adm_stage.h"
@@ -297,6 +298,20 @@ inline Stage* createStage(StageType type, const uint8_t* config, size_t config_s
             break;
         }
 
+        case StageType::ADAPTIVE_BITPACK: {
+            // config[0] holds the DataType of T (INT16 / INT32).
+            DataType dt = (config_size > 0)
+                ? static_cast<DataType>(config[0])
+                : DataType::INT32;
+            if      (dt == DataType::INT16) stage = new AdaptiveBitpackStage<int16_t>();
+            else if (dt == DataType::INT32) stage = new AdaptiveBitpackStage<int32_t>();
+            else throw std::runtime_error(
+                    "Unsupported AdaptiveBitpackStage DataType: "
+                    + std::to_string(static_cast<int>(dt)));
+            stage->deserializeHeader(config, config_size);
+            break;
+        }
+
         default:
             throw std::runtime_error("Unknown stage type: "
                 + std::to_string(static_cast<uint16_t>(type)));