Feature request: add chunked + optionally compressed single-file HDF5 output (`h5chunked`)

[decarlof]

Summary

tomocupy currently offers five reconstruction output formats: tiff, h5, h5sino, h5nolinks, zarr. None of them combines single-file convenience with chunked storage and optional compression. The closest options either sacrifice access performance and disk usage (h5nolinks, current default), or sacrifice the single-file layout (h5, h5sino, zarr).

Proposal: add a new --save-format h5chunked option that writes one .h5 file with a chunked, optionally compressed /exchange/data dataset. Chunks and compression should be configurable via existing-style options (--h5-chunk, --h5-compression).

Current landscape (as of tomocupy 1.1.0, src/tomocupy/dataio/writer.py)

Format	On-disk layout	Chunked?	Compressed?	Random access	Single file?	Notes
tiff	_rec/recon_NNNNN.tiff (many files)	N/A	per-tiff LZW possible	fast on Z-slices only	✗	Interoperable with any viewer; huge inode/file count.
h5	.h5 + _parts/pNNNN.h5 (VDS + parts)	✓ (1, n, n) per part (writer.py:298)	✗	fast on Z-slices	logically ✓ (one master), physically ✗	Virtual dataset points at chunked parts; no compression.
h5sino	.h5 + _parts/pNNNN.h5 (VDS + parts)	✓ (nproj, 1, n) per part (writer.py:305)	✗	good for re-reconstruction	logically ✓, physically ✗	Sinogram-oriented chunks.
h5nolinks (default)	one .h5	✗ (contiguous) (writer.py:178–179 has no chunks= argument)	✗	fast on Z-slices, slow on Y/X planes	✓	Simplest layout, worst storage/access trade-off.
zarr	.zarr/ directory (many small files)	✓ 8,64,64 cube chunks	✓ blosclz	fast in every orientation	✗	Also builds a multi-resolution pyramid up to 6 levels; the recommended big-data format.

Concrete example from a recent 2-BM dataset

The same volume saved as h5nolinks currently:

Pinkshrimp_stained_5x_505mm_test_113_rec.h5     101 GB, contiguous float32, no compression

Same volume in zarr with --zarr-compression blosclz --zarr-chunk 8,64,64 typically lands in the 25–40 GB range for tomography floats. A chunked-HDF5 write with the same chunking and compression codec should land in the same ballpark.

What the current h5nolinks writer does

writer.py:171–198:

elif args.save_format == 'h5nolinks':
    fnameout += '.h5'
    h5w = h5py.File(fnameout, "w")
    z_dim = params.rh if args.lamino_angle != 0 else int(params.nzi/2**args.binning)
    dset_rec = h5w.create_dataset("/exchange/data",
                                  shape=(z_dim, params.n, params.n),
                                  dtype=params.dtype)   # ← no chunks=, no compression=
    ...

The create_dataset call takes no chunks= and no compression=, so h5py defaults to contiguous storage with the entire dataset pre-allocated. Once contiguous, compression is off-limits (h5py rejects it).

write_data_chunk() at writer.py:299–300 writes each Z-strip in place:

elif args.save_format == 'h5nolinks':
    self.h5w['/exchange/data'][st:end, :, :] = rec[:end-st]

Proposal: h5chunked

A new save-format that mirrors h5nolinks's single-file layout but with chunking and optional compression.

New CLI options

Two new options in SECTIONS['output'] (config.py):

'h5-chunk': {
    'default': '8,64,64',
    'type': str,
    'help': "Chunk shape (z,y,x) for --save-format h5chunked. "
            "Use small cube-ish chunks (e.g. 8,64,64) for good access in every orientation, "
            "or Z-slice chunks (e.g. 1,y,x) for slice-oriented reads."},

'h5-compression': {
    'default': 'blosc:lz4',
    'type': str,
    'help': "Compression codec for --save-format h5chunked. "
            "Any hdf5plugin filter or 'gzip', 'lzf', 'none'. "
            "'blosc:lz4' gives ~3-5x on tomography floats with negligible CPU cost."},

And save-format gains a new choice:

'choices': ['tiff', 'h5', 'h5sino', 'h5nolinks', 'h5chunked', 'zarr']

Writer changes

writer.py — new branch that is nearly identical to h5nolinks but chunked:

elif args.save_format == 'h5chunked':
    fnameout += '.h5'
    h5w = h5py.File(fnameout, "w")
    z_dim = params.rh if args.lamino_angle != 0 else int(params.nzi/2**args.binning)

    chunks = tuple(int(c.strip()) for c in args.h5_chunk.split(','))
    compression_kwargs = _parse_compression(args.h5_compression)

    dset_rec = h5w.create_dataset(
        "/exchange/data",
        shape=(z_dim, params.n, params.n),
        dtype=params.dtype,
        chunks=chunks,
        **compression_kwargs,
    )
    # ... same attribute copying and write_meta() as h5nolinks ...
    self.h5w = h5w
    self.dset_rec = dset_rec
    config.update_hdf_process(fnameout, args, sections=config.RECON_STEPS_PARAMS)

Where _parse_compression() translates 'blosc:lz4', 'gzip:4', 'lzf', 'none' etc. into the appropriate compression=/compression_opts= kwargs (using hdf5plugin for blosc).

write_data_chunk() needs no change — the chunked path handles strip writes exactly like the contiguous one.

Add hdf5plugin as an optional dep

Blosc/lz4/zstd on HDF5 requires hdf5plugin. Add it to requirements.txt (or make it optional and fall back to gzip if not installed).

Comparison after the change

Format	Single file	Chunked	Compressed	Random access	Disk size (typical tomo float32)
h5nolinks (today)	✓	✗	✗	Z-slice only	1.0× (baseline, 101 GB in our example)
h5chunked (proposed)	✓	✓	✓ (blosc:lz4 default)	good in every orientation	~0.3–0.4× (est. 30–40 GB)
h5 (with links)	✗ (parts dir)	✓	✗	good on Z	~1.0×
zarr	✗ (dir)	✓	✓	best (cube chunks + pyramid)	~0.3–0.4×

The proposed format fills the empty cell in the matrix: "single file, chunked, compressed."

Motivation

• Disk pressure. Beamline datasets are getting bigger. A 3× reduction on ~100 GB per reconstruction is meaningful when we produce many per session.
• Downstream access patterns. Many analysis tools (napari, dask, ITK, dxchange) work with HDF5 out of the box but not zarr. Chunked HDF5 gives us fast orthogonal cuts without asking users to switch container formats.
• Single-file convention. At 2-BM we keep one _rec.h5 per scan for archival/transfer simplicity. zarr breaks that convention with a directory of thousands of small files, which is painful over network storage.
• Backward-compatible. Existing scripts reading _rec.h5 continue to work — h5py reads chunked-and-compressed datasets transparently, and the group/attribute layout is unchanged.

Non-goals

• Not asking to change the default (h5nolinks) — that's a separate discussion once h5chunked has been in the wild for a while.
• Not proposing a multi-resolution pyramid in HDF5 — if you need pyramids, use zarr.

Open questions for reviewers

• Are the default chunk shape 8,64,64 and codec blosc:lz4 reasonable? These match the zarr defaults, but a Z-slice-oriented default (e.g. 1,2426,3232) might be a better match for how most users read reconstructed volumes.
• Should hdf5plugin be a hard dependency, or optional (fallback to gzip if missing)?
• Any interest in also adding --h5nolinks a chunks opt-in, instead of introducing a new format name? Two names is clearer IMO but doubles the maintenance surface.