Skip to content

fix: read complete event rows from LHEH5 datasets - #437

Merged
APN-Pucky merged 1 commit into
scikit-hep:mainfrom
dhairyajangir:fix/lheh5-complete-event-rows
Nov 12, 2026
Merged

APN-Pucky merged 1 commit into
scikit-hep:mainfrom
dhairyajangir:fix/lheh5-complete-event-rows

Conversation

@dhairyajangir

@dhairyajangir dhairyajangir commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

read_iter_events treats each physical HDF5 chunk as a complete event row. When /events is chunked across columns, a slice such as chunks=(2, 3) supplies only part of each row, so event fields can be wrong and iteration can fail. An empty file produced by the public writer also fails because iter_chunks() rejects the zero-row selection.

Read complete rows in batches using the chunk's row extent, retaining the existing 8192-row fallback for contiguous datasets. This also lets empty datasets yield no events without entering the batch loop.

Validation on Python 3.12.15, NumPy 2.5.4 and h5py 3.16.0:

  • All 32 tests in the complete LHEH5 reader and writer modules pass.
  • The ten layout and empty-file cases produce six failures and four passing controls on unchanged main. They check complete event information, particles, weights, scales, trials and order across contiguous, row-chunked and column-chunked storage, plus empty compressed/uncompressed roundtrips, through both lazy and eager public readers.
  • A separate test checks that consuming the first event keeps reads lazy and batched.
  • Pinned Ruff 0.17.3 lint/format, mypy 2.4.1 over all four source files, and git diff --check pass.

AI assistance: I used Codex to investigate the defect, prepare the implementation and tests, and run the checks.

The five fork workflows currently need maintainer approval. Could a maintainer authorize them?

Physical HDF5 chunks can split event columns and cannot iterate an empty
dataset. Read complete rows using the row-chunk extent, retaining the
8192-row contiguous fallback.

Cover event content and ordering across storage layouts, compressed and
uncompressed empty-file roundtrips, and lazy row-batched reads.

AI assistance: Codex helped investigate, implement and test this change.
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (5d640dc) to head (7bdb515).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #437   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines          972       968    -4     
=========================================
- Hits           972       968    -4     
Flag Coverage Δ
unittests-3.10 100.00% <100.00%> (ø)
unittests-3.11 100.00% <100.00%> (ø)
unittests-3.12 100.00% <100.00%> (ø)
unittests-3.13 100.00% <100.00%> (ø)
unittests-3.14 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the LHEH5 reader changes addressing column-chunked datasets and empty files.

No open issues 🎉

What changed in this PR

Fixes LHEH5 event reading to handle datasets chunked across columns and empty files while preserving lazy, batched reads.

Changes:

  • Reads complete event rows using the chunk’s row extent, retaining the 8192-row fallback for contiguous datasets.
  • Adds regression coverage for storage layouts, empty-file roundtrips, and lazy batch reading.
File Description
tests/​test_lheh5_writer.py Adds layout, empty-file, and lazy-reading regression tests.
src/​pylhe/​lheh5.py Replaces physical chunk iteration with complete-row batches and removes the unused NumPy import.

Review effort: Balanced

@APN-Pucky

APN-Pucky commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

LGTM, I did not know there could be chunks like (2,3). Thanks!

@APN-Pucky
APN-Pucky merged commit d5a6eb7 into scikit-hep:main Nov 12, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants