Skip to content

feat: speedup lheh5/hdf5 read performance - #425

Merged
APN-Pucky merged 11 commits into
mainfrom
speedup_hdf5_read
Sep 25, 2026
Merged

APN-Pucky merged 11 commits into
mainfrom
speedup_hdf5_read

Conversation

@APN-Pucky

@APN-Pucky APN-Pucky commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Closes: #423

Old results:

------------------------------------------------------------------------------------------------------------- benchmark: 10 tests -------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                     Min                    Max                   Mean             StdDev                 Median                IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_lheh5_write_benchmark[uncompressed-chunks] (0002_mr)         29.4234 (1.0)          67.5390 (1.0)          32.5609 (1.0)       8.4478 (4.36)         30.4975 (1.0)       0.5033 (1.0)           2;4  30.7117 (1.0)          34           1
test_lheh5_write_benchmark[gzip-chunks] (0002_mr)                 49.0174 (1.67)         90.0368 (1.33)         54.0876 (1.66)     11.8695 (6.13)         49.8527 (1.63)      1.3506 (2.68)          2;3  18.4885 (0.60)         21           1
test_count_events_benchmark (0002_mr)                            408.1710 (13.87)       414.1582 (6.13)        410.0006 (12.59)     2.0435 (1.06)        409.3474 (13.42)     0.7619 (1.51)          2;2   2.4390 (0.08)         10           1
test_count_events_benchmark (0001_base)                          410.5380 (13.95)       415.8956 (6.16)        412.8877 (12.68)     1.9369 (1.0)         412.7353 (13.53)     3.4415 (6.84)          4;0   2.4220 (0.08)         10           1
test_lheh5_read_benchmark[uncompressed-chunks] (0002_mr)         535.1749 (18.19)       588.1780 (8.71)        551.8369 (16.95)    19.2145 (9.92)        542.9409 (17.80)     8.4004 (16.69)         2;2   1.8121 (0.06)         10           1
test_lheh5_read_benchmark[gzip-chunks] (0002_mr)                 542.3137 (18.43)       596.5376 (8.83)        554.7411 (17.04)    18.1380 (9.36)        546.7035 (17.93)     4.6060 (9.15)          2;2   1.8026 (0.06)         10           1
test_fromfile_and_to_awkward (0002_mr)                         3,013.4692 (102.42)    3,047.6631 (45.12)     3,030.6411 (93.08)    11.1551 (5.76)      3,033.1322 (99.46)    17.9928 (35.75)         4;0   0.3300 (0.01)         10           1
test_fromfile_and_to_awkward (0001_base)                       3,052.5610 (103.75)    3,083.4185 (45.65)     3,067.8272 (94.22)     8.6322 (4.46)      3,068.2745 (100.61)    9.6605 (19.19)         3;0   0.3260 (0.01)         10           1
test_write_random_events_gzip_benchmark (0002_mr)             10,092.5005 (343.01)   10,130.3326 (149.99)   10,112.5390 (310.57)   10.3742 (5.36)     10,112.0906 (331.57)    4.6982 (9.33)          3;3   0.0989 (0.00)         10           1
test_write_random_events_gzip_benchmark (0001_base)           10,166.0645 (345.51)   10,219.1718 (151.31)   10,197.5249 (313.18)   18.1485 (9.37)     10,201.0716 (334.49)   24.9134 (49.50)         4;0   0.0981 (0.00)         10           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

@APN-Pucky APN-Pucky changed the title speedup lheh5/hdf5 read performance feat: speedup lheh5/hdf5 read performance Sep 25, 2026
@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ea3ee7c) to head (ee5e7bb).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #425   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines          962       972   +10     
=========================================
+ Hits           962       972   +10     
Flag Coverage Δ
unittests-3.10 100.00% <100.00%> (ø)
unittests-3.11 100.00% <100.00%> (ø)
unittests-3.12 100.00% <100.00%> (ø)
unittests-3.13 100.00% <100.00%> (ø)
unittests-3.14 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

🏎️ Benchmark Comparison


------------------------------------------------------------------------------------------------------------- benchmark: 10 tests -------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                     Min                    Max                   Mean             StdDev                 Median                IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_lheh5_write_benchmark[uncompressed-chunks] (0002_mr)         29.0674 (1.0)          61.9733 (1.0)          31.7999 (1.0)       7.2387 (6.12)         29.9725 (1.0)       0.7284 (1.0)           2;4  31.4467 (1.0)          35           1
test_lheh5_write_benchmark[gzip-chunks] (0002_mr)                 47.8487 (1.65)         79.3209 (1.28)         51.2585 (1.61)      8.8250 (7.46)         48.5442 (1.62)      0.7667 (1.05)          2;3  19.5090 (0.62)         21           1
test_lheh5_read_benchmark[uncompressed-chunks] (0002_mr)         159.3792 (5.48)        201.4809 (3.25)        167.9834 (5.28)     15.6241 (13.21)       160.4483 (5.35)      4.9158 (6.75)          2;2   5.9530 (0.19)         10           1
test_lheh5_read_benchmark[gzip-chunks] (0002_mr)                 162.7201 (5.60)        205.1595 (3.31)        172.4368 (5.42)     15.5788 (13.18)       165.3341 (5.52)      5.1315 (7.05)          2;2   5.7992 (0.18)         10           1
test_count_events_benchmark (0001_base)                          408.9043 (14.07)       412.3340 (6.65)        410.8099 (12.92)     1.1824 (1.0)         411.0566 (13.71)     2.1318 (2.93)          5;0   2.4342 (0.08)         10           1
test_count_events_benchmark (0002_mr)                            412.1081 (14.18)       418.1326 (6.75)        414.6322 (13.04)     2.2184 (1.88)        414.1059 (13.82)     4.1433 (5.69)          4;0   2.4118 (0.08)         10           1
test_fromfile_and_to_awkward (0002_mr)                         3,037.3067 (104.49)    3,068.4941 (49.51)     3,056.0701 (96.10)    11.1983 (9.47)      3,057.0926 (102.00)   17.3868 (23.87)         5;0   0.3272 (0.01)         10           1
test_fromfile_and_to_awkward (0001_base)                       3,045.5274 (104.77)    3,080.7565 (49.71)     3,062.4232 (96.30)    12.0453 (10.19)     3,060.9624 (102.13)   21.1050 (28.98)         4;0   0.3265 (0.01)         10           1
test_write_random_events_gzip_benchmark (0001_base)           10,103.5738 (347.59)   10,223.0821 (164.96)   10,169.5731 (319.80)   36.9551 (31.25)    10,174.0484 (339.45)   35.0646 (48.14)         4;1   0.0983 (0.00)         10           1
test_write_random_events_gzip_benchmark (0002_mr)             10,107.6786 (347.73)   10,175.9603 (164.20)   10,128.1394 (318.50)   20.4513 (17.30)    10,123.1151 (337.75)   20.6520 (28.35)         2;1   0.0987 (0.00)         10           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

@APN-Pucky
APN-Pucky marked this pull request as ready for review September 25, 2026 09:56
@APN-Pucky
APN-Pucky marked this pull request as draft September 25, 2026 09:57
@APN-Pucky
APN-Pucky requested a lite review from Copilot September 25, 2026 11:35
@APN-Pucky
APN-Pucky marked this pull request as ready for review September 25, 2026 11:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Critical path and batching issues, plus empty-dataset and API compatibility concerns, remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity

Open (2)
What changed in this PR

Improves LHEH5/HDF5 read performance through batched data access and adds benchmarks and documentation.

Changes:

  • Batch event and particle reads.
  • Add NumPy dependency and reader test updates.
  • Add I/O benchmarks and format-comparison documentation.
File Summary
tests/​test_lheh5_reader.py Updates reader tests for particle-column handling.
src/​pylhe/​lheh5.py Implements batched reads; needs batch_size, empty-dataset, and API compatibility fixes.
pyproject.toml Adds NumPy dependency.
docs/​source/​index.rst Includes comparison documentation.
docs/​source/​examples/​benchmark_io_formats.py Adds benchmark utility; documented commands reference the wrong path.
docs/​source/​comparison.rst Documents benchmark results.
benchmarks/​test_lheh5_io_bench.py Adds HDF5 I/O benchmarks.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/pylhe/lheh5.py Resolved
APN-Pucky and others added 2 commits September 25, 2026 13:48
Updated example commands to reference the correct path for benchmark_io_formats.py.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

@eduardo-rodrigues eduardo-rodrigues left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks great!

Note that I did not review the script docs/source/examples/benchmark_io_formats.py.

Comment thread src/pylhe/lheh5.py Resolved
Co-authored-by: Eduardo Rodrigues <eduardo.rodrigues@cern.ch>
@APN-Pucky
APN-Pucky merged commit 13be03c into main Sep 25, 2026
17 checks passed
@APN-Pucky
APN-Pucky deleted the speedup_hdf5_read branch September 25, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve LHEH5 read speed.

3 participants