feat(epoch): bound memory during epoch processing (ADR-048)

[satran004]

Summary

Implements ADR-048's bounded-memory epoch-boundary design and the accepted follow-up optimizations from #97 and #98.

The change replaces mainnet-sized boundary materialization and random-read hot paths with ordered, bounded processing while preserving ledger ordering, rollback semantics, fail-closed fallbacks, and calculation outputs.

Main changes

• stream stake balances from the coordinate-bound utxo_stake_balance index
• maintain a rollback-exact pointer-address UTXO index for pre-Conway boundaries, with a full-scan fallback when readiness cannot be proven
• stream reward inputs and outputs in bounded chunks with resumable boundary/rollback journal state
• use credential-ordered account and delegation merges for snapshots, rewards, and DRep distribution
• bound the decoded ledger-apply backlog, drain it before boundaries, and cooperatively fence body fetch so the network event loop does not block
• apply explicit native heap and RocksDB cache/write-buffer/background-job budgets
• preserve era-specific reward rules, pointer semantics, snapshot validity, and governance ordering
• add phase, heap, RSS, RocksDB/SST, queue, path-selection, and DRep telemetry,
  exposed through /api/v1/node/epoch-calc-status
• add an explicit default | low-memory resource profile; individual settings
  always override profile defaults and there is no host-memory auto-detection

Closes #97.
Advances #98.
Related follow-ups: #99 and #100.

Correctness and recovery

• byte/golden parity tests cover streaming rewards, ordered stake snapshots, pointer overlays, DRep distribution, chunked persistence, and rollback behavior
• pointer-index differential validation passed on retained pre-Conway checkpoints, including rebuild and shadow-scan comparisons
• ordered DRep merge matched the point-lookup baseline for all 866 DRep rows on the same Conway state, including virtual DReps
• startup, rollback, cancellation, queue-drain, cooperative backpressure, and interrupted-boundary paths have focused regression coverage
• chainstate format/readiness checks fail closed; unsupported non-empty preview stores require backup/resync instead of a partial migration

A deliberate live Conway crash-after-SNAP drill was completed on a disposable
epoch-532 clone on 2026-08-30. The process was killed after the durable snapshot
write and before COMPLETE; restart resumed at step 3 without repeating rewards
or snapshot, completed with exact AdaPot, and took 2.5 seconds. Evidence:
#97 (comment) and
#101 (comment).

Test evidence

• pointer-index gate (33a79207): core-api 276, ledger-state 308, runtime 1,439 (3 skipped), app 316; 2,339 tests, 0 failures
• cooperative backpressure gate (4cce4e40): runtime 1,427, 0 failures
• final ordered-DRep mechanism (15410aa3): ledger-state 318, 0 failures
• final 008b7ecd change is diagnostic logging only; the deployed native artifact was built from that exact commit
• final repository suite at 779c713b: ./gradlew test -PskipSigning=true, all modules green in 10m28s
• PR checks green: Clean/Build, native-core, integration, distribution, and npm on macOS/Linux/Windows
• manual Dev Distribution Build 33296009774 green: JVM ZIP plus Oracle GraalVM 25.3 native ZIPs for Windows x64, Linux x64, Linux arm64, and macOS 15 arm64

Live validation

Preprod

• native preprod boundary validation completed with exact embedded AdaPot and Koios treasury/reserves parity
• peak recorded Conway boundary footprint was about 535 MiB in the retained validation report

Mainnet to tip

Artifact and runtime:

• commit deployed for the accepted run: 008b7ecd8d194bfff2d9096bbb9923a283d6c304
• SHA-256: 7f2cb1b3e45d9c8841c67e59df08b7091df2129e71cf4160ab58d9ca79f1b411
• Oracle GraalVM 25.3.4.1 / Java 25.0.4.1 / G1 GC
• -Xmx1024m, 64 MiB decoded apply queue, two RocksDB background jobs
• exit-on-epoch-calc-error=true

Results:

• reached inSync=true, steady state, gap 0, peer RUNNING, runtime not degraded at epoch 652
• embedded AdaPot verifier: 326 consecutive passes, epochs 296-621, zero failures
• post-boundary Koios monitor: epochs 613-652, 40/40 exact treasury/reserves matches with no gaps
• the exact Koios values now extend the embedded mainnet expected-pot fixture through epoch 652 for future fail-fast runs
• final epoch 652 treasury/reserves: 1344850365394442 / 6156552553301488, exactly equal to Koios
• accepted log contains no mismatch, OOM, fatal/degraded state, stale-peer disconnect, no-progress recovery, missing body, or unrepaired nonce signal

Final ten boundaries:

Metric	Observed range
total boundary	46.1-55.4 s
streaming rewards	34.9-44.5 s
ordered DRep pass	1.48-1.60 s
peak boundary heap	up to 724 MiB
macOS physical footprint	typically 0.98-1.1 GiB; 1.5 GiB peak

One transient raw macOS RSS sample reached 2.007 GiB while physical footprint
remained 1.1 GiB and immediately fell. Both raw RSS and physical footprint are
reported; choosing the formal cross-platform acceptance metric remains an
operator decision.

Review notes

• the PR intentionally keeps the pointer scan as a fail-closed fallback
• trial-only DRep oracle/diagnostic cleanup is tracked in Post-validation cleanup: remove trial-only oracles, shadow paths and diagnostics (ADR-048 / #97 / #98) #100
• the pre-existing DRep lifecycle discrepancy against DBSync is isolated in Conway DRep distribution: per-DRep stake exceeds DBSync drep_distr for a stable set of DReps (not visible to AdaPot) #99; same-state old/new mechanism parity and AdaPot parity pass
• no data directory migration is attempted for unsupported non-empty preview chainstates
• intentionally deferred from this PR: 512-768 MiB launcher-heap documentation/testing,
  an optional Serial-GC binary experiment, and live Linux 2 GiB validation through
  three boundaries; no full sync was restarted for these items

Final rollout decision

• The current preview epoch-boundary format remains v1. Unsupported markers fail closed; there is no speculative v2 format or v1-to-v2 promotion. Pointer-index readiness remains an independent integrity gate.
• Mainnet acceptance completed, so streaming rewards are now the bundled, distribution, and injection-fallback default. Legacy remains an explicit complete-boundary rollback option.
• Disposable epoch-628 rehearsal at final-lineage head opened v1 directly and completed 628-to-629 with stake-index plus streaming rewards: rewards 51.369 s, snapshot 7.951 s, governance 2.921 s, total about 62.3 s, physical footprint about 1.0 GiB with 1.5 GiB peak.
• Epoch 629 AdaPot passed and independently matched Koios exactly: treasury 1628737335650758, reserves 6389903206392209.
• PR feat(epoch): bound memory during epoch processing (ADR-048) #101 must merge before the stacked issue-99 fix branch; issue-99 contains only the PV9 lifecycle correction and its tests.

[beanbot-gh]

Review (coordinator) — not ready to merge yet. Blocking items B1–B3, then re-review.

B1 — startup gate throws on an uninitialized store (CI :app:integrationTest failure).
Startup ledger-state recovery failed … StakeBalanceConsistencyException: UTXO stake index coordinate metadata is missing or malformed. LedgerStateSubsystem.completePointerIndexVersionGateIfApplicable → DefaultUtxoStore.isPointerIndexReadyAtCurrentCoordinate() → readStakeBalanceCoordinate() throws when META_LAST_APPLIED_* is absent. A fresh store that has not applied genesis/any block yet (the devnet/integration path) must be treated as not yet applicable (log + skip, version untouched), not rejected and not thrown. Fix in the store: return false/not-applicable when the coordinate metadata is absent (only malformed metadata is an error), and add the "fresh, uninitialized store starts and later gets the marker at genesis" case to LedgerStateSubsystemTest / the integration suite. This was not covered by my earlier review because the unit tests only modelled initialized stores.

B2 — two statements in the ADR-048 text and this PR body are wrong or premature.
(a) "The ADR records physical footprint as the macOS acceptance metric and RSS as the Linux metric" — that decision is still pending with the operator; restore the original gate wording (peak RSS ≤ 1.5 GiB target / 2 GiB ceiling) and state "decision pending (operator): footprint vs raw RSS; both reported".
(b) "A deliberate live Conway crash-after-SNAP drill was not run …" — it was run, on a disposable Conway clone: 2026-08-30 07:08 SGT, epoch 532, kill −9 after the durable snapshot-write line and before COMPLETE; restart resumed from step 3 (stopped at step 2 → Resuming … from step 3), rewards/snapshot skipped in ms, POOLREAP + governance ran, fail-fast AdaPot PASS, recovery 2.5 s, pots equal to Koios. Cite it (with log names) in both places.

B3 — unrelated CI/packaging commits in this PR. 517fdddc (pin Oracle GraalVM 25.3 in workflows, setup-graalvm inputs changed to version: '25.3', docker builder tag 25i3, macOS G1 auto-select), 0a030e38 (Windows verify), 5926281c (macOS runner 14→15, SBOM license-override logic + slf4j-reload4j override) are not part of the reviewed lineage. Move them to chore/ci-graalvm-25.3 with their own review; note that setup-graalvm@v1's version input is the legacy GraalVM-CE selector (Oracle GraalVM is selected via java-version + distribution: graalvm) and that the 25i3 registry tag must be verified to exist.

Non-blocking

• N1 Clean, Build failure: Failed to initialize or auto-start Yano: NPE … this.historyArchive is null during plugin directory capture in an app test — identify whether it is a test-ordering/environment issue or caused by this lineage (main's last Integration run on 2026-08-28 also failed once, so it may be flaky); state the finding.
• N2 Add the extended expected_ada_pots_mainnet.json (through 652) in this PR so the next binary verifies in-process again; keep the "622–652 Koios-monitor-only for this run" note.
• N3 Link the preprod validation report file for the "peak Conway footprint ≈ 535 MiB" claim, and the ADR-048 review record (adr/reports/adr-048-implementation-review-2026-08-29.md) + adr/reports/*.csv for the mainnet tables.
• N4 PR granularity: this bundles ADR-048 + Maintain a pointer-address UTXO index to remove the pre-Conway full-UTXO scan at epoch boundaries (ADR-048 follow-up) #97 + Epoch boundary optimization umbrella (ADR-048 follow-ups): one change at a time, measured against the current baseline #98 (95 files, +10.9k). The review record covers every commit, so a single PR is workable, but it is the maintainer's call whether to split it into the three reviewed lineages for history.

Reference: mainnet validation 448–652 = 205 consecutive exact boundaries on the accepted binaries; last-10 boundaries 46–55 s; footprint 0.98–1.1 GiB (peak 1.5).

[beanbot-gh]

Re-review of head 779c713b — blocking items resolved; approve from the coordinator side.

• B1 (gate on uninitialized store): fixed in 183237c6 + e3df1a4b — a completely uninitialized store is not applicable (skip + log); genesis-only stores derive their coordinate from the pointer marker; devnet time-travel stores genesis after block zero exists. CI integration-test now passes.
• B2 (ADR/PR wording): fixed in 69a9382e — memory metric stated as an operator decision with both measures reported; the epoch-532 kill −9 crash-after-SNAP drill on a disposable Conway clone is cited.
• B3 (CI/packaging commits): kept in this PR per the maintainer's explicit instruction (license completeness warning-only; GraalVM 25.3 pin; runner update).
• N1: historyArchive NPE was a test-setup issue, fixed in 52522008 (test-only).
• N2: expected_ada_pots_mainnet.json extended to epochs 209–652 (verified: 444 entries; epoch 652 = treasury 1344850365394442 / reserves 6156552553301488, exactly Koios).
• Added since: status API (/api/v1/node/epoch-calc-status with last-boundary telemetry, gcMetricsAvailable=false made explicit) and the opt-in default | low-memory resource profile (255f8648). I checked the DEFAULT path is behaviour-neutral (every accepted value unchanged; explicit targetFileSizeBase 64 MiB equals the RocksDB default). The low-memory bundle itself is untested at runtime — please keep it labelled experimental in the docs until the deferred 512–768 MiB / Linux 2 GiB validation runs.
• Independent verification at 779c713b: core-api 276, ledger-state 319, runtime 1,444 (3 skipped), app 316 — 2,355 tests, 0 failures. CI: Clean/Build, Integration/Distribution/Native, npm Testkit all green for this head.

Validation reference: mainnet 448–652 on this lineage's binaries = 205 consecutive exact boundaries (in-process AdaPot 296–621, Koios 322–652); last-10 boundaries 46–55 s; footprint 0.98–1.1 GiB (peak 1.5). Follow-ups tracked separately: #99 (DRep distribution vs DBSync — root cause identified on the issue, fix as its own PR), #100 (post-validation cleanup). PR granularity remains the maintainer's call.