fix: prevent rollup cleanup from permanently freezing rotation (#211)

[NikolayS]

Fixes #211 — two of pg_ash's own scheduled jobs deadlock each other, permanently and silently.

Found by the pre-tag audit — see #160. Draft on purpose: needs REV and your explicit approval.

The bug

rollup_cleanup() deletes the ash.rollup_1m rows that rotate()'s pre-truncation completeness
gate (#81) requires, and rollup_minute()'s watermark only walks forward — so those minutes can
never be re-rolled and the gate can never be satisfied again.

failed: pre-truncation rollup incomplete for 1 group(s) in slot 2; slot not truncated

…forever. Rotation stops, so every new sample piles into the frozen current slot until the volume
fills — exactly the failure the ring exists to prevent. Reachable on stock 30-day retention via
the README's own "increase raw retention" runbook (rebuild_partitions(32,'yes')).

Worst part: it is silent. rotate() returns text on the failure path, so pg_cron records
SUCCESS and ash.status() showed nothing. Only time_since_rotation crept up.

Three parts

1. Break the deadlock — the gate now ignores raw minute groups already past
   rollup_1m_retention_days. Rolling those up would only create rows rollup_cleanup() deletes
   on its next pass, so there is nothing left to protect. Groups still inside retention keep the
   full Critical bug: rollup/rotation can silently lose raw samples #81 check.
2. Make the bad geometry unreachable — new config_raw_rollup_geometry_check constraint:
   (num_partitions - 1) * rotation_period <= rollup_1m_retention_days * interval '1 day',
   enforced both as a table CHECK and in rebuild_partitions(). A table CHECK is used because
   there is no supported setter for these values — direct UPDATE is the interface.
3. Make failure loud — new consecutive_rotate_failures counter on ash.config, surfaced in
   ash.status(), incremented on every silent failed: return and reset by a real rotation.
   Neutral skipped: returns leave it alone.

Verification

RED first against unpatched SQL, with exact-value assertions:

ERROR:  rotation did not recover after cleanup: failed: pre-truncation rollup incomplete ...
ERROR:  status failure metric should be exactly {1}, got NULL
ERROR:  unexpected rebuild geometry result: <accepted>

Then GREEN, including that the #81 gate still refuses to truncate when a group inside retention
is genuinely unrolled (the control that proves the fix didn't just disable the guard), and that the
counter goes {1} → {2} across repeated failures and resets on success.

Fresh install, full 1.0→2.0 chain, double re-apply, fresh-vs-chain schema equivalence, and a
zero-argument smoke of every reader all clean.

Two things I want a reviewer's eye on

• (num_partitions - 1) is deliberately stricter than the (num_partitions - 2) retention
  formula used everywhere else (:2268, :2374, :2315, :2475, README:364 — and documented
  by in-flight fix: correct admin documentation contracts #152). The -1 is correct here: it bounds the age of the oldest row in the slot
  about to be truncated, not the user-visible retention window. But the two formulas now differ in
  the same file with no comment saying why, so a later "consistency fix" to -2 would silently
  reintroduce this deadlock for the boundary slot. Worth adding that comment before merge.
• Upgrade path mutates operator config. An existing install with unsafe geometry has its
  rollup_1m_retention_days raised to the minimum safe whole-day value so the constraint can be
  added. It emits raise warning first, and raising minute retention is the only non-destructive
  direction (reducing num_partitions would drop raw data, and rejecting the installer would deny
  affected systems the very fix they need) — but it does increase disk usage without asking.

🤖 Generated with Claude Code

https://claude.ai/code/session_014HzBGzjFyN8dXZHbdWmYBj