Skip to content

chore: tune job_executions storage for update-heavy churn - #141

Merged
bodymindarts merged 1 commit into
mainfrom
chore/job-executions-storage-tuning
Jul 28, 2026
Merged

chore: tune job_executions storage for update-heavy churn#141
bodymindarts merged 1 commit into
mainfrom
chore/job-executions-storage-tuning

Conversation

@nicolasburtey

Copy link
Copy Markdown
Member

Motivation

job_executions is a small but extremely update-heavy table: state flips (pending ↔ running), alive_at heartbeat bumps, execution_state_json progress writes, and per-attempt reschedules. Observed on a 10 loans/s stress-test sandbox:

  • ~2.5k dead tuples on ~1.4k live rows with autovacuum running every ~4 min on the default schedule
  • the poll query's mean execution time climbing between vacuums (320ms → 374ms over 15 min)

Change

Per-table storage parameters in the setup migration:

ALTER TABLE job_executions SET (
  fillfactor = 70,
  autovacuum_vacuum_scale_factor = 0.01,
  autovacuum_vacuum_threshold = 50,
  autovacuum_analyze_scale_factor = 0.02
);
  • fillfactor = 70 leaves per-page room so updates that touch no indexed column (execution_state_json, alive_at, poller_instance_id) can go HOT — no new page append, no index maintenance, less bloat per update.
  • Aggressive autovacuum (scale factor 0.01, threshold 50) vacuums the table almost constantly; at this size a vacuum is sub-second and keeps dead tuples near zero instead of letting them accumulate to table scale between default-schedule runs.

Notes

  • Trade-off: fillfactor 70 costs ~30% more pages for a tiny table — negligible. If job_executions ever grows large, the autovacuum scale factor should be revisited (0.01 of a huge table is no longer "often").
  • Complementary to the drop-alive_at-index PR: that removes churn at the source, this bounds the cost of the churn that remains. Either can land without the other.
  • Migration edited in place per repo convention; downstream vendors (e.g. lana-bank lana/app/migrations/20250904065521_job_setup.sql) need the same edit when bumping the crate.

Observed under stress-testing load: ~2.5k dead tuples on ~1.4k live
rows with autovacuum running every ~4 min, and the poll query's mean
execution time climbing between vacuums (320ms -> 374ms over 15 min).

Two knobs in one migration:
- fillfactor = 70 leaves per-page room so updates that touch no
  indexed column (execution_state_json, alive_at, poller_instance_id)
  can go HOT instead of appending a new version + updating indexes.
- Per-table autovacuum scale factor 0.01 / threshold 50 vacuums the
  table constantly; at this size a vacuum is sub-second, so dead
  tuples stay near zero instead of accumulating to table-scale.

Complementary to dropping the alive_at heartbeat index (separate
proposal): that reduces write churn at the source, this bounds the
cost of the churn that remains.
@nicolasburtey
nicolasburtey force-pushed the chore/job-executions-storage-tuning branch from aa3e34f to 2ae7838 Compare July 28, 2026 13:11
@nicolasburtey
nicolasburtey marked this pull request as ready for review July 28, 2026 13:11
@bodymindarts
bodymindarts merged commit 84cef67 into main Jul 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants