chore: tune job_executions storage for update-heavy churn - #141
Merged
Conversation
Observed under stress-testing load: ~2.5k dead tuples on ~1.4k live rows with autovacuum running every ~4 min, and the poll query's mean execution time climbing between vacuums (320ms -> 374ms over 15 min). Two knobs in one migration: - fillfactor = 70 leaves per-page room so updates that touch no indexed column (execution_state_json, alive_at, poller_instance_id) can go HOT instead of appending a new version + updating indexes. - Per-table autovacuum scale factor 0.01 / threshold 50 vacuums the table constantly; at this size a vacuum is sub-second, so dead tuples stay near zero instead of accumulating to table-scale. Complementary to dropping the alive_at heartbeat index (separate proposal): that reduces write churn at the source, this bounds the cost of the churn that remains.
nicolasburtey
force-pushed
the
chore/job-executions-storage-tuning
branch
from
July 28, 2026 13:11
aa3e34f to
2ae7838
Compare
nicolasburtey
marked this pull request as ready for review
July 28, 2026 13:11
bodymindarts
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
job_executionsis a small but extremely update-heavy table: state flips (pending ↔ running),alive_atheartbeat bumps,execution_state_jsonprogress writes, and per-attempt reschedules. Observed on a 10 loans/s stress-test sandbox:Change
Per-table storage parameters in the setup migration:
fillfactor = 70leaves per-page room so updates that touch no indexed column (execution_state_json,alive_at,poller_instance_id) can go HOT — no new page append, no index maintenance, less bloat per update.Notes
job_executionsever grows large, the autovacuum scale factor should be revisited (0.01 of a huge table is no longer "often").alive_at-index PR: that removes churn at the source, this bounds the cost of the churn that remains. Either can land without the other.lana/app/migrations/20250904065521_job_setup.sql) need the same edit when bumping the crate.