chore: drop idx_job_executions_running_alive_at (heartbeat churn) - #140
Merged
Conversation
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
alive_at is the heartbeat column: it is bumped on every keep-alive of every running job. Indexing exactly that column turns each heartbeat into an index delete+insert, making the index the single largest source of write churn on job_executions under load (dead tuples, WAL, autovacuum pressure). Its only consumer is reclaim_lost_jobs (state='running' AND alive_at < threshold). The running set is tiny by construction (bounded by poller concurrency), and job_executions itself is small, so a seq scan for the periodic reclaim pass is cheaper than maintaining the index on every heartbeat. Alternative considered: keep the index and heartbeat less often. That delays lost-job detection; dropping the index keeps detection latency unchanged.
nicolasburtey
force-pushed
the
chore/drop-running-alive-at-index
branch
from
July 28, 2026 13:11
8215983 to
073f415
Compare
nicolasburtey
marked this pull request as ready for review
July 28, 2026 13:11
bodymindarts
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
idx_job_executions_running_alive_at (alive_at) WHERE state='running'indexes exactly the column that changes on every keep-alive heartbeat of every running job. Each heartbeat is therefore an index delete+insert — making this partial index the largest single source of write churn (dead tuples, WAL, autovacuum pressure) onjob_executionsunder load.Its only consumer is
reclaim_lost_jobs:The running set is tiny by construction (bounded by poller concurrency), and
job_executionsitself is a small table — a seq scan for the periodic reclaim pass is cheaper than maintaining this index on every heartbeat. Observed on a stress-test sandbox: ~2.5k dead tuples on ~1.4k live rows despite autovacuum running every ~4 min.Change
Drop the index (migration edited in place per repo convention; downstream vendors like lana-bank need the same edit when bumping the crate).
Alternatives considered (feedback welcome)
job_executionsis expected to grow large enough that a periodic seq scan of the whole table (including pending rows) becomes a problem. At current scale (~10³ rows) it is not.Complementary to the storage-tuning PR (fillfactor + aggressive autovacuum), which bounds the cost of the churn that remains; this one removes churn at the source.