Skip to content

chore: drop idx_job_executions_running_alive_at (heartbeat churn) - #140

Merged
bodymindarts merged 1 commit into
mainfrom
chore/drop-running-alive-at-index
Jul 28, 2026
Merged

chore: drop idx_job_executions_running_alive_at (heartbeat churn)#140
bodymindarts merged 1 commit into
mainfrom
chore/drop-running-alive-at-index

Conversation

@nicolasburtey

Copy link
Copy Markdown
Member

Motivation

idx_job_executions_running_alive_at (alive_at) WHERE state='running' indexes exactly the column that changes on every keep-alive heartbeat of every running job. Each heartbeat is therefore an index delete+insert — making this partial index the largest single source of write churn (dead tuples, WAL, autovacuum pressure) on job_executions under load.

Its only consumer is reclaim_lost_jobs:

WHERE state = 'running' AND alive_at < $1 AND job_type = ANY($2) AND ...

The running set is tiny by construction (bounded by poller concurrency), and job_executions itself is a small table — a seq scan for the periodic reclaim pass is cheaper than maintaining this index on every heartbeat. Observed on a stress-test sandbox: ~2.5k dead tuples on ~1.4k live rows despite autovacuum running every ~4 min.

Change

Drop the index (migration edited in place per repo convention; downstream vendors like lana-bank need the same edit when bumping the crate).

Alternatives considered (feedback welcome)

  • Keep the index, heartbeat less often — reduces churn but delays lost-job detection proportionally; dropping the index keeps detection latency unchanged.
  • Keep and accept the churn — fine if job_executions is expected to grow large enough that a periodic seq scan of the whole table (including pending rows) becomes a problem. At current scale (~10³ rows) it is not.

Complementary to the storage-tuning PR (fillfactor + aggressive autovacuum), which bounds the cost of the churn that remains; this one removes churn at the source.

@cursor

cursor Bot commented Jul 28, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

alive_at is the heartbeat column: it is bumped on every keep-alive of
every running job. Indexing exactly that column turns each heartbeat
into an index delete+insert, making the index the single largest
source of write churn on job_executions under load (dead tuples, WAL,
autovacuum pressure).

Its only consumer is reclaim_lost_jobs (state='running' AND alive_at <
threshold). The running set is tiny by construction (bounded by poller
concurrency), and job_executions itself is small, so a seq scan for
the periodic reclaim pass is cheaper than maintaining the index on
every heartbeat.

Alternative considered: keep the index and heartbeat less often. That
delays lost-job detection; dropping the index keeps detection latency
unchanged.
@nicolasburtey
nicolasburtey force-pushed the chore/drop-running-alive-at-index branch from 8215983 to 073f415 Compare July 28, 2026 13:11
@nicolasburtey
nicolasburtey marked this pull request as ready for review July 28, 2026 13:11
@bodymindarts
bodymindarts merged commit 809fcdc into main Jul 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants