Skip to content

fix: only notify pollers when a job becomes eligible - #137

Merged
bodymindarts merged 1 commit into
mainfrom
fix/notify-only-when-eligible
Jul 28, 2026
Merged

fix: only notify pollers when a job becomes eligible#137
bodymindarts merged 1 commit into
mainfrom
fix/notify-only-when-eligible

Conversation

@nicolasburtey

Copy link
Copy Markdown
Member

Motivation

During a 10 loans/s stress test of a lana-bank sandbox (stress-testing preset), the job_executions poll query showed a mean execution time of 320–374ms and the poller was one of the top DB-time consumers. Analysis of the wake-up path shows the poller wakes itself up: the job_executions UPDATE trigger fires pg_notify('job_events', 'execution_ready') on every execute_at change — including the poll query's own UPDATE ... SET execute_at = NULL when taking a job, reclaim_lost_jobs reschedules, and per-attempt failure reschedules.

Every job taken therefore notifies every poller to immediately re-poll. Poll load scales multiplicatively with job throughput: at 10 jobs/s that is 10+ self-triggered polls/s on top of organic wake-ups.

Change

Restrict the UPDATE branch of notify_job_event() to rows that remain eligible — NEW.state = 'pending' — so only genuine (re-)schedules notify:

  • Job taken (state: pending → running, execute_at → NULL): no notify ✅ (was notify)
  • Job completed/failed to terminal: no notify (unchanged)
  • Failure reschedule (pending, new execute_at): notify (unchanged)
  • reclaim_lost_jobs (state → pending, new execute_at): notify (unchanged)
  • INSERT / DELETE branches: unchanged (DELETE still notifies execution_ready to free queue-blocked jobs)

Notes

  • Migration edited in place per repo convention (pre-production; consumers recreate their DBs). Downstream consumers vendoring this migration (e.g. lana-bank lana/app/migrations/20250904065521_job_setup.sql) need the same edit when bumping the crate.
  • Part of a series of poll-loop perf proposals: skip-locked, poll query restructure, drop alive_at index, storage tuning. Independent of all of them.

The job_executions UPDATE trigger fired pg_notify('job_events',
'execution_ready') on every execute_at change. The poll query's own
UPDATE sets execute_at = NULL when taking a job, and reclaim/reschedule
writes do the same — so every job taken woke every poller (including
the taker) to immediately re-poll, multiplying poll load linearly with
throughput. At 10 jobs/s that is 10+ self-triggered polls/s.

Restrict the UPDATE branch to rows that remain in the pending set
(NEW.state = 'pending'), i.e. genuine (re-)schedules. INSERT and
DELETE behavior is unchanged.
@nicolasburtey
nicolasburtey marked this pull request as ready for review July 28, 2026 13:10
@nicolasburtey
nicolasburtey force-pushed the fix/notify-only-when-eligible branch from 6f0c0b5 to b136d48 Compare July 28, 2026 13:11
@bodymindarts
bodymindarts merged commit 3793ee7 into main Jul 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants