perf: restructure poll query — filter due jobs first, fetch wide columns late

[nicolasburtey]

Motivation

Observed on a 10 loans/s lana-bank stress test (sandbox, stress-testing preset): the poll_jobs query (WITH eligible AS ...) had a window-mean execution time of 320–374ms — and climbing over a 15-minute soak — on a job_executions table with only ~1.5k live rows. It was the #2 DB-time consumer on the instance.

Why the old shape is expensive at load:

1. eligible scanned all pending rows of the supported job types — including future-scheduled jobs — and ran the NOT EXISTS anti-join against the running set for every one of them, before any execute_at <= now filtering.
2. eligible carried the wide execution_state_json JSONB; being referenced twice it was materialized, then scanned by min_wait and fed into the ROW_NUMBER() window sort — JSONB payloads included. Temp-file spill waits (BuffileWrite/Read) were observed in pg_wait_sampling.
3. All of the above ran on every poll even though only LIMIT $1 rows are ever used.

Change

Restructure so the cheap, indexable filters run first and the expensive machinery only touches a bounded candidate set:

• due — state='pending' AND job_type = ANY($4) AND execute_at <= now, ordered by execute_at, bounded by LIMIT $1 * 4 overscan. Served directly by the existing idx_job_executions_pending_job_type_execute_at (job_type, execute_at) WHERE state='pending' partial index.
• candidates — the 1-at-a-time-per-queue anti-join and the queue-dedup window now run only over the bounded due set.
• selected_jobs — execution_state_json is fetched by join-back only for the ~$1 winners, never for the whole pending set.
• min_wait — computed straight from the pending partial index without the anti-join (index-only scan).

Trade-offs to review

• Overscan under-fill: if more than $1 * 4 due jobs exist and many of the earliest are queue-blocked, a poll can return fewer than $1 jobs even though eligible ones exist beyond the overscan. The poller loops immediately after a non-empty batch, so worst case is an extra poll cycle. * 4 is a guess — happy to tune.
• min_wait less precise: a queue-blocked nearest future job now causes one early wake-up (the old query excluded blocked jobs from min_wait). Only matters while that queue stays busy.

Validation

• .sqlx regenerated; full nextest suite passes locally.
• test_await_completions_resolves_when_notifications_dropped is flaky under full-suite parallel load — verified it also fails on unmodified main locally (passes standalone); not introduced by this change.
• Recommend an A/B sandbox run (custom-image pipeline) against the current baseline: poll mean 320→374ms, ~1 poll/s at ~10 jobs/s load.

Notes

• Touches the same FOR UPDATE line as the SKIP LOCKED PR — whichever lands second needs a trivial rebase.