perf: SKIP LOCKED when selecting jobs in the poll query

[nicolasburtey]

Motivation

The poll query selects candidate rows with plain FOR UPDATE. When more than one poller instance (or overlapping poll loops in the same process) runs the query concurrently, each blocks on the rows the other has already locked, waits for the peer's transaction to commit, and then finds those rows no longer pending — yielding zero jobs for the wait. With multiple lana-bank pods polling the same database this serializes poll loops behind each other.

Change

FOR UPDATE OF je → FOR UPDATE OF je SKIP LOCKED in poll_jobs.

Each poller now takes the next available unlocked rows instead of queueing behind its peers. The existing UPDATE ... WHERE je.state = 'pending' guard in the updated CTE remains as the correctness backstop, so a row taken between selection and update is still silently skipped.

Single-poller deployments are unaffected.

Why this cannot break queue (queue_id) serialization

The 1-at-a-time-per-queue guarantee is enforced by two layers that both run before row locking, and SKIP LOCKED only changes the locking step:

1. Cross-transaction: the NOT EXISTS anti-join excludes pending jobs whose queue already has a running row.
2. Intra-batch: ROW_NUMBER() PARTITION BY COALESCE(queue_id, id::text) + WHERE rn = 1 means one poll can select at most one job per queue.

Race walk-through — poller A is mid-transaction taking queue head J1 (locked, uncommitted); poller B polls with a snapshot from before A commits:

• B's snapshot shows J1 pending, so B's anti-join passes — but B's window still ranks J1 as rn=1 and the queue's J2 as rn=2. B can only ever attempt the head J1; it can never select J2. SKIP LOCKED doesn't re-run the window — it just drops locked rows from the selection.
• Without SKIP LOCKED: B blocks on J1's lock, READ COMMITTED re-fetches the newest version after A commits (now running), and the updated CTE's WHERE je.state = 'pending' guard rejects it → B starts nothing for that queue.
• With SKIP LOCKED: B skips J1 → B starts nothing for that queue. Identical outcome.

In both variants the final UPDATE ... WHERE je.state = 'pending' re-evaluates on the newest committed row version, so the pending→running transition stays single-winner even if two pollers race the exact same row.

The one real behavioral difference — latency, not correctness: if the lock holder rolls back (crash between selection and commit) rather than commits, a blocked waiter (old behavior) would take the job immediately, while a SKIP LOCKED poller has already moved on. That queue's head then waits for the next wake-up: any notify from other job activity, the min_wait timer, the immediate re-poll after a non-empty batch, or worst case MAX_WAIT (60s, src/poller.rs). Worst case is a bounded start delay for that queue — never two same-queue jobs running concurrently.

Test coverage

Added test_queue_id_serializes_execution_across_two_pollers: two independent Jobs services (two pollers, distinct instance ids) share one database and register the same job_type, so both are woken by the same notifications and race to claim the same queue head. Asserts that while the head job is running (blocked), the racing poller never starts the next same-queue job, and that the queue drains in order after the head completes. Passes 5/5 standalone runs; full suite green except the known pre-existing flake test_await_completions_resolves_when_notifications_dropped (also fails on unmodified main locally).

Notes

• .sqlx regenerated (cargo sqlx prepare --workspace).
• Composes with the poll query restructure PR (perf: restructure poll query — filter due jobs first, fetch wide columns late #139) — whichever lands second needs a trivial one-line rebase on the FOR UPDATE line.

[bodymindarts]

I'm not sure this one might break the queue-id serial contract.