[cristinaponcela]

See issue in super secret repo.

We have a worker health issue where we keep getting errors in Sentry (never wanted to stop getting an email more in my life!) bc absurd.claim_task() does a cancellation sweep on every claim.

Current absurd.t_default:

 total tasks:                  1,546,843
 active pending/sleeping/running:       0 at count time, transiently a few
 active with cancellation:              0
 any task with cancellation:             0

Breakdown:

 completed, cancellation IS NULL: 1,544,940
 failed,    cancellation IS NULL:     1,903

So in current prod, 100% of rows have cancellation IS NULL.

That means changing the sweep to:

 WHERE state IN ('pending', 'sleeping', 'running')
   AND cancellation IS NOT NULL

plus adding the partial index:

 CREATE INDEX ...
 ON absurd.t_default (task_id)
 WHERE state IN ('pending', 'sleeping', 'running')
   AND cancellation IS NOT NULL;

would make the cancellation sweep query an empty/tiny index instead of scanning ~1.55M rows.

This PR adds that :)

[jboga]

We hit the same problem and ended up with nearly the same fix, so here is some extra data, plus one suggestion about the index predicate.

Production impact: on a queue of ~140k tasks, this query was ~70% of our database's total query time.

Reproduction (PostgreSQL 16, absurd 0.5.0 schema, 200k tasks: 50 pending, the rest completed, none with a cancellation policy):

select absurd.create_queue('demo');
insert into absurd.t_demo (task_id, task_name, params, state, attempts, enqueue_at, first_started_at)
select gen_random_uuid(), 'demo-task', jsonb_build_object('payload', repeat('x', 800)),
       case when g <= 50 then 'pending' else 'completed' end, 1,
       now() - interval '1 hour', now() - interval '1 hour'
from generate_series(1, 200000) g;
vacuum analyze absurd.t_demo;
-- then EXPLAIN (ANALYZE, BUFFERS) the cancellation sweep, with now() in place of $1

With a partial index on (task_id) where cancellation is not null: ~33k buffers / 42 ms → 1 buffer / 0.03 ms.

Plans

Before:

Gather Merge  (cost=40063.31..40065.18 rows=16 width=16) (actual time=38.803..42.306 rows=0 loops=1)
  Workers Planned: 2
  Workers Launched: 2
  Buffers: shared hit=15088 read=18318 written=96
  ->  Sort  (actual time=34.999..35.000 rows=0 loops=3)
        Sort Key: task_id
        ->  Parallel Seq Scan on t_demo  (cost=0.00..39063.17 rows=8 width=16) (actual time=34.891..34.891 rows=0 loops=3)
              Filter: ((state = ANY ('{pending,sleeping,running}'::text[])) AND (…cancellation conditions…))
              Rows Removed by Filter: 66667
              Buffers: shared hit=15016 read=18318 written=96
Execution Time: 42.369 ms

After:

Index Scan using t_demo_ci on t_demo  (cost=0.12..8.20 rows=20 width=16) (actual time=0.004..0.004 rows=0 loops=1)
  Filter: ((state = ANY ('{pending,sleeping,running}'::text[])) AND (…cancellation conditions…))
  Buffers: shared hit=1
Execution Time: 0.032 ms

Partitioned queues behave the same way: every partition is scanned (25k buffers, 40 ms for 200k tasks across weekly partitions). The same index on the parent brings it to 6 buffers and 0.27 ms, and partitions created later inherit it.

Suggestion: drop state from the index predicate.

create index if not exists ... on absurd.t_<queue> (task_id)
  where cancellation is not null;

Postgres can't make an update HOT if it changes any column an index uses, and columns in a partial index's WHERE count. With state in (...) in the predicate, every task state change (claim, complete, fail, sleep, cancel) becomes a non-HOT update, even for the many tasks without cancellation. That works against the fillfactor=70 on these tables. cancellation is never updated after a task is created, so an index that depends only on it keeps state changes HOT. The query's state filter still runs, against an index that's empty or tiny.

The cancellation is not null this PR adds to the query is still worth keeping. Postgres could already infer it from the OR branches, which compare against (cancellation->>…)::bigint, but writing it out makes the index match obvious.

Analysis drafted with AI assistance; benchmarks run by me.

[cristinaponcela]

@jboga added your suggestion. TIL, thanks! 😁

[jboga]

Thanks @cristinaponcela for the PR, and thanks Absurd team for the new release.