Skip to content

Requeue interrupted jobs instead of leaving them stuck running - #260

Merged
ericmj merged 1 commit into
mainfrom
requeue-on-shutdown
Jun 11, 2026
Merged

ericmj merged 1 commit into
mainfrom
requeue-on-shutdown

Conversation

@ericmj

@ericmj ericmj commented Jun 10, 2026

Copy link
Copy Markdown
Member

On shutdown (SIGTERM from deploys) the runner now traps exits and its terminate kills in-flight tasks and puts their jobs back at the front of the queue, instead of leaving the rows in running until the maintenance sweep fails them three hours later. Agents report the requeue to the master over a new /api/queue/requeue endpoint.

The maintenance sweep likewise requeues stale running jobs - a job stuck for three hours means its node died hard, not that the work is suspect - capped at two requeues before it is marked failed. Requeueing records no backoff and keeps inserted_at, so interrupted jobs resume promptly.

A crashing task (throw/exit) no longer takes the runner down with it; the abnormal DOWN now marks that job failed and the runner keeps its bookkeeping for the other running jobs.

On shutdown (SIGTERM from deploys) the runner now traps exits and its
terminate kills in-flight tasks and puts their jobs back at the front of
the queue, instead of leaving the rows in running until the maintenance
sweep fails them three hours later. Agents report the requeue to the
master over a new /api/queue/requeue endpoint.

The maintenance sweep likewise requeues stale running jobs - a job stuck
for three hours means its node died hard, not that the work is suspect -
capped at two requeues before it is marked failed. Requeueing records no
backoff and keeps inserted_at, so interrupted jobs resume promptly.

A crashing task (throw/exit) no longer takes the runner down with it;
the abnormal DOWN now marks that job failed and the runner keeps its
bookkeeping for the other running jobs.
@ericmj
ericmj merged commit 501b9e2 into main Jun 11, 2026
4 checks passed
@ericmj
ericmj deleted the requeue-on-shutdown branch June 11, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant