Skip to content

Race condition in _update_parent_job for multi-container KBase jobs at non-terminal states #673

Description

@MrCreosote

Note this requires 2 containers in the same job updating to the same state at the same time with millisecond precision and so is very unlikely.

When a KBase job has num_containers > 1, multiple executor processes run
concurrently and independently report state transitions. When both executors
send the same intermediate state update (e.g. UPLOAD_SUBMITTING) at nearly the
same time:

  1. Executor A updates subjob 0 → queries get_parent_job_update → sees all N
    subjobs reached UPLOAD_SUBMITTING → attempts synchronous parent update →
    succeeds
  2. Executor B updates subjob 1 → queries get_parent_job_update → also sees all
    N → attempts synchronous parent update → gets InvalidJobStateError with
    actual_state=UPLOAD_SUBMITTING

Under the current error-handling logic (only swallow RECOVERING), executor B's
InvalidJobStateError falls through to handle_exception, which transitions the
parent job to ERROR. The job is killed incorrectly.

Potential fix: When catching InvalidJobStateError,
also check if actual_state equals the intended target state (or is a later
state in the progression). If so, another container beat us to the update and
the error is safe to ignore. Need to think through this and make sure this makes
sense

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions