Note this requires 2 containers in the same job updating to the same state at the same time with millisecond precision and so is very unlikely.
When a KBase job has num_containers > 1, multiple executor processes run
concurrently and independently report state transitions. When both executors
send the same intermediate state update (e.g. UPLOAD_SUBMITTING) at nearly the
same time:
- Executor A updates subjob 0 → queries get_parent_job_update → sees all N
subjobs reached UPLOAD_SUBMITTING → attempts synchronous parent update →
succeeds
- Executor B updates subjob 1 → queries get_parent_job_update → also sees all
N → attempts synchronous parent update → gets InvalidJobStateError with
actual_state=UPLOAD_SUBMITTING
Under the current error-handling logic (only swallow RECOVERING), executor B's
InvalidJobStateError falls through to handle_exception, which transitions the
parent job to ERROR. The job is killed incorrectly.
Potential fix: When catching InvalidJobStateError,
also check if actual_state equals the intended target state (or is a later
state in the progression). If so, another container beat us to the update and
the error is safe to ignore. Need to think through this and make sure this makes
sense
Note this requires 2 containers in the same job updating to the same state at the same time with millisecond precision and so is very unlikely.
When a KBase job has num_containers > 1, multiple executor processes run
concurrently and independently report state transitions. When both executors
send the same intermediate state update (e.g. UPLOAD_SUBMITTING) at nearly the
same time:
subjobs reached UPLOAD_SUBMITTING → attempts synchronous parent update →
succeeds
N → attempts synchronous parent update → gets InvalidJobStateError with
actual_state=UPLOAD_SUBMITTING
Under the current error-handling logic (only swallow RECOVERING), executor B's
InvalidJobStateError falls through to handle_exception, which transitions the
parent job to ERROR. The job is killed incorrectly.
Potential fix: When catching InvalidJobStateError,
also check if actual_state equals the intended target state (or is a later
state in the progression). If so, another container beat us to the update and
the error is safe to ignore. Need to think through this and make sure this makes
sense