Skip to content

spur-k8s: launch_job reports success when the Pod already exists #873

Description

@pre

What happens

In Pod mode launch_job in crates/spur-k8s/src/agent.rs creates one Pod per target node. When the Kubernetes API answers HTTP 409 (a Pod with that name already exists) the operator logs "K8s Pod already exists, treating as success" and returns success: true to the controller.

The Pod name is spur-job-<id>-<sanitized node name>, and sanitize_k8s_name maps every character outside [a-z0-9-] to -. Two node names that differ only in such a character, for example node.a and node-a, produce the same Pod name. The second launch then hits 409, the controller records the launch as successful, and no second Pod ever runs. The job's peer list (see #807) names a peer that does not exist, so a multi-node job hangs instead of failing.

A stale Pod from an earlier launch of the same job id, left behind after an operator restart or a controller failover, has the same effect: the new launch is reported as running while the old Pod is what actually exists.

Expected

A launch that did not create the Pod it was asked to create must not be reported as a success, or the controller must be able to tell the two cases apart.

Reproduce

  1. Register two Kubernetes nodes whose names sanitize to the same string, or create a Pod named spur-job-<id>-<node> by hand before the job is dispatched.
  2. Submit a job that targets that node.
  3. launch_job returns success: true, kubectl get pods shows one Pod, and the controller shows the job as running.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions