What happens
In Pod mode launch_job in crates/spur-k8s/src/agent.rs creates one Pod per target node. When the Kubernetes API answers HTTP 409 (a Pod with that name already exists) the operator logs "K8s Pod already exists, treating as success" and returns success: true to the controller.
The Pod name is spur-job-<id>-<sanitized node name>, and sanitize_k8s_name maps every character outside [a-z0-9-] to -. Two node names that differ only in such a character, for example node.a and node-a, produce the same Pod name. The second launch then hits 409, the controller records the launch as successful, and no second Pod ever runs. The job's peer list (see #807) names a peer that does not exist, so a multi-node job hangs instead of failing.
A stale Pod from an earlier launch of the same job id, left behind after an operator restart or a controller failover, has the same effect: the new launch is reported as running while the old Pod is what actually exists.
Expected
A launch that did not create the Pod it was asked to create must not be reported as a success, or the controller must be able to tell the two cases apart.
Reproduce
- Register two Kubernetes nodes whose names sanitize to the same string, or create a Pod named
spur-job-<id>-<node> by hand before the job is dispatched.
- Submit a job that targets that node.
launch_job returns success: true, kubectl get pods shows one Pod, and the controller shows the job as running.
What happens
In Pod mode
launch_jobincrates/spur-k8s/src/agent.rscreates one Pod per target node. When the Kubernetes API answers HTTP 409 (a Pod with that name already exists) the operator logs "K8s Pod already exists, treating as success" and returnssuccess: trueto the controller.The Pod name is
spur-job-<id>-<sanitized node name>, andsanitize_k8s_namemaps every character outside[a-z0-9-]to-. Two node names that differ only in such a character, for examplenode.aandnode-a, produce the same Pod name. The second launch then hits 409, the controller records the launch as successful, and no second Pod ever runs. The job's peer list (see #807) names a peer that does not exist, so a multi-node job hangs instead of failing.A stale Pod from an earlier launch of the same job id, left behind after an operator restart or a controller failover, has the same effect: the new launch is reported as running while the old Pod is what actually exists.
Expected
A launch that did not create the Pod it was asked to create must not be reported as a success, or the controller must be able to tell the two cases apart.
Reproduce
spur-job-<id>-<node>by hand before the job is dispatched.launch_jobreturnssuccess: true,kubectl get podsshows one Pod, and the controller shows the job as running.