OpenMPI mpirun fails on Spur due to unsupported srun --kill-on-bad-exit argument

[cijohnson]

OpenMPI 4.1.6 built with Slurm support successfully selects the plm_slurm launcher when running inside a Spur allocation. However, job launch fails because OpenMPI invokes srun with the Slurm option --kill-on-bad-exit, which is not currently recognized by Spur's srun implementation.

As a result, mpirun cannot launch MPI ranks through the scheduler, even though OpenMPI detects and selects the Slurm launcher.

Environment and Reproduction Steps

spur alloc -N 2 -A amd-burst -t 1:00:00

srun --mpi=list
none
pmix
plugin_dir=/usr/lib/spur

srun -N1 -n1 --pty bash

/opt/openmpi/bin/ompi_info | grep slurm
Configure command line: ... --with-slurm
MCA ess: slurm
MCA plm: slurm
MCA ras: slurm
MCA schizo: slurm

/opt/openmpi/bin/mpirun --mca plm_base_verbose 100 -np 2 hostname
[<hostname>] Query of component [slurm] set priority to 75
[<hostname>] Selected component [slurm]

plm:slurm: final top-level argv:

srun --ntasks-per-node=1 --kill-on-bad-exit --mpi=none --nodes=1 ...

Error: unexpected argument '--kill-on-bad-exit' found

Usage: srun --ntasks-per-node <NTASKS_PER_NODE> [COMMAND]...

An ORTE daemon has unexpectedly failed after launch

Expected Behavior

One of the following:

1. Spur srun accepts --kill-on-bad-exit for Slurm compatibility.
2. Spur provides compatibility handling/translation for this option.
3. Documentation explicitly identifies unsupported Slurm launcher flags and corresponding OpenMPI limitations.

If option 1 or 2 is supported, then
mpirun -np 2 hostname within a spur allocation should allow OpenMPI to launch through the scheduler without requiring SSH.

Impact
This prevents OpenMPI jobs launched via mpirun from interoperating with Spur's Slurm-compatible interface.

Workflows affected include:

python
  -> mpirun
      -> MPI/RCCL applications

Examples:

RCCL all_reduce_perf
MPI benchmarks
Python orchestration frameworks that invoke mpirun
Existing HPC workflows migrated from Slurm

Questions for Spur Team

1. Is mpirun via OpenMPI's plm_slurm launcher an officially supported workflow on Spur?
2. Is --kill-on-bad-exit intentionally unsupported?
3. Is there a compatibility layer or recommended configuration for OpenMPI 4.1.x?
4. Should users launch distributed MPI workloads exclusively through: srun --mpi=pmix ... instead of mpirun within a spur allocation? I yes, we force CSP's to install spur client cli's in all the spur worker nodes