Automatic legacylaunchers (mpiexec, mpirun) in OpenMPI 5.0.x ? about spack

[alalazo]

@hppritcha fyi

[hppritcha]

The most important thing to note here is that the Open MPI 5.0.x release stream will ONLY work with SLURM srun launcher if the SLURM PMIx plugin is available. See https://openpmix.github.io/support/how-to/slurm-support.html for info on how to ensure this happens when installing SLURM on your cluster/cloud.

Applications using MPI dynamic functionality like MPI_Comm_spawn or MPI-4 features like MPI_Comm_create_from_group, will need to be launched in either singleton mode or use the mpirun/mpiexec that comes with Open MPI (or an use externally built PRRTe) as the SLURM PMIx server implementation does not implement PMIx_Spawn or the various PMIx Group constructor functions.

That having been said, if the MPI workloads for the target system are primarily expected to be classic bulk-synchronous MPI apps, then as long as SLURM is built with the PMIx plugin, users can happily (hopefully) continue to use srun with their applications built to using Open MPI.

But, if you are operating a system where, for example, lots of MPI4PY data analytics/AI apps are being run, you probably want to make sure that MPI_Comm_spawn works -

https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpipoolexecutor

For Open MPI we recently added MPI4PY's test suite (master branch) to our github actions for the main and v5.0.x branches as it does a lot of stress testing of MPI dynamic functionality. All those tests die with PMIX unimplemented errors when using srun as, like I said above, the SLURM PMIx server only implements functionality needed for classic bulk synchronous apps as far as I can tell.

Probably the legacy string should be replaced with something else in the spack recipe as its somewhat of a loaded term. Maybe internal?