MPI problems with srun about eckit

[wdeconinck]

Can you check first that eckit has MPI enabled in the CMake configuration log? I guess so but just to be sure.

Then, how eckit detects that it is running in an MPI environment it checks at run-time for following environment variables to choose either true MPI or a Serial MPI stubs implementation.
https://github.com/ecmwf/eckit/blob/develop/src/eckit/mpi/Comm.cc#L43-L45

I should mention that I have been able to use srun in the past without issues, provided that the environment variable export SLURM_EXPORT_ENV=ALL is set, which is required to propagate environment variables to the srun'd program.

If that doesn't work, you can force eckit to choose the "parallel" (MPI) implementation by setting the environment variable:
export ECKIT_MPI_FORCE=parallel

[mmiesch]

Thanks @wdeconinck for the helpful information. It turns out that MPI was enabled and the SLURM_EXPORT_ALL did not help. But, the export ECKIT_MPI_FORCE=parallel command does seem to work!

[mmiesch]

Though this does work for parallel applications, it's not an ideal solution for unit testing with ctest, of course. Setting this variable helps parallel tests to pass but it breaks all the serial tests. So, if you have any other ideas on how one might get around this please let me know. Thanks again.

[wdeconinck]

Ideally the fix is to figure out which environment variable we can add to the list in eckit to check for.
e.g. OMPI_COMM_WORLD_SIZE, ALPS_APP_PE, PMI_SIZE
To figure it out, one could run:

srun -n 1 env

and see which environment variable srun adds that we can detect MPI with. You can then test it by adding it to the code here https://github.com/ecmwf/eckit/blob/develop/src/eckit/mpi/Comm.cc#L43-L45. If it works, we can then patch eckit.

Another ugly workaround or hack which I have used, that would satisfy you for now is to create a srun wrapper called e.g. parallel_run containing the lines:

export ECKIT_MPI_FORCE=parallel
srun "$@"

Then CMake-configure with -DMPIEXEC_EXECUTABLE=<path/to/parallel_run> -DMPIEXEC_NUMPROC_FLAG='-n'

[mmiesch]

Thanks @wdeconinck - the slurm output env variables are listed here - confirmed with the srun -n1 env command you suggested. I think the best bet would likely be SLURM_NTASKS.

I will edit the code as suggested and let you know if it works.

Thanks for the workaround suggestion but I think this would have the same problem as before - all serial jobs would fail because they would invoke the Parallel subclass of eckit::mpi::Comm, right?

[mmiesch]

Thanks again @wdeconinck - your suggested solution works. I could do a formal pull request but it might be easier for you to just apply the patch on your end.

Here is the code that works:

        else if (::getenv("OMPI_COMM_WORLD_SIZE") ||  // OpenMPI
                 ::getenv("ALPS_APP_PE") ||           // Cray PE
                 ::getenv("PMI_SIZE") ||              // Intel
                 ::getenv("SLURM_NTASKS")) {          // slurm srun

[wdeconinck]

Thank you @mmiesch , the modification has been submitted as PR upstream and should make its way soon.

[mmiesch]

Great - Thanks again @wdeconinck for your help.

[wdeconinck]

@tlmquintino This issue can be closed now ( I don't have permissions ).