Comments (9)
Can you check first that eckit has MPI enabled in the CMake configuration log? I guess so but just to be sure.
Then, how eckit detects that it is running in an MPI environment it checks at run-time for following environment variables to choose either true MPI or a Serial MPI stubs implementation.
https://github.com/ecmwf/eckit/blob/develop/src/eckit/mpi/Comm.cc#L43-L45
I should mention that I have been able to use srun in the past without issues, provided that the environment variable export SLURM_EXPORT_ENV=ALL
is set, which is required to propagate environment variables to the srun'd program.
If that doesn't work, you can force eckit to choose the "parallel" (MPI) implementation by setting the environment variable:
export ECKIT_MPI_FORCE=parallel
from eckit.
Thanks @wdeconinck for the helpful information. It turns out that MPI was enabled and the SLURM_EXPORT_ALL did not help. But, the export ECKIT_MPI_FORCE=parallel
command does seem to work!
from eckit.
Though this does work for parallel applications, it's not an ideal solution for unit testing with ctest, of course. Setting this variable helps parallel tests to pass but it breaks all the serial tests. So, if you have any other ideas on how one might get around this please let me know. Thanks again.
from eckit.
Ideally the fix is to figure out which environment variable we can add to the list in eckit to check for.
e.g. OMPI_COMM_WORLD_SIZE
, ALPS_APP_PE
, PMI_SIZE
To figure it out, one could run:
srun -n 1 env
and see which environment variable srun
adds that we can detect MPI with. You can then test it by adding it to the code here https://github.com/ecmwf/eckit/blob/develop/src/eckit/mpi/Comm.cc#L43-L45. If it works, we can then patch eckit.
Another ugly workaround or hack which I have used, that would satisfy you for now is to create a srun wrapper called e.g. parallel_run
containing the lines:
export ECKIT_MPI_FORCE=parallel
srun "$@"
Then CMake-configure with -DMPIEXEC_EXECUTABLE=<path/to/parallel_run> -DMPIEXEC_NUMPROC_FLAG='-n'
from eckit.
Thanks @wdeconinck - the slurm output env variables are listed here - confirmed with the srun -n1 env
command you suggested. I think the best bet would likely be SLURM_NTASKS
.
I will edit the code as suggested and let you know if it works.
Thanks for the workaround suggestion but I think this would have the same problem as before - all serial jobs would fail because they would invoke the Parallel subclass of eckit::mpi::Comm
, right?
from eckit.
Thanks again @wdeconinck - your suggested solution works. I could do a formal pull request but it might be easier for you to just apply the patch on your end.
Here is the code that works:
else if (::getenv("OMPI_COMM_WORLD_SIZE") || // OpenMPI
::getenv("ALPS_APP_PE") || // Cray PE
::getenv("PMI_SIZE") || // Intel
::getenv("SLURM_NTASKS")) { // slurm srun
from eckit.
Thank you @mmiesch , the modification has been submitted as PR upstream and should make its way soon.
from eckit.
Great - Thanks again @wdeconinck for your help.
from eckit.
@tlmquintino This issue can be closed now ( I don't have permissions ).
from eckit.
Related Issues (20)
- ctest failures when setting multiple MPI_ARGS HOT 4
- Travis-ci build is failing on AMD64 HOT 6
- Not sure what installdir, builddir and srcdir to use
- an initialized argv issue in Parallel.cc HOT 3
- literal input of string array in configuration? HOT 1
- Error with ecbuild (CRITICAL - Please define a version for eckit) HOT 9
- Better adhere to BUILD_SHARED_LIBS instead of enforcing shared build? HOT 5
- Eckit fails to build on macOS using Intel compilers
- ISSUE ON INSTALLING eckit-1.20.0.tar.gz HOT 2
- Intermittant eckit_test_mpi_parallel failures
- Any documents for Eckit?
- eckit installation of Cray environments HOT 4
- How can logging-levels be set for eckit/log/log.h?
- Intermittant failures in eckit_test_container_sharedmemarray HOT 5
- Need of MPI_Sendrecv_replace interfaces HOT 1
- Nested SECTIONs in tests do not appear to work HOT 2
- Encountered error during compilation with nvhpc22.1 HOT 2
- Definition of eckit::offset:dump(DumpLoad&) missing HOT 2
- Segmentation fault caused by buffer overflow in ChannelBuffer HOT 3
- Build issue with eigen (MappedSparseMatrix) HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eckit.