OpenMPI on CentOS-HPC 7.9: unable to use ucx because `UCP worker does not support MPI_THREAD_MULTIPLE` about azhpc-images

[jithinjosepkl]

@ltalirz , any reason we cannot use HPC-X (which is OMPI+UCX) here? The mt-init scripts are there in the HPC-X folder. (/opt/hpcx*). The UCX mt build is also here.

[ltalirz]

Hi @jithinjosepkl , sorry for the long delay on our side.

First, my counter question would be: what speaks against having the OpenMPI in /opt be compiled against UCX with thread support? Would you be open to a PR that fixes this?

Second, we tried using the OpenMPI from HPC-X but ran into a number of issues.

1. The HPCX module files are actually broken, see #80 . PR #102 that tried to fix this was closed by you without follow-up (?)
2. After fixing this, mpicc and mpifort from the mpi/hpcx module pick up the outdated gcc 4.8.5 from the CentOS image which fails to compile even a simple MPI program (this despite the fact that ompi_info tells that the OpenMPI bundled with HPC-X is built with gcc 9.2.0).
   This can be fixed by explicitly loading module load gcc-9.2.0 mpi/hpcx (we suggest making gcc-9.2.0 a dependency of hpi/hpcx)
3. Even after this step, we still run into errors when using the hpcx openmpi as an external mpi library in spack, since the libtool .la files shipped with hpcx point to non-existent libraries in /hpc (that were used to compiled hpcx).

Given your comment in #102 we understand that we should switch to AlmaLinux 8.6.
We will open a new issue, should any of these issues persist there.
cc @matt-chan