cannot find lgrid2grid about cosma

[kabicm]

Dear Yi, thanks for spotting this! You are totally right, grid2grid was changed to COSTA and this will be corrected in the README soon, we are preparing a new release for both COSTA and COSMA.

In your case, you should link to libcosta_scalapack.a, because of this line:

COSMA/src/cosma/CMakeLists.txt

Line 54 in 35e2dda

costa_scalapack

Let me know if it works for you!

[kabicm]

In general, if you use pure cosma library (without pxgemm wrappers) then you link to costa. If you use cosma_pxgemm then you link to costa_scalapack. Pay attention that costa_scalapack overwrites scalapack's pxtran (transpose) and pxgemr2d (layout transformation) functions. So, by using cosma_pxgemm you are implicitly also using our implementations of pxtran and pxgemr2d provided by COSTA. We do this on purpose, as COSTA offers great speedups.

[yaoyi92]

Dear Marko,

Thank you so much.

So, I end up using these commands to compile a Fortran test file. It seems the -qopenmp and -lstdc++ are necessary here. Is this the correct way to do the linking?

mpiifort -O3 -g -traceback -ip -fp-model precise -xCORE-AVX2 -qopenmp -c Scalapack.f90
mpiifort -qopenmp -lstdc++ Scalapack.o -L/nas/longleaf/home/yiy/softwares-dogwood/cosma/COSMA/install_2/lib64 -lcosma_pxgemm -lcosma -lcosta_scalapack -lcosta -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_sequential -lmkl_core

The test file is

Scalapack.f90

program test
  call pdgemm
end program

[yaoyi92]

Dear Marko,

However, if I tried this test file

Scalapack.f90

program test
  call pdgemr2d
end program

It gave me

/nas/longleaf/home/yiy/softwares-dogwood/cosma/COSMA/install_2/lib64/libcosta_scalapack.a(pxgemr2d.cpp.o): In function `pigemr2d':
pxgemr2d.cpp:(.text+0x4d): undefined reference to `void costa::pxgemr2d<int>(int, int, int const*, int, int, int const*, int*, int, int, int const*, int)'
/nas/longleaf/home/yiy/softwares-dogwood/cosma/COSMA/install_2/lib64/libcosta_scalapack.a(pxgemr2d.cpp.o): In function `pigemr2d_':
pxgemr2d.cpp:(.text+0x3ad): undefined reference to `void costa::pxgemr2d<int>(int, int, int const*, int, int, int const*, int*, int, int, int const*, int)'
/nas/longleaf/home/yiy/softwares-dogwood/cosma/COSMA/install_2/lib64/libcosta_scalapack.a(pxgemr2d.cpp.o): In function `pigemr2d__':
pxgemr2d.cpp:(.text+0x58d): undefined reference to `void costa::pxgemr2d<int>(int, int, int const*, int, int, int const*, int*, int, int, int const*, int)'
/nas/longleaf/home/yiy/softwares-dogwood/cosma/COSMA/install_2/lib64/libcosta_scalapack.a(pxgemr2d.cpp.o): In function `PIGEMR2D':
pxgemr2d.cpp:(.text+0x76d): undefined reference to `void costa::pxgemr2d<int>(int, int, int const*, int, int, int const*, int*, int, int, int const*, int)'

Am I missing anything?

[kabicm]

Dear Yi, thanks for checking. The latter caused by COSMA using an outdated version of COSTA. I have just updated the COSTA submodule in COSMA. Can you git clone recursively again the whole COSMA repo and try again? The latter should now be fixed.

[kabicm]

Sorry, I would need some more time, I have to adapt few more things in COSMA to the latest COSTA version before it can work.

[yaoyi92]

I see. No worry. Thank you for letting me know.

[kabicm]

The latest commit should fix the issues with the latest version of COSTA. Since the COSTA library was also modified, it would be best to get a fresh new git clone of COSMA.

To answer your question regarding the OpenMP: you are right, we require it. However, most user codes already have OpenMP as a dependency (especially if they use a multithreaded versions of scalapack/lapack/blas). Your way of linking seems correct to me.

Let me know if this works for you and if you have any other issues!

[kabicm]

Hi Yi (@yaoyi92), i just wanted to follow up on my previous comment: does it work for you now?

[yaoyi92]

Yes, it works! Thank you so much!

[kabicm]

Great, thanks for the feedback, I am glad it works now! The README will be improved soon, so I will now close this issue.