'omp.h' file not found on Platform("aarch64", "macos") about galahad

[amontoison]

The compilation of GALAHAD on Mac works with gcc/gfortran.

[dpo]

What version of gcc?

[amontoison]

It was gcc 11.1.

[amontoison]

https://stackoverflow.com/a/33401294

[nimgould]

The SOMP = -fopenmp compiler flag should provide all the relevant include data here. Could you please go to the $GALAHAD/src/ssids directory, and try

make -f (architecture)

where (architecture) is whatever name you have in $GALAHAD/makefiles/ (e.g., mac64.osx.gfo). This will tell us what flags are being passed

[amontoison]

I have an error when I try make -f binarybuilder.bb.fc:

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f binarybuilder.bb.fc
make: binarybuilder.bb.fc: No such file or directory
make: *** No rule to make target 'binarybuilder.bb.fc'.  Stop.

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # ls
C                        config.h                 fkeep.F90                gpu_subtree_no_cuda.f90  solve.cu
LICENCE                  contrib.f90              gpu                      inform.f90               ssids.f90
NumericSubtree.cxx       contrib.h                gpu_alloc.f90            ldlt_app.cxx             ssidss.data
README                   contrib_free.f90         gpu_datatypes.f90        ldlt_nopiv.cxx           ssidss.f90
SymbolicSubtree.cxx      cpu                      gpu_dense_factor.f90     ldlt_tpp.cxx             subtree.f90
ThreadStats.cxx          cpu_iface.f90            gpu_factor.f90           makemaster               syrk.cu
akeep.f90                cpu_solve.f90            gpu_interfaces.f90       profile.cxx              updata_ssids
anal.f90                 cpu_subtree.f90          gpu_smalloc.f90          profile.hxx              wrappers.cxx
assemble.cu              datatypes.f90            gpu_solve.f90            profile_iface.f90
cholesky.cxx             dense_factor.cu          gpu_subtree.f90          reorder.cu
sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # 

[nimgould]

Sorry, Alexis, I should have said make -f $GALAHAD/makefiles/binary builder.bb.fc Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Alexis ***@***.***> Sent: Tuesday, November 8, 2022 6:38:50 PM To: ralna/GALAHAD ***@***.***> Cc: Gould, Nick (STFC,RAL,SC) ***@***.***>; Comment ***@***.***> Subject: Re: [ralna/GALAHAD] 'omp.h' file not found on Platform("aarch64", "macos") (Issue #18) I have an error when I try make -f binarybuilder.bb.fc: sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f binarybuilder.bb.fc make: binarybuilder.bb.fc: No such file or directory make: *** No rule to make target 'binarybuilder.bb.fc'. Stop. sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # ls C config.h fkeep.F90 gpu_subtree_no_cuda.f90 solve.cu LICENCE contrib.f90 gpu inform.f90 ssids.f90 NumericSubtree.cxx contrib.h gpu_alloc.f90 ldlt_app.cxx ssidss.data README contrib_free.f90 gpu_datatypes.f90 ldlt_nopiv.cxx ssidss.f90 SymbolicSubtree.cxx cpu gpu_dense_factor.f90 ldlt_tpp.cxx subtree.f90 ThreadStats.cxx cpu_iface.f90 gpu_factor.f90 makemaster syrk.cu akeep.f90 cpu_solve.f90 gpu_interfaces.f90 profile.cxx updata_ssids anal.f90 cpu_subtree.f90 gpu_smalloc.f90 profile.hxx wrappers.cxx assemble.cu datatypes.f90 gpu_solve.f90 profile_iface.f90 cholesky.cxx dense_factor.cu gpu_subtree.f90 reorder.cu sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # I have the following files $GALAHAD/makefiles/: sandbox:${WORKSPACE}/srcdir/GALAHAD/makefiles # cat binarybuilder.bb.fc # Architecture dependent makefile # (automatically generated by install_galahad) VERSION = binarybuilder.bb.fc CUTESTVERSION = # Basic system commands CP = cp MV = mv RM = rm -f LN = ln MKDIR = mkdir SED = sed CAT = cat GREP = grep AR = ar ARREPFLAGS = -rcuU RANLIB = ranlib DLEXT = dylib LOADALL = -Wl,-all_load LOADNONE = # Directory for binaries PRECIS = double OBJ = $(GALAHAD)/objects/$(VERSION)/$(PRECIS) OBJS = $(GALAHAD)/objects/$(VERSION)/single OBJD = $(GALAHAD)/objects/$(VERSION)/double MOD = $(GALAHAD)/modules/$(VERSION)/$(PRECIS) CUTESTMOD = SEDS = $(GALAHAD)/seds/$(PRECIS).sed MVMODS = $(MV) -f $(OBJ)/*.mod $(MOD)/ || true # Compiler options FORTRAN = $(FC) BASIC = -c -fno-second-underscore -fPIC MBASIC = -fno-second-underscore -fPIC MODULES = -I$(MOD) OPTIMIZATION = -O2 NOOPTIMIZATION = -O0 DEBUG = SHARED = -shared OPENMP = -fopenmp SOMP = -fopenmp MOMP = -lgomp F77 = -ffixed-form F90 = F95 = NOFMAIN = USUAL = SPECIAL = F77SUFFIX = f90 F95SUFFIX = f90 TIMER = GEN NOT64 = IS64 PLPLOTUSED = unused IEEECK = AMPLDIR = CC = cc CCBASIC = -c -fPIC CCISO = -ansi -pedantic CCONDEF = CCDEBUG = #-DDEBUG_GALAHAD HWLOC = GETCPU = un CXX = c++ CXXBASIC = -c -std=c++11 -fPIC CXXOPT = -O2 CXXNOOPT = -O0 CXXONDEF = CXXDEBUG = CUDA = CUDACOMP = $(FC) CUDAARCH = CUDACODE = CUDABASIC = CUDALIBS = CUDAOPENMP = -fopenmp PYTHONVERSION = 0.0 PYTHONINCLUDE = /usr/include/python$(PYTHONVERSION) NUMPYINCLUDE = /usr/lib/python3/dist-packages/numpy/core/include SOABI = PYSITEPACKDIR = # Special flags MA97BUG = # Libraries PLPLOT = # -- BLAS -- BLAS = -lblastrampoline # -- LAPACK -- LAPACK = -lblastrampoline # -- HSL -- HSL = -lgalahad_hsl # -- SPRAL -- SPRAL = -lgalahad_spral -lstdc++ -lhwloc # -- Metis -- METIS = -lgalahad_metis # -- Pardiso -- PARDISO = -lgalahad_pardiso # -- MKL Pardiso -- MKL_PARDISO = -lgalahad_mkl_pardiso # -- WSMP -- WSMP = -lgalahad_wsmp # -- PASTIX -- PASTIX = -lgalahad_pastix # -- MUMPS -- MUMPS = -lgalahad_mumps # -- UMFPACK -- UMFPACK = -lgalahad_umfpack # proper or dummy SSIDS package? SSIDS = ssids # Shell used BINSHELL = bash # Set directories for optional packages include $(GALAHAD)/src/makedefs/packages # Body of makefile include $(PWD)/makemaster — Reply to this email directly, view it on GitHub<#18 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACW4A6RD47E63L2HZ3CWVPDWHKM3VANCNFSM6AAAAAARZZJBIY>. You are receiving this because you commented.Message ID: ***@***.***> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

[amontoison]

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f $GALAHAD/makefiles/binarybuilder.bb.fc
 Compiling guess_topology  		mkdir -p /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology
sed -f /workspace/srcdir/GALAHAD/seds/have_hwloc_def.sed ../spral/config.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h
sed -f /workspace/srcdir/GALAHAD/seds/have_getcpu_undef.sed /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config.h
cp  ../spral/compat.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/
cp  ../spral/guess_topology.cxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/
cp  ../spral/guess_topology.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/
cp  ../spral/hwloc_wrapper.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/
cd /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double ; c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O2   -fopenmp \
          guess_topology.cxx \
                 || ( printf ' %-26s' "=> Disabling optimization " ; \
                    c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O0  -fopenmp \
                      guess_topology.cxx )
guess_topology.cxx:13:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
 => Disabling optimization guess_topology.cxx:13:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
make: *** [/workspace/srcdir/GALAHAD/src/ssids/makemaster:310: /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/libgalahad_spral.a(guess_topology.o)] Error 1

[nimgould]

Are c and c++ valid c compilers on your system? Are they aliased somehow to gcc and g++? Sorry I don't know what is available. -fopenmp is a gcc etc flag, and should pull in omp.h Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Alexis ***@***.***> Sent: Tuesday, November 8, 2022 7:03:58 PM To: ralna/GALAHAD ***@***.***> Cc: Gould, Nick (STFC,RAL,SC) ***@***.***>; Comment ***@***.***> Subject: Re: [ralna/GALAHAD] 'omp.h' file not found on Platform("aarch64", "macos") (Issue #18) sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f $GALAHAD/makefiles/binarybuilder.bb.fc Compiling guess_topology mkdir -p /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology sed -f /workspace/srcdir/GALAHAD/seds/have_hwloc_def.sed ../spral/config.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h sed -f /workspace/srcdir/GALAHAD/seds/have_getcpu_undef.sed /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config.h cp ../spral/compat.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/ cp ../spral/guess_topology.cxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/ cp ../spral/guess_topology.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/ cp ../spral/hwloc_wrapper.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/ cd /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double ; c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O2 -fopenmp \ guess_topology.cxx \ || ( printf ' %-26s' "=> Disabling optimization " ; \ c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O0 -fopenmp \ guess_topology.cxx ) guess_topology.cxx:13:10: fatal error: 'omp.h' file not found #include <omp.h> ^~~~~~~ 1 error generated. => Disabling optimization guess_topology.cxx:13:10: fatal error: 'omp.h' file not found #include <omp.h> ^~~~~~~ 1 error generated. make: *** [/workspace/srcdir/GALAHAD/src/ssids/makemaster:310: /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/libgalahad_spral.a(guess_topology.o)] Error 1 — Reply to this email directly, view it on GitHub<#18 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACW4A6SMLNKS67U4FJ42YB3WHKPZ5ANCNFSM6AAAAAARZZJBIY>. You are receiving this because you commented.Message ID: ***@***.***> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

[amontoison]

Yes, cc and c++ are alias for C and C++ compilers.
It's gcc and g++ for Linux / Windows and Clang / Clang++ on Mac and FreeBSD.

[dpo]

could you try gcc/gfortran for macOS?

[amontoison]

Yes, I restart the compilation.
I just checked and we use gcc/gfortran for SPRAL:
https://github.com/JuliaPackaging/Yggdrasil/blob/master/S/SPRAL/build_tarballs.jl#L16-L19

@jfowkes
Do you test the compilation of SPRAL on Mac with CI?
It seems that modifications are needed for Clang / Clang++ compilers (native compilers on Mac / FreeBSD).

[dpo]

I would suggest that we forget about clang for now, and first stabilize everything with gcc/gfortran on every platform of interest.

[nimgould]

I believe that Alexis mentioned the work-around here. If
c++ is from clang, one needs SOMP = '-fopenmp=libomp'
in the compiler.binarybuilder.bb.fc file in archdefs, which
is copied into GALAHAD's makefiles/binarybuilder.bb.fc
on installation. Give that a try ... but I agree with Dominique that
we should concentrate on gcc for the time being.

[jfowkes]

@amontoison SPRAL is only tested on Linux GCC in the CI but I have successfully compiled it with Homebrew GCC on Mac.

I believe it would take quite a bit of nontrivial work to make SPRAL work with Clang given how complex it is.

[nimgould]

I tried flang/clang, and the flag

SOMP = -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include

did the trick here ... but as you see it needs to know where the system header is. The alternative

-I/usr/lib/gcc/x86_64-linux-gnu/11/include/omp.h

also worked, and is marginally more portable.

Unfortunately, a following ssids compile

clang++ -I. -o NumericSubtree.o -c -std=c++11 -fPIC -O -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include NumericSubtree.cxx

produces a wave of errors, typically
./ssids/cpu/kernels/assemble.hxx:212:42: error: variable 'add_a_blk_sz' must have explicitly specified data sharing attributes
add_a_block(iblk, std::min(iblk+add_a_blk_sz,snode.num_a), node, aval, scaling);

As I don't know c++, I have no idea what this means, and I can't find any compiler flag to disable it (if that is even possible). This does not happen with icx which is also built on top of LLVM, so I suspect it is just clang being too fussy.

[jfowkes]

Looking into the clang++ error, this is an issue with how Clang interprets the OpenMP standard with regards to const variables and the default(none) directive: Clang is of the opinion that const add_a_blk_sz should have its data sharing attribute explicitly specified (e.g. shared, private, etc, see https://www.openmp.org/spec-html/5.0/openmpsu106.html ). Technically I'd say Clang is correct here as the default(none) clause on line 209 should require that all data sharing attributes be explicitly specified as that is the whole point of default(none) (much like implicit none forces all types to be specified in Fortran). Basically, Clang doesn't know whether add_a_blk_sz should be shared or not within the for loop on line 212 whereas GCC assumes it to be one or the other (presumably shared?). A fix we have implemented for previous occurrences of this issue that GCC broke on is to simply remove the default(none) clause (so that such const variables simply use the default data sharing attribute) which has the benefit of being backwards compatible: ralna/spral@6547e55 Jari

…

________________________________ From: nimgould ***@***.***> Sent: 09 November 2022 11:50 To: ralna/GALAHAD ***@***.***> Cc: Fowkes, Jaroslav (STFC,RAL,SC) ***@***.***>; Mention ***@***.***> Subject: Re: [ralna/GALAHAD] 'omp.h' file not found on Platform("aarch64", "macos") (Issue #18) I tried flang/clang, and the flag SOMP = -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include did the trick here ... but as you see it needs to know where the system header is. The alternative

-I/usr/lib/gcc/x86_64-linux-gnu/11/include/omp.h also worked, and is marginally more portable. Unfortunately, a following ssids compile clang++ -I. -o NumericSubtree.o -c -std=c++11 -fPIC -O -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include NumericSubtree.cxx produces a wave of errors, typically ./ssids/cpu/kernels/assemble.hxx:212:42: error: variable 'add_a_blk_sz' must have explicitly specified data sharing attributes add_a_block(iblk, std::min(iblk+add_a_blk_sz,snode.num_a), node, aval, scaling); As I don't know c++, I have no idea what this means, and I can't find any compiler flag to disable it (if that is even possible). This does not happen with icx which is also built on top of LLVM, so I suspect it is just clang being too fussy. — Reply to this email directly, view it on GitHub<#18 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKT2EEXIMBB3J77OSCQLJLWHOFXDANCNFSM6AAAAAARZZJBIY>. You are receiving this because you were mentioned.Message ID: ***@***.***> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

[nimgould]

Very interesting. In the block

  #pragma omp taskgroup
  for(int iblk=0; iblk<snode.num_a; iblk+=add_a_blk_sz) {
     #pragma omp task default(none) \
        firstprivate(iblk) \
        shared(snode, node, aval, scaling, ldl)
     add_a_block(iblk, std::min(iblk+add_a_blk_sz,snode.num_a), node, aval, scaling);
  }

what precisely would you remove? I'm willing to try, but need hand holding

[jfowkes]

If my understanding is correct then removing the default(none) after #pragma omp task should fix it.

[nimgould]

I'll try tomorrow once I "get back" from Edinburgh

[amontoison]

Ok, so the issue is in spral and not in GALAHAD. We could move the issue to the GitHub SPRAL repository directly.
I agree with Dominique, we should compile GALAHAD with gcc/gfortran on all relevant platforms.

@jfowkes I did two CI scripts for a C++ code a few months ago, you can easily adapt them for SPRAL:
https://github.com/bbopt/nomad/blob/master/.github/workflows/ci.yml
https://github.com/bbopt/nomad/blob/master/.cirrus.yml

[jfowkes]

Thanks @amontoison, I've created a SPRAL pull request to test with Clang/Flang:
ralna/spral#97
Flang doesn't seem to be working though, any suggestions? We should discuss there.

[nimgould]

Good call, Jari. removing the three default (none)s fixed the compiler errors, and the sls exhaustive test now runs ... and ssids gives the right answer. I should add that this is with flang/clang as compiled from https://github.com/flang-compiler/llvm.git . Next I need to make sure that these changes don't ruin ssids under other compilers. Progress on an otherwise hopeless day!

[nimgould]

Oh, I should add that I needed to set the galahad variables
OPENMP = -fopenmp -lstdc++ -lhwloc
SOMP = -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include
and the other is of course not portable (but presumably will be when flang is finally
released as part of clang

[nimgould]

OK, the updates also work for gcc and icc. icx fails, but I think it did before. Jari, are you planning to make these changes in spral as well?

[jfowkes]

@nimgould yes I will make these changes to SPRAL, could you point me to the three default(none) statements you removed?

[nimgould]

Yes, in assemble.hxx (as per $GALAHAD/src/ssids/cpu/kernel/), lines 209, 281 and 389

[jfowkes]

@nimgould done, fixed in SPRAL.