Coder Social home page Coder Social logo

key4hep-spack's Introduction

Key4hep

.. toctree::
    :maxdepth: 3
    :includehidden:
    :caption: Contents:

    setup-and-getting-started/README.md
    how-tos/README.md
    tutorials/README.md
    developing-key4hep-software/README.md
    spack-build-instructions-for-librarians/README.md
    talks-and-presentations/README.md
    call-for-logos/README.md
    CONTRIBUTING.md

.. toctree::
    :maxdepth: 2
    :includehidden:
    :caption: External links:

    FCC software  <https://cern.ch/fccsw>
    CLIC software <https://twiki.cern.ch/twiki/bin/view/CLIC/CLICSoftwareComputing>
    ILC software <https://ilcsoft.desy.de/portal>
    CEPC software <http://cepcsoft.ihep.ac.cn/>
    Muon Collider software <https://mcd-wiki.web.cern.ch/software/>

key4hep-spack's People

Contributors

andresailer avatar bartosik-hep avatar bcouturi avatar brettviren avatar brieucf avatar clementhelsens avatar drbenmorgan avatar fdplacido avatar gartung avatar gganis avatar giovannimarchiori avatar graeme-a-stewart avatar haralmha avatar hegner avatar iarspider avatar javiercvilla avatar jlingema avatar jmcarcell avatar joequant avatar kjvbrt avatar mirguest avatar paulgessinger avatar scott-snyder avatar tmadlener avatar vvolkl avatar wdconinc avatar zehvogel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

key4hep-spack's Issues

Disable adding the latest commit as version if GITHUB_USER or GITHUB_TOKEN are not set?

In #61 the possibility to add the latest commit as version was added. If GITHUB_USER and GITHUB_TOKEN are not set this leads to a lot of warnings of the kind

Warning: could not fetch latest commit for edm4hep
Warning: could not fetch latest commit for k4fwcore
# similar for all packages

Since this is arguably an "expert" feature, maybe we should consider that we omit these checks if the two environment variables are not set, as querying the github api for several packages can lead to significantly longer times to concretize a spec that uses k4_add_latest_commit_as_[version|dependency]. Additionally, the emitted warning messages might be confusing for users who just want the latest stable version of everything.

Error trying to run tests with spack

~$ spack install --test=all key4hep-stack
==> Error: maximum recursion depth exceeded while calling a Python object

Setting sys.setrecursionlimit(10000) in spack/lib/spack/cmd/install.py didn't help, so probably a bug leading to infinite recursion?

DD4hep@master concretizing issues

Trying to build dd4hep against both geant4 and root from the main sw.hsf.org repository fails with the error:

~$ spack spec -I dd4hep@master ^/xjjdyis ^/mblf2s6
Input spec
--------------------------------

 -   dd4hep@master
[+]      ^[email protected]%[email protected]~motif~opengl~qt+threads~vecgeom~x11 build_type=RelWithDebInfo cxxstd=11 arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] build_type=RelWithDebInfo cxxstd=11 arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]+libbsd arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] cxxstd=11 transcoder=gnuiconv arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]+optimize+pic+shared arch=linux-centos7-broadwell
[+]      ^[email protected]%[email protected]~aqua+davix~emacs+examples~fftw~fits~fortran+gdml+gminimal~graphviz+gsl~http~jemalloc+math~memstat+minuit~mlp~mysql+opengl~postgres~pythia6~pythia8+python~qt4~r+root7+rootfit+rpath~shadow~sqlite+ssl~table+tbb+threads~tmva+unuran~vc+vdt~vmc+x+xml~xrootd build_type=RelWithDebInfo cxxstd=17 patches=22af3471f3fd87c0fe8917bf9c811c6d806de6c8b9867d30a1e3d383a1b929d7 arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] build_type=RelWithDebInfo cxxstd=11 arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected]~python arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected]+optimize+pic+shared arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected]+systemcerts arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected]+shared arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]+shared build_type=RelWithDebInfo patches=001908e385de3940afd29f2cf36133dd33cb8931194cd5c419c8bc8f3096e3f0 arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected]+glx~llvm+opengl~opengles+osmesa patches=55a5611ca9fcbe8324c4d68a07b338134954ff12c5b122dc78ff376f012a1414 swr=none arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected]+libbsd arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected]~symlinks+termlib arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]~doc+png+zlib build_type=RelWithDebInfo arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] build_type=RelWithDebInfo arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]~external-cblas arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]+shared+tm cxxstd=default patches=62ba015ebd1819c45bef47411540b789b493e31ca668c4ff4cb2afcbc306b476,ce1fb16fb932ce86a82ca87cf0431d1a8c83652af9f552b264213b2ff2945d73 arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-ttf,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]~jit+multibyte+utf arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]+blas+lapack arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                      ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]                  ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]~gsl+rngstreams+shared arch=linux-centos7-broadwell
[+]              ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] build_type=RelWithDebInfo arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected] arch=linux-centos7-broadwell
[+]          ^[email protected]%[email protected]+pic arch=linux-centos7-broadwell

Concretized
--------------------------------
==> Error: Invalid Spec DAG: [email protected]%[email protected]+libbsd arch=linux-centos7-broadwell does not satisfy [email protected]%[email protected]+libbsd arch=linux-centos7-broadwell ^[email protected]%[email protected] arch=linux-centos7-broadwell

fcalcclusterer fails to compile with stdc++-17

fcalcclusterer fails to compile when set to stdc++-17 because std::random_shuffle has been removed

/tmp/user/spack-stage/spack-stage-fcalclusterer-1.0.1-6uszwdiiq25ca4yyeofqasn6mufg6tmb/spack-src/source/BeamCalReco/src/BeamCalBkgPregen.cpp:70:8: error: no member named 'random_shuffle' in namespace 'std' [clang-diagnostic-error]
std::random_shuffle(bg_files.begin(), bg_files.end());
^
1 error generated.
Error while processing /tmp/user/spack-stage/spack-stage-fcalclusterer-1.0.1-6uszwdiiq25ca4yyeofqasn6mufg6tmb/spack-src/source/BeamCalReco/src/BeamCalBkgPregen.cpp.
Found compiler error(s).
:

Dump out version for k4-spack and builtin

Since the versions of k4-spack and builtin are constantly changing it may be difficult to track versions and create reproduciable builds. spack does a good job of tracking the versions of individual packages, but there is no current mechanism for recording what build was used.

One thing that could be done is to modify the package.py for key4hep-stack to dump out the commit id of the builtin and k4-spack versions so that this can be recorded. Also we can put some diagnostic tools in the key4hep-stack package to for example dump out the system characteristics of a build.

Thoughts?

$PATH env var: gcc 8.3.0 should be inserted before the system compiler in the setup script `/cvmfs/sw.hsf.org/key4hep/setup.sh`

Please see the following logs, the system path is inserted before gcc 8.3.0

[lint@lxslc703]$ source /cvmfs/sw.hsf.org/key4hep/setup.sh
[lint@lxslc703]$ echo $PATH | tr ":" "\n" | tail -n20
/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/ncurses-6.2-4gq3xlbh3azlfkfmnuyiak6xbhjexazt/bin
/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/expat-2.2.9-b6lho6srpc6xpvxundk4wlzrbzgnv242/bin
/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/bzip2-1.0.8-e272slzh2s7uketdxplns3bie3kpzxc2/bin
/home/vavolkl/spack/bin
/afs/cern.ch/user/v/vavolkl/scripts
/usr/sue/bin
/usr/local/bin
/usr/bin
/usr/local/sbin
/usr/sbin
/opt/puppetlabs/bin
/afs/cern.ch/sw/XML/texlive/latest/bin/x86_64-linux/
/afs/cern.ch/user/v/vavolkl/bin
/cvmfs/sw.hsf.org/spackages/linux-centos7-haswell/gcc-4.8.5/gcc-8.3.0-avsmzt7bekq7ispf6zlarx6vwdretbae/bin
/usr/lib64/qt-3.3/bin
/usr/local/bin
/usr/bin
/usr/local/sbin
/usr/sbin
/opt/puppetlabs/bin

Build failure

Looks like the Pythia API changed

distcc[35634] Warning: failed to distribute /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8_EDM4HEP.cpp to 172.17.0.1,lzo, running locally instead
In file included from /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8_EDM4HEP.cpp:3:
/tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8Reader.h: In member function 'virtual bool DelphesPythia8Reader::init(Delphes*, int, char**, std::string&)':
/tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8Reader.h:212:26: error: 'class Pythia8::CombineMatchingInput' has no member named 'getHook'; did you mean 'setHook'?
212 | matching = combined->getHook(pythia);
| ^~~~~~~
| setHook
/tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8Reader.h:217:29: error: cannot convert 'Pythia8::UserHooks
' to 'Pythia8::UserHooksPtr' {aka 'std::shared_ptrPythia8::UserHooks'}
217 | pythia->setUserHooksPtr(matching);
| ^~~~~~~~
| |
| Pythia8::UserHooks*
In file included from /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8Reader.h:22,
from /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/plugins/delphes/DelphesPythia8_EDM4HEP.cpp:3:
/opt/spack/linux-mageia8-skylake/gcc-10.2.0/pythia8-8303-hhdw6phfk25dyvxzxrw32gutitzaivv4/include/Pythia8/Pythia.h:136:37: note: initializing argument 1 of 'bool Pythia8::Pythia::setUserHooksPtr(Pythia8::UserHooksPtr)'
136 | bool setUserHooksPtr(UserHooksPtr userHooksPtrIn) {
| ~~~~~~~~~~~~~^~~~~~~~~~~~~~
cd /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-build && /opt/spack/linux-mageia8-skylake/gcc-10.2.0/cmake-3.18.2-2elvn6eghzdxgcinedwbqumkvhl6dw5n/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-src/edm4hep /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-build /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-build/edm4hep /tmp/user/spack-stage/spack-stage-edm4hep-0.2.0-pyko3gyskg7y4vyrjonx2xrqhx76aird/spack-build/edm4hep/CMakeFiles/edm4hepDict.dir/DependInfo.cmake --color=
Scanning dependencies of target edm4hepDict
make[2]: Leaving directory '/tmp/user/spack-stage/

Set up a share/data CVMFS space

Some data packages can be shared across all installations (this mostly concerns geant4-data right now). With future versions of spack, that can declare the compilers as dependencies this can be handled more elegantly, but right now there is already a good solution (install the data packages to /cvmfs/sw.hsf.org/spackages/share/data, and set them as externals with buildable: False in the installations)

Interaction between spack and other package managers (rpm, conda, flatpak, deb)

This is a general issue for describing interactions between spack and other package managers. Right now I've got spack working as an rpm for Mageia linux. Also would like to figure out how to get this to work with packaging systems such as conda, flatpack, etc. etc.

https://rpmfind.net/linux/RPM/mageia/cauldron/aarch64/media/core/release/spack-0.14.2-2.mga8.noarch.html

The reason for installing spack as an RPM is so that a user can have an spack distribution out of the box and can install a distribution and have an spack environment ready to roll. This is particularly useful in situations where you want to use virtualization workflows from scratch.

Part of the issue is that HEP users have different workflows than enterprise users. HEP users would prefer to create their own environment in their home directories and then tweak the system for maximum performance Enterprise users typically require that a system admin control software installs centrally.

Trying to figure out how to get spack to work with other package managers is of limited use to academic users right now, but there are two larger issues.

  1. there is starting to be a flowering of different package systems, namely rpm, conda, flatpak, and the like. Having everyone work with their package in their own ecosystem will eventually result in huge images with a lot of duplication

  2. using package manager X gets you the support and tools for ecosystem X. For example, by putting spack as an rpm for Mageia Linux, spack gets distributed on their servers and the like.

Right now, I'm looking into these issues and seeing what's possible. Maybe some sort of "view" system where something looks like an rpm to an rpm system and spack/flatpak to another system. Or a "sandbox" virtual environment model.

GNU compiler issue when sourcing edm4hep

When sourcing EDM4Hep

source /cvmfs/sw.hsf.org/spackages/edm4hep.sh

The GNU compilers stays default. So on lxplus we are left using a very old version 4.8.5.

The setup script needs some adjustments.

Build failure: whizard@master

Building from the master branch of whizard requires a different build procedure than building from the tarballs (configure has to be created first). Need to update the package recipe for this (or drop the master).

@joequant did you encounter this in your nightly builds by any chance?

Problem: pick the system GSL (1.15) when building marlinutil

I have tried to install key4hep-stack from scratch on my computer (scientific linux 7.5, gcc 8.3.0), without the external libraries from /cvmfs/sw.hsf.org. The problem is that even though the spack already installs the GSL 2.5, but when building marlinutil, it chooses the system GSL.

1 error found in build log:
     64    -- Check for CED_CED_LIBRARY: /home/lintao/k4spack-without-hsf/spack/opt/spack/linux-scientific7-broadwell/gcc-8.3.0/ced-1.9.3-cehr3rif2rrxmwdueeladstn4mp2xmec/lib/libCED.so -- ok
     65    -- Found CED: /home/lintao/k4spack-without-hsf/spack/opt/spack/linux-scientific7-broadwell/gcc-8.3.0/ced-1.9.3-cehr3rif2rrxmwdueeladstn4mp2xmec (Required is at least version "1.4")
     66    -- Check for GSL (1.15)
     67    -- Check for GSL_LIBRARIES: gsl;gslcblas
     68    -- Check for GSL_GSL_LIBRARY: /usr/lib64/libgsl.so -- ok
     69    -- Check for GSL_GSLCBLAS_LIBRARY: /usr/lib64/libgslcblas.so -- ok
  >> 70    CMake Error at /home/lintao/k4spack-without-hsf/spack/opt/spack/linux-scientific7-broadwell/gcc-8.3.0/cmake-3.18.2-vfaqkt6bcebp5f7u74dnpwt7mkmqdbnw/share/cmake-3.18/Modules/FindPackageHandleStandardArg
           s.cmake:165 (message):
     71      Could NOT find GSL (missing: PACKAGE_VERSION_COMPATIBLE) (Required is at
     72      least version "2.1")
     73    Call Stack (most recent call first):
     74      /home/lintao/k4spack-without-hsf/spack/opt/spack/linux-scientific7-broadwell/gcc-8.3.0/cmake-3.18.2-vfaqkt6bcebp5f7u74dnpwt7mkmqdbnw/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (
           _FPHSA_FAILURE_MESSAGE)
     75      /home/lintao/k4spack-without-hsf/spack/opt/spack/linux-scientific7-broadwell/gcc-8.3.0/ilcutil-1.6-x2hyzkx34usdcc2znxudohfxelpk7t4e/cmakemodules/FindGSL.cmake:84 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
     76      CMakeLists.txt:28 (FIND_PACKAGE)


See build log for details:
  /tmp/lintao/spack-stage/spack-stage-marlinutil-1.15.1-zun2vhuaj7ae2yejxzpavid7aengpzon/spack-build-out.txt

It looks like the FindGSL.cmake from ilcutil is used.

Getting an error

key4hep-container@ run-incremental-nightly /home/joe/local2/bench/git/key4hep-container/buildbot/worker2/run-incremental-nightly/build
podman run -ti -u user --group-add wheel --group-add spack -v key4hep-container-nightly_home:/home -v key4hep-container-nightly_opt:/opt joequant/key4hep-container /usr/sbin/build-spack.sh key4hep-stack@nightly
running proxy
running distcc
~/.spack/linux /
/
==> Error: name 'HipPackage' is not defined
 ERROR  Command failed with exit code 1.

Better Handling of BINARY_TAG in K4FWCore Recipe

The recipe for K4FWCore tries to construct the environment variable from information that spack can provide about the system. The Gaudi cmake configuration fails if this variable doesn't match the one with which Gaudi was configured, so this is good in principle, however it currently works only for centos7 and slc6. It may be better to check if this variable was already set for the system (Gaudi should have set it).

Spack package install path needs to be longer than new path after relocation

==> Error: New path longer than old path: binary text replacement not possible.
The new path /cvmfs/sw-nightlies.hsf.org/key4hep/nightlies/LCG_96c_LS/linux-scientific7-broadwell/gcc-8.3.0/py-pyyaml-5.1.2-yvwxzwo4td2qelzt2w7taftoseigkbra is longer than the old path /home/gitlab-runner/spack/opt/spack/linux-scientific7-broadwell/gcc-8.3.0/py-pyyaml-5.1.2-yvwxzwo4td2qelzt2w7taftoseigkbra.
Text replacement in binaries will not work.
Create buildcache from an install path longer than new path.

LLVM and Gaudi conflict

LLVM 11 needs python 3.8 but Gaudi needs 3.7

Is there any way to fix package.yaml so that it builds both?

Fix Fork Relationship of Repository

This is not really a fork of hsf/hep-spack anymore, but k4-spack being a fork tends to mess up pull requests. I think there is not other way but to delete and recreate this repository in order to get rid of the fork information.

Unsatisfiable variant constraint in key4hep-stack spec

When trying to concretize the key4hep-stack spec spack runs into an

$ spack spec key4hep-stack 
Input spec
--------------------------------
key4hep-stack

Concretized
--------------------------------

==> Error: An unsatisfiable variant constraint has been detected for spec:

[email protected]%[email protected]~aqua build_type=RelWithDebInfo cxxstd=11 +davix~emacs+examples~fftw~fits~fortran+gdml+gminimal~graphviz+gsl~http~jemalloc+math~memstat+minuit~mlp~mysql+opengl~postgres~pythia6~pythia8+python~qt4~r+root7+rootfit+rpath~shadow~sqlite+ssl~table+tbb+threads~tmva+unuran~vc+vdt~vmc+x+xml~xrootd arch=linux-ubuntu18.04-sandybridge
        ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu18.04-sandybridge
            ^[email protected]%[email protected]~symlinks+termlib arch=linux-ubuntu18.04-sandybridge
                ^[email protected]%[email protected] arch=linux-ubuntu18.04-sandybridge
[...]

while trying to concretize the partial spec:

    [email protected]%[email protected] build_type=Release cxxstd=17 arch=linux-ubuntu18.04-sandybridge
        ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu18.04-sandybridge
            ^[email protected]%[email protected]~symlinks+termlib arch=linux-ubuntu18.04-sandybridge
                ^[email protected]%[email protected] arch=linux-ubuntu18.04-sandybridge
            ^[email protected]:
                ^[email protected]:
                    ^gdbm
                        ^readline
                ^[email protected]%[email protected]+optimize+pic+shared arch=linux-ubuntu18.04-sandybridge


podio requires root variant cxxstd=17, but spec asked for cxxstd=11

A similar thing happens when trying to do the same thing for edm4hep

The problem is I think that root by default uses cxxstd=11 but that is not available for podio. Having a suitable root version in the system packages.yaml with cxxstd=17 (or 14) makes everything work.

build error for key4hep-spack

==> Error: An unsatisfiable version constraint has been detected for spec:
    [email protected]%[email protected]~ipo build_type=Release cxxstd=17 arch=linux-mageia8-broadwell
        ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-mageia8-broadwell
            ^[email protected]%[email protected]~symlinks+termlib arch=linux-mageia8-broadwell
                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
            ^[email protected]:
                ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]+optimize+pic+shared arch=linux-mageia8-broadwell
        ^[email protected]:34.99
            ^aida
            ^[email protected]%[email protected]+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options+python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]+shared arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-mageia8-broadwell
                    ^expat
                    ^[email protected]:+libxml2
                        ^libxml2
                            ^[email protected]%[email protected]~pic arch=linux-mageia8-broadwell
                    ^libffi
                    ^libuuid
                    ^[email protected]:
            ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo cxxstd=11 arch=linux-mageia8-broadwell
            ^cppgsl
            ^intel-tbb
            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
            ^py-xenv@1:
                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
            ^range-v3
            ^[email protected]%[email protected]~aqua+davix~emacs+examples~fftw~fits~fortran+gdml+gminimal~graphviz+gsl~http~ipo~jemalloc+math~memstat+minuit+mlp~mysql+opengl~postgres~pythia6+pythia8+python~qt4+r+root7+rootfit+rpath~shadow+sqlite+ssl~table+tbb+threads+tmva+unuran+vc+vdt+vmc+x+xml+xrootd build_type=RelWithDebInfo cxxstd=17 arch=linux-mageia8-broadwell
                ^[email protected]:
                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                ^[email protected]:
                    ^[email protected]%[email protected]+glx~llvm+opengl~opengles+osmesa buildtype=release swr=none arch=linux-mageia8-broadwell
                        ^binutils
                        ^bison
                            ^help2man
                            ^[email protected]:
                        ^flex
                            ^findutils
                        ^[email protected]:
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^libbsd
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^libxt
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^libsm
                        ^[email protected]:
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]:
                            ^[email protected]:
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                ^gl2ps
                    ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^libdrm
                        ^[email protected]:
                            ^libtool
                    ^libxdamage
                        ^[email protected]:
                    ^libxmu
                    ^libxxf86vm
                        ^[email protected]:
                ^glew
                ^[email protected]%[email protected]~external-cblas arch=linux-mageia8-broadwell
                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^nasm
                ^libxft
                    ^[email protected]:
                        ^font-util
                            ^autoconf
                            ^automake
                            ^bdftopcf
                                ^[email protected]:
                                ^libxfont
                                    ^libfontenc
                            ^mkfontdir
                                ^mkfontscale
                ^libxpm
                ^lz4
                ^[email protected]%[email protected]~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]~jit+multibyte+utf arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]+blas+lapack arch=linux-mageia8-broadwell
                    ^[email protected]:
                ^pythia8
                    ^rsync
                        ^popt
                ^[email protected]:
                    ^cairo+fc+ft+pdf
                        ^glib
                        ^pixman
                    ^curl
                        ^libidn2
                            ^libunistring
                    ^icu4c
                    ^java
                    ^libtiff
                    ^pango
                        ^gobject-introspection
                        ^harfbuzz
                    ^tcl
                ^[email protected]:
                ^r-rinside
                ^unuran
                ^vc
                ^vdt
                ^xrootd@:4.99.99
                ^xxhash
                ^zstd
        ^[email protected]%[email protected]~ipo~sio build_type=Release arch=linux-mageia8-broadwell
            ^[email protected]:
                ^[email protected]:
                    ^[email protected]:
            ^[email protected]%[email protected]+libyaml arch=linux-mageia8-broadwell
                ^libyaml
while trying to concretize the partial spec:
    [email protected]%[email protected]~ipo build_type=RelWithDebInfo cxxstd=17 arch=linux-mageia8-broadwell
        ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo cxxstd=11 arch=linux-mageia8-broadwell
            ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]~symlinks+termlib arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                ^[email protected]:
                    ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected]+optimize+pic+shared arch=linux-mageia8-broadwell
        ^[email protected]%[email protected]~assimp~debug+geant4~hepmc3~ipo+lcio~xercesc build_type=RelWithDebInfo arch=linux-mageia8-broadwell
            ^[email protected]%[email protected]+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options+python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]+shared arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-mageia8-broadwell
                    ^expat
                    ^[email protected]:+libxml2
                        ^libxml2
                            ^[email protected]%[email protected]~pic arch=linux-mageia8-broadwell
                    ^libffi
                    ^libuuid
                    ^[email protected]:
            ^[email protected]:
            ^[email protected]%[email protected]~examples~ipo~jar+rootdict build_type=RelWithDebInfo cxxstd=17 arch=linux-mageia8-broadwell
                ^[email protected]%[email protected]~aqua+davix~emacs+examples~fftw~fits~fortran+gdml+gminimal~graphviz+gsl~http~ipo~jemalloc+math~memstat+minuit+mlp~mysql+opengl~postgres~pythia6+pythia8+python~qt4+r+root7+rootfit+rpath~shadow+sqlite+ssl~table+tbb+threads+tmva+unuran+vc+vdt+vmc+x+xml+xrootd build_type=RelWithDebInfo cxxstd=17 arch=linux-mageia8-broadwell
                    ^[email protected]:
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^[email protected]:
                        ^[email protected]%[email protected]+glx~llvm+opengl~opengles+osmesa buildtype=release swr=none arch=linux-mageia8-broadwell
                            ^binutils
                            ^bison
                                ^help2man
                                ^[email protected]:
                            ^flex
                                ^findutils
                            ^[email protected]:
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                        ^libbsd
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^libxt
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^libsm
                            ^[email protected]:
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                            ^[email protected]:
                                ^[email protected]:
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                    ^gl2ps
                        ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=linux-mageia8-broadwell
                            ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                                ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^libdrm
                            ^[email protected]:
                                ^libtool
                        ^libxdamage
                            ^[email protected]:
                        ^libxmu
                        ^libxxf86vm
                            ^[email protected]:
                    ^glew
                    ^[email protected]%[email protected]~external-cblas arch=linux-mageia8-broadwell
                    ^intel-tbb
                    ^[email protected]%[email protected] arch=linux-mageia8-broadwell
                        ^nasm
                    ^libxft
                        ^[email protected]:
                            ^font-util
                                ^autoconf
                                ^automake
                                ^bdftopcf
                                    ^[email protected]:
                                    ^libxfont
                                        ^libfontenc
                                ^mkfontdir
                                    ^mkfontscale
                    ^libxpm
                    ^lz4
                    ^[email protected]%[email protected]~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected]~jit+multibyte+utf arch=linux-mageia8-broadwell
                    ^[email protected]%[email protected]+blas+lapack arch=linux-mageia8-broadwell
                        ^[email protected]:
                    ^pythia8
                        ^rsync
                            ^popt
                    ^[email protected]:
                        ^cairo+fc+ft+pdf
                            ^glib
                            ^pixman
                        ^curl
                            ^libidn2
                                ^libunistring
                        ^icu4c
                        ^java
                        ^libtiff
                        ^pango
                            ^gobject-introspection
                            ^harfbuzz
                        ^tcl
                    ^[email protected]:
                    ^r-rinside
                    ^unuran
                    ^vc
                    ^vdt
                    ^xrootd@:4.99.99
                    ^xxhash
                    ^zstd
        ^[email protected]%[email protected]+ddg4_edm4hep_plugin+delphes~ipo build_type=Release cxxstd=17 arch=linux-mageia8-broadwell
            ^[email protected]%[email protected]~ipo+pythia8 build_type=RelWithDebInfo cxxstd=17 arch=linux-mageia8-broadwell
            ^[email protected]%[email protected]~ipo~sio build_type=Release arch=linux-mageia8-broadwell
                ^[email protected]:
                    ^[email protected]:
                        ^[email protected]:
                ^[email protected]%[email protected]+libyaml arch=linux-mageia8-broadwell
                    ^libyaml
cepcsw requires k4fwcore version 0.1.1, but spec asked for 0.2.0
 ERROR  Command failed with exit code 1.
program finished with exit code 1
elapsedTime=33.010532

Problems building whizard

I am unable to get whizard to build within spack. I get the following error:

==> Installing whizard
==> No binary for whizard found: installing from source
==> Warning: Suspicious requests to set or unset 'CC' found
==> Warning: 	    	env.set('CC', os.path.join(link_dir, compiler.link_paths['cc'])) at /spack/lib/spack/spack/build_environment.py:225
==> Warning: 	--->	env.set('CXX', self.compiler.cxx) at /spack/var/spack/repos/builtin/packages/whizard/package.py:78
==> Warning: Suspicious requests to set or unset 'CXX' found
==> Warning: 	    	env.set('CXX', os.path.join(link_dir, compiler.link_paths['cxx'])) at /spack/lib/spack/spack/build_environment.py:228
==> Warning: 	--->	env.set('FC', self.compiler.fc) at /spack/var/spack/repos/builtin/packages/whizard/package.py:79
==> Warning: Suspicious requests to set or unset 'F77' found
==> Warning: 	    	env.set('F77', os.path.join(link_dir, compiler.link_paths['f77'])) at /spack/lib/spack/spack/build_environment.py:231
==> Warning: 	--->	 at /spack/var/spack/repos/builtin/packages/whizard/package.py:81
==> Warning: Suspicious requests to set or unset 'FC' found
==> Warning: 	    	env.set('FC', os.path.join(link_dir, compiler.link_paths['fc'])) at /spack/lib/spack/spack/build_environment.py:234
==> Warning: 	--->	env.set('F77', self.compiler.fc) at /spack/var/spack/repos/builtin/packages/whizard/package.py:80
==> Fetching https://whizard.hepforge.org/downloads/?f=whizard-2.8.5.tar.gz
######################################################################## 100.0%
==> whizard: Executing phase: 'autoreconf'
==> whizard: Executing phase: 'configure'
==> whizard: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j12'

4 errors found in build log:
     801    cp -f bigarray_module.ml bigarray_compat.ml
     802      OCAMLC    bigarray_compat.cmi
     803      OCAMLC    events.cmi
     804      OCAMLOPT  bigarray_compat.cmx
     805      OCAMLOPT  float.cmx
     806    File "bigarray_compat.ml", line 1:
  >> 807    Error: Could not find the .cmi file for interface bigarray_compat.mli.
     808      OCAMLC    diffmaps.cmi
  >> 809    make[2]: *** [bigarray_compat.cmx] Error 2
     810    make[2]: *** Waiting for unfinished jobs....
     811    make[2]: Leaving directory `/tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-src/circe2/src'
  >> 812    make[1]: *** [all-recursive] Error 1
     813    make[1]: Leaving directory `/tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-src/circe2'
  >> 814    make: *** [all-recursive] Error 1

See build log for details:
  /tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-build-out.txt

I can consistently reproduce it using the following container running centos7: https://hub.docker.com/repository/docker/tmadlener/ilcsof-cvmfs-install-centos7

One way to reproduce it is to first build a suitable gcc (e.g. [email protected] or [email protected]) using spack inside the container and then try to build whizard. However, this will take a few hours until all the necessary dependencies are built. A quicker way is to use the cvmfs installed packages as upstream. To do so, make sure to locally mount /cvmfs/sw.hsf.org and then start the container

docker run -it -v /cvmfs/sw.hsf.org:/cvmfs/sw.hsf.org tmadlener/ilcsof-cvmfs-install-centos7

Inside the container, setup spack and the correct environment

git clone https://github.com/spack/spack
git clone https://github.com/key4hep/k4-spack
. spack/share/spack/setup-env.sh
mkdir -p ~/.spack/linux
cp k4-spack/config/packages.yaml ~/.spack/linux

Add the upstream repository and load the pre-installed [email protected] and add it to the known compilers for spack

cat <<EOT >> $HOME/.spack/linux/upstreams.yaml
upstreams:
  spack-instance-1:
      install_tree: /cvmfs/sw.hsf.org/spackages/
EOT

spack load [email protected] && spack compiler find

Now check that all the dependencies are present (specify a pythia8 version that is already installed, to not have to build the new default version 8303).

spack spec -Il whizard ^pythia8@8244

This should show only [email protected]/v7sytfv as missing package and trying to install it leads to the above described error

spack install whizard ^pythia8@8244

For completeness spack debug report yields the following as of writing this issue.

* **Spack:** 0.15.4-1359-796561e
* **Python:** 2.7.5
* **Platform:** linux-centos7-skylake

"Illegal instruction" error using stack build

Hi,

I am trying to setup the fcc environment on my centos 7 machine [1] (non lxplus) using the spack build by calling

source /cvmfs/fcc.cern.ch/sw/latest/setup.sh

as detailed in the startup kit at

https://hep-fcc.github.io/fcc-tutorials/fast-sim-and-analysis/FccFastSimAnalysis.html

When I do that, I get no strange output from the setup script but then basically all commands give "Illegal instructions" e.g.

[gwilliam@phi build]$ fccrun Illegal instruction (core dumped) [gwilliam@phi build]$ diff Illegal instruction (core dumped)

I talked about this with Clement Helsens and he suggested I try an LCG build rather than a spack build by calling instead

https://github.com/HEP-FCC/FCCSW/blob/master/init_lcg_97a_FCC_2.sh

This works fine.

Cheers,

Carl

My system is

`[gwilliam@phi build]$ echo $SHELL
/bin/bash
[gwilliam@phi build]$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[gwilliam@phi build]$ uname -a
Linux phi.ph.liv.ac.uk 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux\

gaudimarlinwrapper looking for cmake file not in Gaudi 35.0

This seems to be because the files are different with Gaudi 35.0

CMake Warning at CMakeLists.txt:50 (find_package):
By not providing "FindGaudiProject.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"GaudiProject", but CMake did not find one.
Could not find a package configuration file provided by "GaudiProject" with
any of the following names:
GaudiProjectConfig.cmake
gaudiproject-config.cmake
Add the installation prefix of "GaudiProject" to CMAKE_PREFIX_PATH or set
"GaudiProject_DIR" to a directory containing one of the above files. If
"GaudiProject" provides a separate development package or SDK, be sure it
has been installed.
CMake Error at CMakeLists.txt:55 (gaudi_project):
Unknown CMake command "gaudi_project".

Problem building whizard using cvmfs packages as upstreams

While investigating #71 I found a second build error, that might occur when building whizard. In a parallel build the order in which the two surface is not deterministic, but this error seems to be consistently the first to be hit, when not building in parallel. So following the instructions described in #71 and then doing

spack install -j 1 whizard ^python@8244

should trigger this.

The error output is:

==> Error: ProcessError: Command exited with status 2:
    'make'

5 errors found in build log:
     844      OCAMLC    lexer.cmi
     845      OCAMLOPT  lexer.cmx
     846      OCAMLC    commands.cmi
     847      OCAMLOPT  commands.cmx
     848      OCAMLOPT  circe2_tool.cmx
     849      OCAMLOPT  circe2_tool.opt
  >> 850    sh: /home/vavolkl/spack/lib/spack/env/gcc/gcc: No such file or directory
     851    File "caml_startup", line 1:
  >> 852    Error: I/O error: /home/vavolkl/spack/lib/spack/env/gcc/gcc -O2 -fno-strict-aliasing -fwrapv -Wall -fno-common -fno-tree-vrp -ffunction-sections -D_FILE_OFFSET_BITS=64 -D_REENTRANT -DCAML_NAME_SPACE  -Wl,-E -o 'circe2_tool.opt' -L/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/ocaml-4.10.0-yizmnfrjhe63iebtom6xlxbmc4zq4m4l/lib/ocaml'  '/tmp/camlstartup0f5a4c.o' '/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/ocaml-4.10.0-yizmnfrjhe63iebtom6xlxbmc4zq4m4l/lib/ocaml/std_exit.o' 'circe2_tool.o' 'histogram.o' 'commands.o' 'parser.o' 'lexer.o' 'syntax.o' 'events.o' 'grid.o' 'division.o' 'filter.o' 'diffmaps.o' 'diffmap.o' 'float.o' 'thoMatrix.o' 'thoArray.o' 'OUnitDiff.o' 'OUnit.o' 'bigarray_compat.o' '/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/ocaml-4.10.0-yizmnfrjhe63iebtom6xlxbmc4zq4m4l/lib/ocaml/unix.a' '/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/ocaml-4.10.0-yizmnfrjhe63iebtom6xlxbmc4zq4m4l/lib/ocaml/stdlib.a' '-lunix' '/cvmfs/sw.hsf.org/spackages/linux-centos7-broadwell/gcc-8.3.0/ocaml-4.10.0-yizmnfrjhe63iebtom6xlxbmc4zq4m4l/lib/ocaml/libasmrun.a' -lm -ldl
  >> 853    make[2]: *** [circe2_tool.opt] Error 2
     854    make[2]: Leaving directory `/tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-src/circe2/src'
  >> 855    make[1]: *** [all-recursive] Error 1
     856    make[1]: Leaving directory `/tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-src/circe2'
  >> 857    make: *** [all-recursive] Error 1

Warnings about missing version informations when using ncurses+termlib

Loading this library (dependency of ROOT et al.) results in constant warnings of:

/bin/bash: /cvmfs/sw.hsf.org/spackages/linux-ubuntu20.04-broadwell/gcc-9.3.0/ncurses-6.2-c5onutekmwbsn3rq2tjq3x5jj746xjtj/lib/libtinfo.so.6: no version information available (required by /bin/bash)

as far as I can tell this can be simply avoiding by using ncurses~termlib

build errors on gcc10

I'm getting build errors trying to build on gcc10. There are compiler failures on

edm4hep and fcc-edm

would appreciate some help in figuring out to make these build

spack.log

Add common python tools

Those python modules are really useful for user analyses. I would recommend we had them to the stack when possible.

uproot4
awkward1
matplotlib
particle
tensorflow
zfit

Always enable lcio variant for DD4hep

For the key4hep-stack the dd4hep +lcio variant is enabled (#55). But if one builds another package, .e.g. MarlinTrkProcessors, this doesn't necessarily pick up the variant, which might then trigger an unnecessary re-build of dd4hep?

need to add +llvm to work around mesa

Need to add to packages.yaml

mesa:
  variants: +llvm

To avoid following error

==> mesa: Executing phase: 'meson'
==> Error: NameError: name 'SpecError' is not defined
/var/spack/repos/builtin/packages/mesa/package.py:158, in meson_args:
        155
        156        if args_swr_arches:
        157            if '+llvm' not in spec:
  >>    158                raise SpecError('Variant swr requires +llvm')
        159            args_gallium_drivers.append('swr')
        160            args.append('-Dswr-arches=' + ','.join(args_swr_arches))
        161
See build log for details:
  /tmp/user/spack-stage/spack-stage-mesa-20.2.1-4kkkaud5rhdzzo3v7y72mk2ymxhrmuly/spack-build-out.txt

Getting invalid checksum in downloading k4fwcore

==> Installing k4fwcore
==> No binary for k4fwcore found: installing from source
==> Fetching https://github.com/key4hep/K4FWCore/archive/v00-01-01.tar.gz
#=#=#
######################################################################## 100.0%##O#-#
##=O#- #
#-#O=# #
==> Error: ChecksumError: sha256 checksum failed for /tmp/user/spack-stage/spack-stage-k4fwcore-0.1.1-atvoiqohdaxte7ygceue4pix3drrdbb7/v00-01-01.tar.gz
Expected 3a8b8267dc16691dbfef10d8faa8c0ec84ebf2d243069401db07a6706125d3a4 but got 9c4e4b487f7d9c982547c13570345399505e763fb369b76ceadb35c1d52bf6aa
/usr/lib/spack/spack/package.py:1274, in do_fetch:
1271 self.stage.fetch(mirror_only, err_msg=err_msg)
1272 self._fetch_time = time.time() - start_time
1273

1274 if checksum and self.version in self.versions:
1275 self.stage.check()
1276
1277 self.stage.cache_local()

Default variants for MESA pull in LLVM

... which is not needed for the key4hep build, as far as I can tell. Since LLVM is one of the packages that take longest to compile, there should be a packages.yaml that includes

  mesa:
    variants: ~llvm

Set up Nightly Builds

The HEAD of all experiment packages should be installed to /cvmfs/sw-nightlies.hsf.org

Need to add python 3.7 for Gaudi

need to add the following to key4hep-stack for gaudi

# gaudi needs 3.7.99
depends_on('python@:3.7.99')

Right now the dependencies I have for key4hep-stack

# be explicit to avoid concretizer errors
depends_on('root cxxstd=17 +root7 +ssl')
depends_on('boost +python')
depends_on('clhep cxxstd=17')
# gaudi needs 3.7.99
depends_on('python@:3.7.99')

Add regression tests to build

This is item for general discussion, but now that I've got automated builds running, I'd like to start doing automated regression testing, and I'm interested in the best way of specifying tests to be run. This may be an upstream spack issue

environment modules now required for setup

The setup script for the spack build uses the module load command, which is available on lxplus, but not on every centos7 installation, unfortunately also not on the clicdp/cc7-lcg docker image used in the CI.

Add Debug Builds

Should be fairly easy by setting up an environment with

packages:
  all:
    variants: build_type=Debug

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.