Spack spec very slow about spack

[agseaton]

I think I figured out what is making this slow. I had added the develop binary mirror (https://binaries.spack.io/develop). Removing it gives me more reasonable timings (4 seconds now for spack spec gcc), though at the cost of having to compile many more packages from source.

Any ideas on why this causes such a large slowdown?

[haampie]

It's a 200MB or more database of stuff built in our CI ;p all those entries may be considered for an "optimal" solution.

Did you find that binary cache somewhere in the docs? Maybe we could stop promoting it, cause I think there are also smaller (weekly / monthly ?) snapshots.

[alalazo]

Cannot reproduce slowdowns on:

• Spack: 0.22.0.dev0 (5676164)
• Python: 3.11.5
• Platform: linux-ubuntu20.04-icelake
• Concretizer: clingo

Can you post:

$ spack config get packages
$ spack config get concretizer

?

[agseaton]

Sure:

spack config get packages
packages:
  all:
    compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
    providers:
      awk: [gawk]
      blas: [openblas, amdblis]
      D: [ldc]
      daal: [intel-oneapi-daal]
      elf: [elfutils]
      fftw-api: [fftw, amdfftw]
      flame: [libflame, amdlibflame]
      fuse: [libfuse]
      gl: [glx, osmesa]
      glu: [mesa-glu, openglu]
      golang: [go, gcc]
      go-or-gccgo-bootstrap: [go-bootstrap, gcc]
      iconv: [libiconv]
      ipp: [intel-oneapi-ipp]
      java: [openjdk, jdk, ibm-java]
      jpeg: [libjpeg-turbo, libjpeg]
      lapack: [openblas, amdlibflame]
      libglx: [mesa+glx, mesa18+glx]
      libllvm: [llvm]
      libosmesa: [mesa+osmesa, mesa18+osmesa]
      lua-lang: [lua, lua-luajit-openresty, lua-luajit]
      luajit: [lua-luajit-openresty, lua-luajit]
      mariadb-client: [mariadb-c-client, mariadb]
      mkl: [intel-oneapi-mkl]
      mpe: [mpe2]
      mpi: [openmpi, mpich]
      mysql-client: [mysql, mariadb-c-client]
      opencl: [pocl]
      onedal: [intel-oneapi-dal]
      pbs: [openpbs, torque]
      pil: [py-pillow]
      pkgconfig: [pkgconf, pkg-config]
      qmake: [qt-base, qt]
      rpc: [libtirpc]
      scalapack: [netlib-scalapack, amdscalapack]
      sycl: [hipsycl]
      szip: [libaec, libszip]
      tbb: [intel-tbb]
      unwind: [libunwind]
      uuid: [util-linux-uuid, libuuid]
      xxd: [xxd-standalone, vim]
      yacc: [bison, byacc]
      ziglang: [zig]
      zlib-api: [zlib-ng+compat, zlib]
    permissions:
      read: world
      write: user

and

spack config get concretizer
concretizer:
  reuse: dependencies
  targets:
    # Determine whether we want to target specific or generic
    # microarchitectures. Valid values are: "microarchitectures" or "generic".
    # An example of "microarchitectures" would be "skylake" or "bulldozer",
    # while an example of "generic" would be "aarch64" or "x86_64_v4".
    granularity: microarchitectures
    # If "false" allow targets that are incompatible with the current host (for
    # instance concretize with target "icelake" while running on "haswell").
    # If "true" only allow targets that are compatible with the host.
    host_compatible: true
  # When "true" concretize root specs of environments together, so that each unique
  # package in an environment corresponds to one concrete spec. This ensures
  # environments can always be activated. When "false" perform concretization separately
  # on each root spec, allowing different versions and variants of the same package in
  # an environment.
  unify: true
  duplicates:
    # "none": allows a single node for any package in the DAG.
    # "minimal": allows the duplication of 'build-tools' nodes only (e.g. py-setuptools, cmake etc.)
    # "full" (experimental): allows separation of the entire build-tool stack (e.g. the entire "cmake" subDAG)
    strategy: minimal

[alalazo]

@agseaton you can check weekly snapshots here: https://cache.spack.io/tag/develop-2023-12-17/

I think using these snapshots, and updating them from time to time is usually a better solution than depending on the develop mirror - which is mainly of use for our CI.