dsloanm-epcc / moap

MOAP

Spack Setup

On ARCHER2, a pre-prepared Spack install is available for use at /work/d435/d435/shared/spack alongside a working copy of this repository.

If the pre-prepared install cannot be used, to install from scratch:

# Check out the latest Spack
git clone https://github.com/spack/spack.git
# Replace recipes with those from this repository's `spack_recipes`
cd spack/var/spack/repos/builtin/packages
for pkg in blitz lfric py-psyclone rose-picker xios yaxt; do
  # Delete any old recipes
  rm -r ${pkg}
  # Symlink new recipes
  ln -s /work/d435/d435/shared/spack_recipes/${pkg} ${pkg}
done

Passwordless SVN access to:

https://code.metoffice.gov.uk/svn/lfric/GPL-utilities/trunk
https://code.metoffice.gov.uk/svn/lfric/LFRic/trunk

must be set up to enable installation of the rose-picker dependency and LFRic itself. Failure to do so will result in permissions issues during the install.

GCC 10+11 Build

Create a Spack environment with the env.yaml configuration file:

source /work/d435/d435/shared/spack/share/spack/setup-env.sh
spack env create moap-gcc10 /work/d435/d435/shared/spack_config/env.yaml

The config file sets various ARCHER2-specific options, such as the use of cray-mpich rather than a custom MPI.

Activate the new environment and install the lfric spec:

spack env activate moap-gcc10
# User's choice of %[email protected] or %[email protected]
spack add lfric %[email protected]
# View and confirm the spec then install
spack spec
spack install

Note, spack compiler find may be needed to set up gcc.

CCE 13 Build

Follow GCC instructions above but specify %[email protected] compiler instead:

spack add lfric %[email protected]

See below for build issue workarounds.

Workaround for OpenSSL Build Stall

Cray compilers <= [email protected] indefinitely stall when building OpenSSL due to bugs with Cray-specific optimisations.

Work around by running spack install until Spack hangs on OpenSSL. Then interrupt build (ctrl+c) and install only OpenSSL with -fno-cray flag:

CFLAGS="-fno-cray" CPPFLAGS="-fno-cray" spack install --dirty openssl %[email protected]

Argument --dirty necessary for Spack to pick up on FLAGS environment variables.

Resume spack install afterwards.

Workaround for LFRic Failing to Find dgemm, etc.

When build fails with missing references to _dgemm_ and other BLAS/LAPACK routines, restart with:

spack install --dirty

It is believed Spack is failing to pick up on cray-libsci without the --dirty flag.

Intel 2021.1.1 Build

Unload all central MPI, compiler and BLAS modules and setup the Intel modules:

source scripts/setup-intel.sh

or manually:

module use /work/z19/shared/adrianj/intel/oneapi/modulefiles 
module load compiler
module load mpi
module load mkl
module unload cray-mpich PrgEnv-cray cce cray-libsci

Create a new environment using intel-env.yaml:

source /work/d435/d435/shared/spack/share/spack/setup-env.sh
spack env create moap-intel /work/d435/d435/shared/spack_config/intel-env.yaml

Activate the new environment and install the lfric spec:

spack env activate moap-intel
spack add lfric %[email protected]
# View and confirm the spec then install
spack spec
spack install

Manual LFRic Build

To rebuild a working copy of LFRic, e.g. to test local changes, Spack can be used to set up all dependent packages then a manual build of LFRic performed.

First, complete an LFRic install in a Spack environment as detailed above. Switch compiler to match the version used for the environment install, set relevant build environment variables and perform a manual make.

GCC Setup

spack env activate moap-gcc11
module swap PrgEnv-cray PrgEnv-gnu
# Using the moap-gcc11 environment so use gcc/11 compiler
module swap gcc gcc/11.2.0

# Set compiler names
export FC=ftn
export LDMPI=ftn

CCE Setup

spack env activate moap-cce13
# Using the moap-gcc13 environment so use cce/13 compiler
module swap cce cce/13.0.2

# Set compiler names
export FC=ftn
export LDMPI=ftn

Intel Setup

spack env activate moap-intel
source scripts/setup-intel.sh

# Set compiler names 
export FC=ifort
export LDMPI=mpiifort

Perform the Build

Set up remaining compiler-agnostic variables and build:

export FPP="cpp -traditional-cpp"
# HACK: Get the prefix directory for the Spack environment, e.g.
#   /mnt/lustre/a2fs-work3/work/d435/d435/shared/spack/var/spack/environments/moap-gcc11/.spack-env/view
# Get the path to `psyclone`, use `rev` to reverse the string, `cut` to remove
# `/bin/psyclone` from the "beginning" and `rev` again to restore the original
# order.
INSTALL_DIR=$(which psyclone | rev | cut -d/ -f3- | rev)
export FFLAGS="-I$INSTALL_DIR/include"
export LDFLAGS="-L$INSTALL_DIR/lib"
export CPPFLAGS="-I$INSTALL_DIR/include"

# Set target to enable OpenMP use
export LFRIC_TARGET_PLATFORM=meto-xc40

# Build
cd trunk/gungho
make -j64 build

New binary will be produced in bin/gungho. CAUTION: the old gungho binary provided by the chosen environment will still be on $PATH. Carefully specify your paths to ensure only the intended binary is run.

Running the New Build

# Ensure libraries provided by environment are linked correctly
export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
cd example
../bin/gungho configuration.nml

moap's Issues

LFRic CCE Build Fails With Missing References to _dgemm_, etc.

Believe Spack is failing to pick up on cray-libsci with a clean build. Workaround is wait for LFRic build to fail then run:

spack install --dirty

More investigation needed.

Note this is only seen with [email protected] and not [email protected] or [email protected].

Intel Build Failures - error #8889: Explicit declaration of the EXTERNAL attribute is required

Intel builds require ifort.mk.patch applied to the LFRic source code to remove -warn errors from FFLAGS and avoid error #8889: Explicit declaration of the EXTERNAL attribute is required warnings being promoted to errors. A cleaner solution needs found.

Build with ARCHER2 Intel Compiler

Intel compiler is available on ARCHER2 in z19 shared directory:

$ module use /work/z19/shared/adrianj/intel/oneapi/modulefiles
$ module load mpi/2021.1.1
$ module load compiler/2021.1.1
$ mpiicc --version
icc (ICC) 2021.1 Beta 20201112
Copyright (C) 1985-2020 Intel Corporation.  All rights reserved.

Further modules can be found with module list.

Needs to be integrated with Spack - both the compiler and Intel MPI replacing cray-mpich.

Needs to be integrated with Slurm - can Intel MPI mpirun be used in place of srun?

Hang with CCE when Building OpenSSL

Builds of OpenSSL (a build dependency) hang with <= [email protected] due to bugs with Cray-specific optimisations.

Work around by running spack install until Spack hangs on OpenSSL. Then interrupt build (ctrl+c) and install only OpenSSL with -fno-cray flag:

CFLAGS="-fno-cray" CPPFLAGS="-fno-cray" FFLAGS="-fno-cray" spack install --dirty openssl %[email protected]

Argument --dirty necessary for Spack to pick up on FLAGS environment variables.

Resume spack install afterwards.

This bug should be fixed in the next Cray software release but it's unknown when this will be made available on ARCHER2.

Race Condition on IAXIS.mod Building XIOS with CCE

Building XIOS with --job 16 and [email protected] gives error:

ftn-3178 crayftn: LIMIT in command line
    The compiler cannot open file "IAXIS.mod".  This file was either created or previously accessed in th
    is compilation, so should be available.

Currently worked around by forcing serial make in XIOS recipe but proper solution needs found·