kokkos / hpcbind

Binding utilities used for MPI, OpenMP and GPUs

Kokkos: Core Libraries

Kokkos Core implements a programming model in C++ for writing performance portable applications targeting all major HPC platforms. For that purpose it provides abstractions for both parallel execution of code and data management. Kokkos is designed to target complex node architectures with N-level memory hierarchies and multiple types of execution resources. It currently can use CUDA, HIP, SYCL, HPX, OpenMP and C++ threads as backend programming models with several other backends in development.

Kokkos Core is part of the Kokkos C++ Performance Portability Programming Ecosystem.

Kokkos is a Linux Foundation project.

Learning about Kokkos

To start learning about Kokkos:

• Kokkos Lectures: they contain a mix of lecture videos and hands-on exercises covering all the important capabilities.

• Programming guide: contains in "narrative" form a technical description of the programming model, machine model, and the main building blocks like the Views and parallel dispatch.

• API reference: organized by category, i.e., core, algorithms and containers or, if you prefer, in alphabetical order.

• Use cases and Examples: a serie of examples ranging from how to use Kokkos with MPI to Fortran interoperability.

Obtaining Kokkos

The latest release of Kokkos can be obtained from the GitHub releases page.

The current release is 4.3.00.

curl -OJ -L https://github.com/kokkos/kokkos/archive/refs/tags/4.3.00.tar.gz
# Or with wget
wget https://github.com/kokkos/kokkos/archive/refs/tags/4.3.00.tar.gz

To clone the latest development version of Kokkos from GitHub:

git clone -b develop  https://github.com/kokkos/kokkos.git

Building Kokkos

To build Kokkos, you will need to have a C++ compiler that supports C++17 or later. All requirements including minimum and primary tested compiler versions can be found here.

Building and installation instructions are described here.

You can also install Kokkos using Spack: spack install kokkos. Available configuration options can be displayed using spack info kokkos.

For the complete documentation: kokkos.org/kokkos-core-wiki/

Support

For questions find us on Slack: https://kokkosteam.slack.com or open a GitHub issue.

For non-public questions send an email to: crtrott(at)sandia.gov

Contributing

Please see this page for details on how to contribute.

Citing Kokkos

Please see the following page.

License

Under the terms of Contract DE-NA0003525 with NTESS, the U.S. Government retains certain rights in this software.

The full license statement used in all headers is available here or here.

hpcbind's Issues

Need review with testbed and production environment stakeholders

Long setup time due to hwloc. Add store/load cpuset configuration

hpcbind can take several seconds to inspect the architecture and get the final cpuset desired. For a script that launches several simulations with the same cpuset configuration, this can cause a significant overhead (especially if simulations themselves are "fast").

I would suggest adding the capability to store the HPCBIND_* variables to file, and load them on a subsequent run. The idea is that loading the env variables from file should be faster than continuously calling hwloc-ls.

Make OMP_NESTED false by default

Since Kokkos 'usually' does not use this feature, it makes sense to disable it by default, so that a distract user does not leave it on in OpenMP by mistake. I'm not 100% sure, but there may be some performance hit in leaving it on. Either way, it makes more sense to have a feature not commonly used off, unless the user specifically asks for it.

Deprecated --openmp-percent is actually broken

If I specify openmp-percent instead of openmp-ratio, I get the error message

/home/lbertag/bin/hpcbind: line 240: 100: command not found

It looks like the syntax for dividing 100 by openmp-percent value is wrong. But perhaps openmp-percent can be removed altogether?

Nodes without GPUs may have nvidia-smi installed leading to incorrect GPU settings

Some nodes may have nvidia-smi installed without having GPUs or GPU driver installed.
In this case nvidia-smi will be found (sets HPCBIND_VISIBLE_GPUS) and the
command used to find the number of GPUs outputs:

"NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

and this registers as 2 lines and hence sets NUM_GPUS to 2 (because it is 2 lines long).
This may be harmless...

GPUs are not assigned correctly

I have CUDA_VISIBLE_DEVICES=0,1,2,3 in the shell. Running:

mpiexec -np 4  hpcbind --distribute=4 --output-prefix=test --output-mode=all --lstopo -- ./test

shows basically the same GPU setting for each test.hpcbind.[0-3]:

[HPCBIND]
HPCBIND_HAS_HWLOC=1
HPCBIND_HAS_NVIDIA=1
HPCBIND_HWLOC_CPUSET=0x00001001
HPCBIND_HWLOC_DISTRIBUTE=4
HPCBIND_HWLOC_DISTRIBUTE_PARTITION=0
HPCBIND_HWLOC_PARENT_CPUSET=0x00555555
HPCBIND_HWLOC_PROC_BIND=all
HPCBIND_HWLOC_VERSION=2.2.0
HPCBIND_NUM_CORES=1
HPCBIND_NUM_NUMAS=1
HPCBIND_NUM_PUS=2
HPCBIND_NUM_SOCKETS=1
HPCBIND_NVIDIA_ENABLE_GPU_MAPPING=1
HPCBIND_NVIDIA_VISIBLE_GPUS=0,1,2,3
HPCBIND_OPENMP_RATIO=1/1
HPCBIND_OPENMP_VERSION=4.0
HPCBIND_QUEUE_MAPPING=0
HPCBIND_QUEUE_NAME=openmpi
HPCBIND_QUEUE_RANK=0
HPCBIND_QUEUE_SIZE=4
[HWLOC]
[CUDA]
CUDA_HOME=/home/aznb/spack/var/spack/environments/trilinos-cudacc61/.spack-env/view
CUDA_LAUNCH_BLOCKING=1
CUDA_VISIBLE_DEVICES=0,1,2,3
[OPENMP]
OMP_NESTED=false
OMP_NUM_THREADS=2
OMP_PLACES=threads
OMP_PROC_BIND=spread
[GOMP] (gcc, g++, and gfortran)
[KMP] (icc, icpc, and ifort)
[XLSMPOPTS] (xlc, xlc++, and xlf)
[BINDINGS]
Machine (128GB total)
  Package L#0
    NUMANode L#0 (P#0 64GB)
    L3 L#0 (15MB) + L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
      PU L#0 (P#0)
      PU L#1 (P#12)
  Package L#1
    NUMANode L#1 (P#1 64GB)

and my app './test' report the same GPU id for all processes.
However, if I run

mpiexec -np 4  hpcbind --distribute=4 --output-prefix=test --output-mode=all --lstopo -- ./test --kokkos-num-devices=4

then each process reports a unique GPU id