kokkos / kokkos-fortran-interop

The Kokkos Fortran Interop repository contains tools and interfaces which help interactions between Fortran portions of an applications and C++ portions using Kokkos.

Kokkos: Core Libraries

Kokkos Core implements a programming model in C++ for writing performance portable applications targeting all major HPC platforms. For that purpose it provides abstractions for both parallel execution of code and data management. Kokkos is designed to target complex node architectures with N-level memory hierarchies and multiple types of execution resources. It currently can use CUDA, HIP, SYCL, HPX, OpenMP and C++ threads as backend programming models with several other backends in development.

Kokkos Core is part of the Kokkos C++ Performance Portability Programming Ecosystem.

Kokkos is a Linux Foundation project.

Learning about Kokkos

To start learning about Kokkos:

• Kokkos Lectures: they contain a mix of lecture videos and hands-on exercises covering all the important capabilities.

• Programming guide: contains in "narrative" form a technical description of the programming model, machine model, and the main building blocks like the Views and parallel dispatch.

• API reference: organized by category, i.e., core, algorithms and containers or, if you prefer, in alphabetical order.

• Use cases and Examples: a serie of examples ranging from how to use Kokkos with MPI to Fortran interoperability.

Obtaining Kokkos

The latest release of Kokkos can be obtained from the GitHub releases page.

The current release is 4.3.00.

curl -OJ -L https://github.com/kokkos/kokkos/archive/refs/tags/4.3.00.tar.gz
# Or with wget
wget https://github.com/kokkos/kokkos/archive/refs/tags/4.3.00.tar.gz

To clone the latest development version of Kokkos from GitHub:

git clone -b develop  https://github.com/kokkos/kokkos.git

Building Kokkos

To build Kokkos, you will need to have a C++ compiler that supports C++17 or later. All requirements including minimum and primary tested compiler versions can be found here.

Building and installation instructions are described here.

You can also install Kokkos using Spack: spack install kokkos. Available configuration options can be displayed using spack info kokkos.

For the complete documentation: kokkos.org/kokkos-core-wiki/

Support

For questions find us on Slack: https://kokkosteam.slack.com or open a GitHub issue.

For non-public questions send an email to: crtrott(at)sandia.gov

Contributing

Please see this page for details on how to contribute.

Citing Kokkos

Please see the following page.

License

Under the terms of Contract DE-NA0003525 with NTESS, the U.S. Government retains certain rights in this software.

The full license statement used in all headers is available here or here.

kokkos-fortran-interop's Issues

gnu related changes to tests

Issue pointed out by team member related to bounds checking / relative tolerance in gnu:

Can't run the examples when using Kokkos::Cuda backend

I may be missing something obvious but all examples end up in a memory related error.

I built and installed Kokkos with cuda enabled (and cross-check that afterwards in flcl examples by adding "call kokkos_print_configuration").

Here is the the result of kokkos_print_configuration:

Kokkos Version: 3.6.1
Compiler:
KOKKOS_COMPILER_GNU: 1120
KOKKOS_COMPILER_NVCC: 1
Architecture:
Default Device: N6Kokkos4CudaE
KOKKOS_ENABLE_ISA_KNC: no
KOKKOS_ENABLE_ISA_POWERPCLE: no
KOKKOS_ENABLE_ISA_X86_64: no
Atomics:
KOKKOS_ENABLE_GNU_ATOMICS: no
KOKKOS_ENABLE_INTEL_ATOMICS: no
KOKKOS_ENABLE_WINDOWS_ATOMICS: no
Vectorization:
KOKKOS_ENABLE_PRAGMA_IVDEP: no
KOKKOS_ENABLE_PRAGMA_LOOPCOUNT: no
KOKKOS_ENABLE_PRAGMA_SIMD: no
KOKKOS_ENABLE_PRAGMA_UNROLL: no
KOKKOS_ENABLE_PRAGMA_VECTOR: no
Memory:
KOKKOS_ENABLE_HBWSPACE: no
KOKKOS_ENABLE_INTEL_MM_ALLOC: no
KOKKOS_ENABLE_POSIX_MEMALIGN: no
Options:
KOKKOS_ENABLE_ASM: yes
KOKKOS_ENABLE_CXX14: yes
KOKKOS_ENABLE_CXX17: no
KOKKOS_ENABLE_CXX20: no
KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK: no
KOKKOS_ENABLE_HWLOC: yes
KOKKOS_ENABLE_LIBRT: no
Host Serial Execution Space:
  KOKKOS_ENABLE_SERIAL: yes
Serial Atomics:
  KOKKOS_ENABLE_SERIAL_ATOMICS: no

Serial Runtime Configuration:
Device Execution Space:
  KOKKOS_ENABLE_CUDA: yes
Cuda Atomics:
  KOKKOS_ENABLE_CUDA_ATOMICS: no
Cuda Options:
  KOKKOS_ENABLE_CUDA_LAMBDA: yes
  KOKKOS_ENABLE_CUDA_LDG_INTRINSIC: yes
  KOKKOS_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE: no
  KOKKOS_ENABLE_CUDA_UVM: yes
  KOKKOS_ENABLE_CXX11_DISPATCH_LAMBDA: yes

Cuda Runtime Configuration:
macro  KOKKOS_ENABLE_CUDA      : defined
macro  CUDA_VERSION          = 11070 = version 11.7
Kokkos::Cuda[ 0 ] NVIDIA RTX A3000 Laptop GPU capability 8.6, Total Global Memory: 5.807 G, Shared Memory per Block: 48 K : Selected

Example 01 fails like this:

 allocating memory
 initializing kokkos
 setting up arrays
 PASSED f_y and c_y the same before axpys
 performing an axpy in fortran
 performing an axpy with kokkos
c_axpy_ndarray
y.data() = 0x560914a16590
x.data() = 0x560914a201e0
Before parallel_for
After  parallel_for
 FAILED f_y and c_y the same before axpys
 norm2(f_y-c_y)   11.004194752360011     
 (1.0e-14)*norm2(f_y)   4.9579939933916309E-013
 finalizing kokkos
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaMemcpyAsync(dst, src, n, cudaMemcpyDefault, instance.cuda_stream()) error( cudaErrorIllegalAddress): an illegal memory access was encountered /home/kestenerp/install/kokkos/github/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp:99

Program received signal SIGABRT: Process abort signal.

I also tried to add a fence right after the parallel_for, but there is still and error:

 allocating memory
 initializing kokkos
 setting up arrays
 PASSED f_y and c_y the same before axpys
 performing an axpy in fortran
 performing an axpy with kokkos
c_axpy_ndarray
y.data() = 0x5589320b4590
x.data() = 0x5589320be1e0
Before parallel_for
After  parallel_for
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered /home/kestenerp/install/kokkos/github/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:151

The memory addresses of views x and y seem ok, I can acces data from host code, but not from inside parallel_for lambda.

I tried cuda-gdb, here is the state of view x right before the parallel_for dispatch:

(cuda-gdb) p x
$1 = {<Kokkos::ViewTraits<double*, Kokkos::LayoutStride, Kokkos::CudaUVMSpace>> = {<No data fields>}, m_track = {m_tracker = {m_record_bits = 1, m_record = 0x1}}, 
  m_map = {m_impl_handle = 0x5555557551e0, m_impl_offset = {m_dim = {<Kokkos::Impl::ViewDimension0<0, 1>> = {static ArgN0 = 0, 
          N0 = 5000}, <Kokkos::Impl::ViewDimension1<18446744073709551615, 1>> = {static ArgN1 = 1, 
          static N1 = 1}, <Kokkos::Impl::ViewDimension2<18446744073709551615, 1>> = {static ArgN2 = 1, 
          static N2 = 1}, <Kokkos::Impl::ViewDimension3<18446744073709551615, 1>> = {static ArgN3 = 1, 
          static N3 = 1}, <Kokkos::Impl::ViewDimension4<18446744073709551615, 1>> = {static ArgN4 = 1, 
          static N4 = 1}, <Kokkos::Impl::ViewDimension5<18446744073709551615, 1>> = {static ArgN5 = 1, 
          static N5 = 1}, <Kokkos::Impl::ViewDimension6<18446744073709551615, 1>> = {static ArgN6 = 1, 
          static N6 = 1}, <Kokkos::Impl::ViewDimension7<18446744073709551615, 1>> = {static ArgN7 = 1, static N7 = 1}, <No data fields>}, m_stride = {S0 = 1}}}, 
  static is_layout_left = false, static is_layout_right = false, static is_layout_stride = true, static is_default_map = true}

I can see anything wrong but I feel I may be missing something obvious.
Can anyone help ?

char interoperability issues in xlf

More details later.

Possible use of SWIG-Fortran

Hi, I'm the PI of the ForTrilinos project in ECP, currently bundled with ALExa. I think you and I chatted briefly at an ECP meeting two years ago about automating Fortran bindings. The SWIG-Fortran tool has matured enough to a point where I think we could help out. Would you be interested in collaborating to help automate your kokkos wrappers?

update CI for AMD related compilers/targets

(and fix any found bugs/wrong assumptions)

Kokkos OpenMP + XL 16 broken

With a Fortran main, and OpenMP initialization in C++ (through Kokkos) errors of the following type occur (failing the FLCL unit tests):
80: terminate called after throwing an instance of 'std::runtime_error'
80: what(): Kokkos::Impl::HostThreadTeamData::organize_pool ERROR pool already exists
80: Traceback functionality not available

add to_nd_array 5d - 7d versions

new release 0.999.0

Add in "missing" from_nd_array and new CI.

add global typedefs for consistency

Something like a stereotypical define_kind module. Potentially user override-able.

deal with zero length array dimensions in ndarray in a way that doesn't make gcc mad

Something like

If (size of array dimension is 0)
  Set the stride directly
Else
  Do the normal stuff
End

@cferenba

Please add an example

I have some users who might be interested in this, and it would be great if there were an example in the README or in a simple source file to give potential users some idea of how this might help them.

cmake build support

Would like to add support to build with cmake for easier integration with other projects.

add debug builds to CI

Sometimes GNU can be overly pedantic with bounds checking on things which may not need to be bounds checked such as c_loc(). Building with debug builds of GNU, Intel, etc. will help to catch problems in implementation wrt this.

add spack support

add basic spack support using the existing Makefile

02-axpy-dualview: Kokkos_ENABLE_CUDA=ON

I'd like to start by thanking you all for the awesome job you've done with Kokkos.

I tried to build (cmake 3.16) the "02-axpy-dualview" example with -DKokkos_ENABLE_CUDA=ON (-DKokkos_ENABLE_CUDA_LAMBDA=ON and all the other options needed for in-tree-building; at the end: KOKKOS_DEVICES=Cuda make -j ) and it worked, but after running:

This error comes out from line 77 of the Fortran main : " call random_number(f_y)".

So I substituted all the "call random_number()" by assigning some values, and it worked:

Then I tried to export data from the Fortran main, but just using OPEN(..) and CLOSE(), without writing, it returned the same error signal SIGSEGV.
If I try to use OPEN() and CLOSE() before "kokkos_initialize()" the program runs, but simply doesn't make nothing (I need to press Ctrl+C).

add 4d-7d support for kokkos_allocate_view (including unit tests)

Existing: 1d-3d
"Missing": 4d-7d

cmake version versus git version

Currently cmake project version is 0.5.0 but the latest git release is 0.99.0

Can one the maintainers make a new release where both version number agree ?

new feature: gpu allocate and deallocate

Is there any interest in adding gpu independent allocate and deallocate wrappers (for kokkos_malloc) to FLCL? What about things like Memcpy and Memset?

I am working on a port of the GENE fusion code (http://genecode.org/) to run on AMD and Intel GPUs (it is already working on CUDA). It currently uses a mix of Fortran (original CPU only code) and CUDA C/C++, together with the gtensor (https://github.com/wdmapp/gtensor) multi-d C++ library, with lazy array evaluation and automatic generation of kernels.

Currently the memory management lives in Fortran, via a family of "reg_storage" types that call regular allocate for CPU or Fortran wrapped cudaMalloc underneath. gtensor has an experimental gpuMalloc wrapper that calls hipMalloc or sycl::malloc under the hood, but we are interested in leveraging existing solutions and collaborating as much as possible. We are exploring how gtensor's lazy evaluation model could be implemented on top of RAJA or Kokkos, and /or inter-operate with using mdspan-compatible view types. And because we are working on a Fortran application, ease of interoperability is also a major factor.

add matching deallocations to unit tests

add unit tests for kokkos_allocate_view

reason out support for COMPLEX and Kokkos::Complex

update CI for Kokkos 3.4+

add 4d-7d support for kokkos_allocate_dualview (including unit tests)