noaa-gsl / sena-c_sw

Computational kernel, c_sw, extracted from FV3 for performance and design analysis.

This repository is a scientific product and is not official communication
of the National Oceanic and Atmospheric Administration, or the United States
Department of Commerce. All NOAA GitHub project code is provided on an ‘as
is’ basis and the user assumes responsibility for its use. Any claims against
the Department of Commerce or Department of Commerce bureaus stemming from
the use of this GitHub project will be governed by all applicable Federal
law. Any reference to specific commercial products, processes, or service
by service mark, trademark, manufacturer, or otherwise, does not constitute
or imply their endorsement, recommendation or favoring by the Department of
Commerce. The Department of Commerce seal and logo, or the seal and logo of
a DOC bureau, shall not be used in any manner to imply endorsement of any
commercial product or activity by DOC or the United States Government.

This project contains derivative work from the Geophysical Fluid Dynamics Laboratory GitHub project for the FV3 dynamical core (https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere). It is a stand alone subset of the original specifically licensed for use under the Apache License. The application of the Apache License to this standalone work does not imply nor shall it be construed as any change to the original source license.

Overview

This repository contains a stand-alone kernel for the c_sw subroutine, extracted from the FV3 model. The goal of this repository is to facilitate direct comparisons of different implementations of c_sw. Alternative implementations may use different programming languages, parallel programming models, or numerical modeling libraries. They may also target different hardware such as GPUs or ARM processors. In numerical modeling, metrics for comparison traditionally include performance and portability. However, this work encourages comparison of developer productivity and design metrics such as ease of use, extensibility, and maintainability.

A Fortran 90 implementation of c_sw is provided as a baseline reference to which other implementations should be compared. Additional implementations will be added as they are developed.

Contents

This repository is organized as follows.

test/

Contains the reference test baselines, input and output data all implementations must use for testing and validation.

ref/

Contains the source tree for the reference implementation. See the README in that directory for details about how to run and test it.

Julia/

Contains the source tree for the Julia implementation. See the README in that directory for details about how to run and test it.

Contributing

Please see the Contributing Guide for information about contributing to this repository.

sena-c_sw's Issues

Clean up ifdef Dead Code

After looking at the cleaned up code in SENA-c_sw where the hydrostatic dead code was eliminated, the ifdefs are another cause of dead code and ought to be removed.

Move baseline output tests to project root/base directory

Issue checklist

Put an x in all the boxes that apply

• I have read the README and ref/README, including the Troubleshooting Guide.
• I have reviewed existing issues to ensure this has not already been reported.

I am submitting a ...

Put an x in the appropriate box

• Suggestion for improving existing code or documentation
• Documentation error

The problem

Move baseline output tests ( ref/test/test_output/ ) into project root, and update ref/test/CMakeLists.txt to point to the new location.

With addition implementations of the kernel, the baseline tests should live outside of the ref/ directory.

Document interpFactor

Issue checklist

• I have read the README and ref/README, including the Troubleshooting Guide.
• I have reviewed existing issues to ensure this has not already been reported.

Describe the bug

There is a new entry into the namelist files, interpFactor, that expands the data base. This is useful in moving to exascale where more powerful nodes and larger memories are expected. Unfortunately, this is not documented in the README.md file. So we need to document this so users of the kernel can understand and use this feature.

OMP_NUM_THREADS

I do not see OMP_NUM_THREADS being set. I do see all the available threads being used when ctest is run. Suggest setting OMP_NUM_THREADS to 4 to standardize the timings.

NetCDF files are written in CDF-1 classic format

The current code writes NetCDF output in CDF-1 classic format. This format limits the size of files to 2GB. While 2GB files are sufficient for current needs, upcoming scaling of input data will require file sizes beyond 2GB. Moving to NetCDF4 format will fix this problem.

Add support for GPTL

Add support for building with GPTL so users can insert custom timers for performance analysis purposes.

On ARM with flang

On the Sandia NL ARM machine "stria", I tried to get flang working with the c_sw kernel. The command:

export CMAKE_PREFIX_PATH=/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5

tells CMAKE where the netcdf directory is located. CMAKE then tries to tell flang where the netcdf directory is with the
"-isystem" flag. But flang ignores it, which causes an error in finding file: netcdf.mod. Is there flang on Hera or Orion?
If so, this would be easier to debug.

module swap gnu7 arm/20.0
source /projects/marvell/setup_env_llvm

rm -rf build ; mkdir build ; cd build
export CC=clang
export FC=flang
export CMAKE_PREFIX_PATH=/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5

[lmstrin@stria-login2 build]$ cmake -DCMAKE_BUILD_TYPE=release ..
-- The Fortran compiler identification is Flang 99.99.1
-- The C compiler identification is Clang 9.0.1
-- Check for working Fortran compiler: /projects/marvell/llvm/9.0.1/bin/flang
-- Check for working Fortran compiler: /projects/marvell/llvm/9.0.1/bin/flang -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /projects/marvell/llvm/9.0.1/bin/flang supports Fortran 90
-- Checking whether /projects/marvell/llvm/9.0.1/bin/flang supports Fortran 90 -- yes
-- Check for working C compiler: /projects/marvell/llvm/9.0.1/bin/clang
-- Check for working C compiler: /projects/marvell/llvm/9.0.1/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found OpenMP_C: -fopenmp=libomp (found version "3.1")
-- Could NOT find OpenMP_Fortran (missing: OpenMP_Fortran_FLAGS OpenMP_Fortran_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_Fortran_FOUND Fortran) (found version "3.1")
NetCDF_C_INCLUDE_FILE: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include/netcdf.h
NetCDF_Fortran_INCLUDE_FILE: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include/netcdf.mod
NetCDF_C_CONFIG_EXECUTABLE: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/bin/nc-config
NetCDF_Fortran_CONFIG_EXECUTABLE: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/bin/nf-config
NetCDF_C_LIBRARY: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/lib/libnetcdf.so
NetCDF_Fortran_LIBRARY: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/lib/libnetcdff.so
-- Found NetCDF: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include;/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include (found version "4.6.3") found components: C Fortran
-- FindNetCDF defines targets:
-- - NetCDF_VERSION [4.6.3]
-- - NetCDF_PARALLEL [TRUE]
-- - NetCDF_C_CONFIG_EXECUTABLE [/opt/atse/libs/arm/openmpi4/netcdf/4.6.3/bin/nc-config]
-- - NetCDF::NetCDF_C [SHARED] [Root: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3] Lib: /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/lib/libnetcdf.so
-- - NetCDF_Fortran_CONFIG_EXECUTABLE [/opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/bin/nf-config]
-- - NetCDF::NetCDF_Fortran [SHARED] [Root: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5] Lib: /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/lib/libnetcdff.so
-- Configuring done
-- Generating done
-- Build files have been written to: /ascldap/users/lmstrin/SENA-c_sw/ref/build
[lmstrin@stria-login2 build]$ make VERBOSE=1
/opt/atse/utils/cmake/3.12.2/bin/cmake -H/ascldap/users/lmstrin/SENA-c_sw/ref -B/ascldap/users/lmstrin/SENA-c_sw/ref/build --check-build-system CMakeFiles/Makefile.cmake 0
/opt/atse/utils/cmake/3.12.2/bin/cmake -E cmake_progress_start /ascldap/users/lmstrin/SENA-c_sw/ref/build/CMakeFiles /ascldap/users/lmstrin/SENA-c_sw/ref/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory /home/lmstrin/SENA-c_sw/ref/build' make -f src/CMakeFiles/c_sw.dir/build.make src/CMakeFiles/c_sw.dir/depend make[2]: Entering directory /home/lmstrin/SENA-c_sw/ref/build'
cd /ascldap/users/lmstrin/SENA-c_sw/ref/build && /opt/atse/utils/cmake/3.12.2/bin/cmake -E cmake_depends "Unix Makefiles" /ascldap/users/lmstrin/SENA-c_sw/ref /ascldap/users/lmstrin/SENA-c_sw/ref/src /ascldap/users/lmstrin/SENA-c_sw/ref/build /ascldap/users/lmstrin/SENA-c_sw/ref/build/src /ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/DependInfo.cmake --color=
Dependee "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/DependInfo.cmake" is newer than depender "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/depend.internal".
Dependee "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/ascldap/users/lmstrin/SENA-c_sw/ref/build/src/CMakeFiles/c_sw.dir/depend.internal".
Scanning dependencies of target c_sw
make[2]: Leaving directory /home/lmstrin/SENA-c_sw/ref/build' make -f src/CMakeFiles/c_sw.dir/build.make src/CMakeFiles/c_sw.dir/build make[2]: Entering directory /home/lmstrin/SENA-c_sw/ref/build'
[ 16%] Building Fortran object src/CMakeFiles/c_sw.dir/netCDFModule.f90.o
cd /ascldap/users/lmstrin/SENA-c_sw/ref/build/src && /projects/marvell/llvm/9.0.1/bin/flang -DNDEBUG -isystem /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include -isystem /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include -O3 -DNDEBUG -c /ascldap/users/lmstrin/SENA-c_sw/ref/src/netCDFModule.f90 -o CMakeFiles/c_sw.dir/netCDFModule.f90.o
clang-9: warning: argument unused during compilation: '-isystem /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include' [-Wunused-command-line-argument]
clang-9: warning: argument unused during compilation: '-isystem /opt/atse/libs/arm/openmpi4/netcdf-fortran/4.4.5/include' [-Wunused-command-line-argument]
F90-F-0004-Unable to open MODULE file netcdf.mod (/ascldap/users/lmstrin/SENA-c_sw/ref/src/netCDFModule.f90: 7)
F90/aarch64 Linux Flang - 1.5 2017-05-01: compilation aborted
make[2]: *** [src/CMakeFiles/c_sw.dir/netCDFModule.f90.o] Error 1
make[2]: Leaving directory /home/lmstrin/SENA-c_sw/ref/build' make[1]: *** [src/CMakeFiles/c_sw.dir/all] Error 2 make[1]: Leaving directory /home/lmstrin/SENA-c_sw/ref/build'
make: *** [all] Error 2
[lmstrin@stria-login2 build]$ ls /opt/atse/libs/arm/openmpi4/netcdf/4.6.3/include
netcdf_aux.h netcdf_filter.h netcdf.h netcdf_mem.h netcdf_meta.h netcdf_par.h

Add Support for MPI Parallel Execution

Issue checklist

• I have read the README and ref/README, including the Troubleshooting Guide.
• I have reviewed existing issues to ensure this has not already been reported.

Is your feature request related to a problem? Please describe.

The existing code only has support for parallelism via OpenMP threading. This limits the utility of the kernel for use in evaluating performance tradeoffs of various types of implementations. A proper evaluation must include analysis of multi-node as well as single-node parallel performance.

Describe the solution you'd like

A full featured MPI capability is needed. This will include:

• Ability to run with any number of MPI tasks between 1 and N
• Ability to run with MPI+OpenMP threads such that each MPI rank spawns a configurable number of OpenMP threads
• A customizable processor grid for distribution of MPI ranks (e.g. 4x4, 2x8, 1x16)
• A default processor grid that maximizes the "square-ness" of the grid for to minimize communication
• Automatic domain decomposition of data onto the chosen processor grid
• All options configurable at runtime in the input namelist

Describe alternatives you've considered

MPI is a ubiquitous and standard means of parallelizing across nodes of a supercomputer. While there may be other ways of achieving that, having MPI parallelism is necessary for establishing a baseline of performance against which other implementations should be compared.

Invalid scientific notation output by `print_state()`

Issue checklist

• I have read the README and ref/README, including the Troubleshooting Guide.
• I have reviewed existing issues to ensure this has not already been reported.

Describe the bug

The print_state() function is outputting non-standard scientific notation, and incompatible with standards used by other languages (C, C++, Julia) and other printf based I/O.

Recipe to reproduce

Steps to reproduce the behavior:

1. Go to the print_state() function in sw_core.F90
2. Observe print_state()'s calls to print_3d_variable() and print_2d_variable() functions. These functions' write statement should be altered to comply with the 'conventional' way that printf represents floating point numbers in scientific notation.

Expected behavior, Screenshots, and terminal output

The current output in scientific notation:

-0.10000000000E-07

printf's equivalent of how this value is expressed in scientific notation:

-1.00000000000E-08

This is what the modified Fortran code should output.

See current test outputs for full context.

Additional context

Fortran is the outlier in how scientific notation is expressed. The current output will make it very difficult for others to use baselines for testing.

Not only will the write statements in the print_3d_variable & print_2d_variable functions need to be changed, all of the files within the ref/test/test_output will need alterations to reflect the new scientific notation necessary to comply with future testing of the kernel.