gridtools / serialbox

Data serialization library and tools for C/C++ and Fortran

The GridTools framework is a set of libraries and utilities to develop performance portable applications in the area of weather and climate. To achieve the goal of performance portability, the user-code is written in a generic form which is then optimized for a given architecture at compile-time. The core of GridTools is the stencil composition module which implements a DSL embedded in C++ for stencils and stencil-like patterns. Further, GridTools provides modules for halo exchanges, boundary conditions, data management and bindings to C and Fortran.

GridTools is successfully used to accelerate the dynamical core of the COSMO model with improved performance on CUDA-GPUs compared to the current official version, demonstrating production quality and feature-completeness of the library for models on lat-lon grids. The GridTools-based dynamical core is shipped with COSMO v5.7 and later, see release notes COSMO v5.7.

Although GridTools was developed for weather and climate applications it might be applicable for other domains with a focus on stencil-like computations.

A detailed introduction can be found in the documentation.

Installation instructions

git clone https://github.com/GridTools/gridtools.git
cd gridtools
mkdir -p build && cd build
cmake ..
make -j8
make test

For choosing the compiler, use the standard CMake techniques, e.g. setting the environment variables

CXX=`which g++` # full path to the C++ compiler
CC=`which gcc` # full path to theC compiler
FC=`which gfortran` # full path to theFortran compiler
CUDACXX=`which nvcc` # full path to NVCC
CUDAHOSTCXX=`which g++` # full path to the C++ compiler to be used as CUDA host compiler

Requirements

• C++17 compiler (see also list of tested compilers)
• Boost headers (1.73 or later)
• CMake (3.18.1 or later)
• CUDA Toolkit (11.0 or later, optional)
• MPI (optional, CUDA-aware MPI for the GPU communication module gcl_gpu)

Supported compilers

The GridTools libraries are currently nightly tested with the following compilers on CSCS supercomputers.

Compiler	Backend	Tested on	Comments
Cray clang version 12.0.3	all backends	Piz Daint	P100 GPU
Cray clang version 10.0.2 + NVCC 11.2	all backends	Piz Daint	P100 GPU
Cray clang version 12.0.3	all backends	Piz Daint	with -std=c++20
GNU 11.2.0 + NVCC 11.0	all backends	Piz Daint	P100 GPU
GNU 11.2.0 + NVCC 11.2	all backends	Dom	P100 GPU
GNU 8.3.0 + NVCC 11.2	all backends	Tsa	V100 GPU

Known issues

• Some tests are failing with ROCm3.8.0 (Clang 11).
• CUDA 11.0.x has a severe issue, see #1522. Under certain conditions, GridTools code will not compile for this version of CUDA. CUDA 11.1.x and later should not be affected by this issue.
• Cray Clang version 11.0.0 has a problem with the gridtools::tuple conversion constructor, see #1615.

Partly supported (expected to work, but not tested regularly)

Compiler	Backend	Date	Comments
Intel 19.1.1.217	all backends	2021-09-30	with cmake . -DCMAKE_CXX_FLAGS=-qnextgen
NVHPC 23.3	all backends	2023-04-20	only compilation is tested regularly in CI

Contributing

Contributions to the GridTools framework are welcome. Please open an issue for any bugs that you encounter or provide a fix or enhancement as a PR. External contributions to GridTools require us a signed copy of a copyright release form to ETH Zurich. We will contact you on the PR.

serialbox's Issues

[compare.py] Exclude certain meta-data from comparison

It would be nice if the comparison tool has an option to configure which meta-data is included or excluded for comparison.

Background: The new FortrantTestGenerator (FTG) frontend always stores the memory address of a variable in a meta-data field called loc. This is done, because sometimes when working with FTG it is necessary to identify pointers that refer to the same variable. When comparing the results of multiple runs of the same program, those value are obviously different, but shall not count as deviation. My current workaround is to remove this information before using compare.py.

There would be several solutions:

• In general, only compare the standard meta-data (starting with __)
• Add option to include (additional) meta-data fields
• Add option to exclude meta-data fields
• Either by naming concrete fields or by regular expression
• ...

Generated NetCDF nc file is read as zeroes in the adjusted `examples/fortran/perturbation` example.

After modifying the source to support Python preprocessor (#14) I ran examples again.

I used the NetCDF archive format in the adjusted serialbox2/examples/fortran/perturbation/m_ser.f90 with modification to all 3 !$ser init directives:

!------------------------------------------------------------*- Fortran -*-----
!
!                              S E R I A L B O X
!
! This file is distributed under terms of BSD license. 
! See LICENSE.txt for more information.
!
!------------------------------------------------------------------------------

MODULE m_ser

  IMPLICIT NONE

  CONTAINS

  SUBROUTINE serialize(a)
    IMPLICIT NONE
    REAL(KIND=8), DIMENSION(:,:,:) :: a

    !$ser init directory='.' prefix='SerialboxTest' archive='NetCDF'
    !$ser savepoint sp1
    !$ser mode write
    !$ser data ser_a=a

  END SUBROUTINE serialize

  SUBROUTINE deserialize(a)
    IMPLICIT NONE
    REAL(KIND=8), DIMENSION(:,:,:) :: a

    !$ser init directory='.' prefix='SerialboxTest-output' prefix_ref='SerialboxTest' archive='NetCDF'
    !$ser savepoint sp1
    !$ser mode read
    !$ser data ser_a=a
    !$ser mode write
    !$ser data ser_a=a

  END SUBROUTINE deserialize

  SUBROUTINE deserialize_with_perturb(a)
    IMPLICIT NONE
    REAL(KIND=8), DIMENSION(:,:,:) :: a
    REAL(KIND=8) :: rprecision
    rprecision = 10.0**(-PRECISION(1.0))

    !$ser init directory='.' prefix='SerialboxTest-output' prefix_ref='SerialboxTest' rprecision=rprecision rperturb=1.0e-5_8 archive='NetCDF'
    !$ser savepoint sp1
    !$ser mode read-perturb
    !$ser data ser_a=a

  END SUBROUTINE deserialize_with_perturb

END MODULE m_ser

Everything compiles. Producer generates 7037 bytes long SerialboxTest_ser_a.nc (I can't attach its contents to the github issue.)

When I call producer everything seems fine:

$ ./producer 
 CALL serialize with sum(a)=   625.00000000000000     
 >>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<
 >>> WARNING: SERIALIZATION IS ON <<<
 >>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<

But consumer reads its contents as zeroes:

$./consumer
 Before read from serializer: sum(a)=   0.0000000000000000     
 >>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<
 >>> WARNING: SERIALIZATION IS ON <<<
 >>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<
 After read from serializer: sum(a)=   0.0000000000000000     

The same with consumer_perturb:

$./consumer_perturb 
 Before read from serializer: sum(a)=   0.0000000000000000     
 >>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<
 >>> WARNING: SERIALIZATION IS ON <<<
 >>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<
 After read from serializer: sum(a)=   0.0000000000000000     

$ncview SerialboxTest_ser_a.nc 
Ncview 2.1.6 David W. Pierce  29 Oct 2015
http://meteora.ucsd.edu:80/~pierce/ncview_home_page.html
Copyright (C) 1993 through 2015, David W. Pierce
Ncview comes with ABSOLUTELY NO WARRANTY; for details type `ncview -w'.
This is free software licensed under the Gnu General Public License version 3; type `ncview -c' for redistribution details.

Note: could not open file /home/adam/.ncviewrc for reading
no displayable variables found!

Add proper tests for gridtools layouts with GPU backend

Currently we test the gridtools GPU layouts by using the host backend but a custom layout. We should use proper GPU tests including testing of sync to host and back.

Needs a protection for travis which is not CUDA enabled.

Reenable netcdf on CSCS machines

I disabled netcdf because we need to find a clean way how to propagate shared libraries via cmake config.
The problem: currently SerialboxConfig.cmake tries to find netcdf if it was enabled on build of serialbox. When building another application which depends on this installation and the module is not loaded it cannot locate the netcdf libraries and will not link.

Need help from someone with good knowledge in cmake best practices...

Fix version in doc

@thfabian If I remember correctly you wanted to fix the problem of versions in too many places?

Allow Fortran field which changes size during the run

lxavier

Currently we get error when trying to write field which do not have a constant size (e.g. data assimilation):
Error: sizes of field i_prof passed with sizes (2658x1x1x1), but registered with sizes (2657x1x1x1)

Import Serialbox1 documentation into sphinx

I copied the old documentation to docs/archive/serialbox1. We should integrate it in sphinx and update outdated parts.

Dependency problem in CMake when compiling Fortran examples

Sometimes the compilation fails with the following error

Fatal Error: Can't delete temporary module file './m_ser.mod0': No such file or directory
Fatal Error: Can't delete temporary module file './m_ser.mod0': No such file or directory
make[3]: *** [CMakeFiles/main_consumer.dir/pp/m_ser.f90.o] Error 1
make[3]: *** [CMakeFiles/main_consumer_perturb.dir/pp/m_ser.f90.o] Error 1
make[2]: *** [CMakeFiles/main_consumer.dir/pp/m_ser.f90.o.provides] Error 2
make[2]: *** [CMakeFiles/main_consumer_perturb.dir/pp/m_ser.f90.o.provides] Error 2
make[1]: *** [CMakeFiles/main_consumer.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/main_consumer_perturb.dir/all] Error 2
make: *** [all] Error 2

Drop support for multiple hashing algorithms

Problem in building serialbox2 with gridtools unittests

[  8%] Built target gtest
[ 30%] Built target SerialboxCoreLibraryObjects
[ 31%] Built target SerialboxCoreLibraryShared
[ 32%] Built target SerialboxCoreLibraryStatic
[ 41%] Built target SerialboxCLibraryObjects
[ 42%] Built target SerialboxCLibraryStatic
[ 42%] Built target sdbPython
[ 42%] Built target SerialboxPython
[ 43%] Built target SerialboxCLibraryShared
[ 47%] Built target SerialboxUnittestUtility
[ 48%] Building CXX object test/serialbox/core/CMakeFiles/SerialboxCppUnittest.dir/frontend/gridtools/UnittestStorageView.cpp.o
In file included from /home/adam/tmp/gridtools/include/storage/meta_storage.hpp:39:0,
                 from /home/adam/tmp/gridtools/include/storage/meta_storage_extender.hpp:37,
                 from /home/adam/tmp/gridtools/include/storage/storage.hpp:51,
                 from /home/adam/tmp/gridtools/include/storage/storage_metafunctions.hpp:44,
                 from /home/adam/tmp/gridtools/include/storage/storage-facility.hpp:38,
                 from /home/adam/tmp/serialbox2/test/utility/GridTools.h:27,
                 from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:15:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h: In instantiation of ‘std::vector<int, std::allocator<int> > serialbox::gridtools::internal::get_strides(const StorageType&, const MetaDataType&) [with MetaDataType = gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 1u>, gridtools::layout_map<0, 1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u> > >; StorageType = gridtools::storage<gridtools::base_storage<gridtools::wrap_pointer<int, true>, gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 1u>, gridtools::layout_map<0, 1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u> > >, 1u> >]’:
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:181:3:   required from ‘void GridToolsStorageViewTest_Construction_Test<gtest_TypeParam_>::TestBody() [with gtest_TypeParam_ = int]’
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:485:1:   required from here
/home/adam/tmp/gridtools/include/storage/meta_storage_base.hpp:134:42: error: ‘gridtools::array<int, 2ul> gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 1u>, gridtools::layout_map<0, 1>, false>::m_strides’ is protected
         array< int_t, space_dimensions > m_strides;
                                          ^
In file included from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:23:0:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h:43:41: error: within this context
   const auto& strides_array = meta_data.m_strides;
                                         ^
In file included from /home/adam/tmp/gridtools/include/storage/meta_storage.hpp:39:0,
                 from /home/adam/tmp/gridtools/include/storage/meta_storage_extender.hpp:37,
                 from /home/adam/tmp/gridtools/include/storage/storage.hpp:51,
                 from /home/adam/tmp/gridtools/include/storage/storage_metafunctions.hpp:44,
                 from /home/adam/tmp/gridtools/include/storage/storage-facility.hpp:38,
                 from /home/adam/tmp/serialbox2/test/utility/GridTools.h:27,
                 from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:15:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h: In instantiation of ‘void* serialbox::gridtools::internal::get_origin_ptr(const StorageType&, const MetaDataType&, unsigned int) [with StorageType = gridtools::storage<gridtools::base_storage<gridtools::wrap_pointer<int, true>, gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 1u>, gridtools::layout_map<0, 1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u> > >, 1u> >; MetaDataType = gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 1u>, gridtools::layout_map<0, 1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u> > >]’:
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:181:3:   required from ‘void GridToolsStorageViewTest_Construction_Test<gtest_TypeParam_>::TestBody() [with gtest_TypeParam_ = int]’
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:485:1:   required from here
/home/adam/tmp/gridtools/include/storage/meta_storage_base.hpp:134:42: error: ‘gridtools::array<int, 2ul> gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 1u>, gridtools::layout_map<0, 1>, false>::m_strides’ is protected
         array< int_t, space_dimensions > m_strides;
                                          ^
In file included from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:23:0:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h:91:41: error: within this context
   const auto& strides_array = meta_data.m_strides;
                                         ^
In file included from /home/adam/tmp/gridtools/include/storage/meta_storage.hpp:39:0,
                 from /home/adam/tmp/gridtools/include/storage/meta_storage_extender.hpp:37,
                 from /home/adam/tmp/gridtools/include/storage/storage.hpp:51,
                 from /home/adam/tmp/gridtools/include/storage/storage_metafunctions.hpp:44,
                 from /home/adam/tmp/gridtools/include/storage/storage-facility.hpp:38,
                 from /home/adam/tmp/serialbox2/test/utility/GridTools.h:27,
                 from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:15:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h: In instantiation of ‘std::vector<int, std::allocator<int> > serialbox::gridtools::internal::get_strides(const StorageType&, const MetaDataType&) [with MetaDataType = gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 3u>, gridtools::layout_map<0, 1, -1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u, 3u> > >; StorageType = gridtools::storage<gridtools::base_storage<gridtools::wrap_pointer<int, true>, gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 3u>, gridtools::layout_map<0, 1, -1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u, 3u> > >, 1u> >]’:
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:201:3:   required from ‘void GridToolsStorageViewTest_Construction_Test<gtest_TypeParam_>::TestBody() [with gtest_TypeParam_ = int]’
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:485:1:   required from here
/home/adam/tmp/gridtools/include/storage/meta_storage_base.hpp:134:42: error: ‘gridtools::array<int, 3ul> gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 3u>, gridtools::layout_map<0, 1, -1>, false>::m_strides’ is protected
         array< int_t, space_dimensions > m_strides;
                                          ^
In file included from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:23:0:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h:43:41: error: within this context
   const auto& strides_array = meta_data.m_strides;
                                         ^
In file included from /home/adam/tmp/gridtools/include/storage/meta_storage.hpp:39:0,
                 from /home/adam/tmp/gridtools/include/storage/meta_storage_extender.hpp:37,
                 from /home/adam/tmp/gridtools/include/storage/storage.hpp:51,
                 from /home/adam/tmp/gridtools/include/storage/storage_metafunctions.hpp:44,
                 from /home/adam/tmp/gridtools/include/storage/storage-facility.hpp:38,
                 from /home/adam/tmp/serialbox2/test/utility/GridTools.h:27,
                 from /home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:15:
/home/adam/tmp/serialbox2/src/serialbox/core/frontend/gridtools/StorageViewHelper.h: In instantiation of ‘void* serialbox::gridtools::internal::get_origin_ptr(const StorageType&, const MetaDataType&, unsigned int) [with StorageType = gridtools::storage<gridtools::base_storage<gridtools::wrap_pointer<int, true>, gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 3u>, gridtools::layout_map<0, 1, -1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u, 3u> > >, 1u> >; MetaDataType = gridtools::meta_storage<gridtools::meta_storage_aligned<gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 3u>, gridtools::layout_map<0, 1, -1>, false>, gridtools::aligned<0u>, gridtools::halo<1u, 2u, 3u> > >]’:
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:201:3:   required from ‘void GridToolsStorageViewTest_Construction_Test<gtest_TypeParam_>::TestBody() [with gtest_TypeParam_ = int]’
/home/adam/tmp/serialbox2/test/serialbox/core/frontend/gridtools/UnittestStorageView.cpp:485:1:   required from here
/home/adam/tmp/gridtools/include/storage/meta_storage_base.hpp:134:42: error: ‘gridtools::array<int, 3ul> gridtools::meta_storage_base<mpl_::integral_c<unsigned int, 3u>, gridtools::layout_map<0, 1, -1>, false>::m_strides’ is protected
         array< int_t, space_dimensions > m_strides;
                                          ^

Steps to reproduce:

Install vanilla Ubuntu 16.04 server and then

sudo apt install cmake build-essential gfortran git libboost-filesystem-dev libboost-system-dev libboost-chrono-dev libboost-log-dev
git clone --depth 1 https://github.com/eth-cscs/serialbox2.git
git clone --depth 1 [email protected]:GridTools/gridtools.git
mkdir serialbox2/build
cd serialbox2/build
cmake -DSERIALBOX_TESTING=ON -DSERIALBOX_TESTING_GRIDTOOLS=ON -DSERIALBOX_GRIDTOOLS_ROOT=../../gridtools -DGRIDTOOLS_ROOT=../../gridtools ..
make

Fix linker problem with Fortran examples

My changes in #68 fix PGI but break gnu and cray...

Compare.py doesn't display double precision numbers

When we are looking at error smaller than 10e-10 we cannot see them

compare.py -t 1e-10

Failed values (displayed 10 of 10998):
    (0, 0, 0): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 1): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 2): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 3): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 4): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 5): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 6): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 7): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 8): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000
    (0, 0, 9): value_1 = 0.1498301625, value_2 = 0.1498301625, error = 0.0000000000

Drop STELLA support once STELLA is decommissioned

Fortran example doesn't compile with PGI

/UnittestErrorHandling.cpp.o
/usr/lib/../lib64/crt1.o: In function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.22/csu/../sysdeps/x86_64/start.S:114: undefined reference to `main'
/usr/bin/ld: link errors found, deleting executable `fortran_consumer'
/usr/bin/X11/sha1sum: fortran_consumer: No such file or directory
collect2: error: ld returned 1 exit status
examples/fortran/simple/CMakeFiles/fortran_consumer.dir/build.make:131: recipe for target 'examples/fortran/simple/fortran_consumer' failed
make[2]: *** [examples/fortran/simple/fortran_consumer] Error 1
CMakeFiles/Makefile2:1020: recipe for target 'examples/fortran/simple/CMakeFiles/fortran_consumer.dir/all' failed

Travis: enable Fortran unit tests

Requires to build pfunit.

Implement serialization for gridtools data_store_field

STELLA: Inconsistency between copy constructor and assignment operator

The interface for the Savepoint is inconsistent as the operator= is doing a deep copy while the copy constructor is doing a shallow copy. The reason is STELLA's Init pattern. We won't fix this. We keep this issue open as a reference in case someone runs into this problem.

thfabian:

Just as a side node the problem with the copy constructor in Savepoint (due to Init patterm) is probably also present in all the other stella classes (MetainfoSet and DataFieldInfo).
Maybe the proper solution would be to add an assert if Init has not been called and the user attempts to copy the object?

@thfabian @andyspiros Is this a correct summary of the issue? Maybe you can expand if you think there should be more info.

Rename json files

I think we should rename the two json files to something like
ArchiveMetaData-<Field>.json -> <Field>.archive.json
MetaData-<Field>.json -> <Field>.metadata.json

It is much easier to copy all relevant files cp <Field>* and should be easy to do it in a backward compatible way (rename files when they have the old name)...

Strategy for versioning

Currently we pretend to be backwards compatible, i.e. a new serialbox version will read an archive from an earlier version. (This was changed recently, before it was a requirement to have the same version for writing and reading.)

Probably the patch version should be backwards compatible i.e. 2.1.x
but minor version should be incompatible 2.x.0.

Compilation fail with gcc 8, Ubuntu 17.10, toolchain repo

The error message is as follows:

Scanning dependencies of target SerialboxCoreLibraryObjects
[  2%] Building CXX object src/serialbox/core/CMakeFiles/SerialboxCoreLibraryObjects.dir/FieldMap.cpp.o
In file included from /home/piotrows/propoze/serialbox2/src/serialbox/core/MetainfoMapImpl.h:20:0,
                 from /home/piotrows/propoze/serialbox2/src/serialbox/core/FieldMetainfoImpl.h:20,
                 from /home/piotrows/propoze/serialbox2/src/serialbox/core/FieldMap.h:18,
                 from /home/piotrows/propoze/serialbox2/src/serialbox/core/FieldMap.cpp:15:
/home/piotrows/propoze/serialbox2/src/serialbox/core/MetainfoValueImpl.h: In instantiation of ‘T serialbox::MetainfoValueImpl::as() const [with T = long unsigned int]’:
/home/piotrows/propoze/serialbox2/src/serialbox/core/MetainfoValueImpl.h:87:18:   required from ‘serialbox::MetainfoValueImpl::operator T() const [with T = long unsigned int]’
/usr/include/c++/8/type_traits:935:12:   required from ‘struct std::is_constructible<long unsigned int, serialbox::MetainfoValueImpl>’
/home/piotrows/propoze/serialbox2/src/serialbox/core/external/json/json.hpp:1476:96:   required by substitution of ‘template<class CompatibleNumberUnsignedType, typename std::enable_if<((std::is_constructible<long unsigned int, CompatibleNumberUnsignedType>::value && std::numeric_limits<_Tp>::is_integer) && (! std::numeric_limits<_Tp>::is_signed)), CompatibleNumberUnsignedType>::type <anonymous> > nlohmann::basic_json<>::basic_json(CompatibleNumberUnsignedType) [with CompatibleNumberUnsignedType = serialbox::MetainfoValueImpl; typename std::enable_if<((std::is_constructible<long unsigned int, CompatibleNumberUnsignedType>::value && std::numeric_limits<_Tp>::is_integer) && (! std::numeric_limits<_Tp>::is_signed)), CompatibleNumberUnsignedType>::type <anonymous> = <missing>]’
/usr/include/c++/8/type_traits:935:12:   required from ‘struct std::is_constructible<nlohmann::basic_json<>, serialbox::MetainfoValueImpl>’
/home/piotrows/propoze/serialbox2/src/serialbox/core/external/json/json.hpp:1145:117:   required by substitution of ‘template<class CompatibleObjectType, typename std::enable_if<(std::is_constructible<std::__cxx11::basic_string<char>, typename CompatibleObjectType::key_type>::value && std::is_constructible<nlohmann::basic_json<>, typename CompatibleObjectType::mapped_type>::value), int>::type <anonymous> > nlohmann::basic_json<>::basic_json(const CompatibleObjectType&) [with CompatibleObjectType = serialbox::MetainfoMapImpl; typename std::enable_if<(std::is_constructible<std::__cxx11::basic_string<char>, typename CompatibleObjectType::key_type>::value && std::is_constructible<nlohmann::basic_json<>, typename CompatibleObjectType::mapped_type>::value), int>::type <anonymous> = <missing>]’
/usr/include/c++/8/ext/new_allocator.h:136:4:   [ skipping 3 instantiation contexts, use -ftemplate-backtrace-limit=0 to disable ]
/usr/include/c++/8/bits/shared_ptr_base.h:662:4:   required from ‘std::__shared_count<_Lp>::__shared_count(std::_Sp_make_shared_tag, _Tp*, const _Alloc&, _Args&& ...) [with _Tp = serialbox::MetainfoMapImpl; _Alloc = std::allocator<serialbox::MetainfoMapImpl>; _Args = {const serialbox::MetainfoMapImpl&}; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/8/bits/shared_ptr_base.h:1328:35:   required from ‘std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<serialbox::MetainfoMapImpl>; _Args = {const serialbox::MetainfoMapImpl&}; _Tp = serialbox::MetainfoMapImpl; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/8/bits/shared_ptr.h:360:64:   required from ‘std::shared_ptr<_Tp>::shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<serialbox::MetainfoMapImpl>; _Args = {const serialbox::MetainfoMapImpl&}; _Tp = serialbox::MetainfoMapImpl]’
/usr/include/c++/8/bits/shared_ptr.h:706:14:   required from ‘std::shared_ptr<_Tp> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = serialbox::MetainfoMapImpl; _Alloc = std::allocator<serialbox::MetainfoMapImpl>; _Args = {const serialbox::MetainfoMapImpl&}]’
/usr/include/c++/8/bits/shared_ptr.h:722:39:   required from ‘std::shared_ptr<_Tp> std::make_shared(_Args&& ...) [with _Tp = serialbox::MetainfoMapImpl; _Args = {const serialbox::MetainfoMapImpl&}]’
/home/piotrows/propoze/serialbox2/src/serialbox/core/FieldMetainfoImpl.h:37:87:   required from here
/home/piotrows/propoze/serialbox2/src/serialbox/core/MetainfoValueImpl.h:77:5: error: static assertion failed: ValueType is not supported
     static_assert(IsSupported<typename MakePrimitive<T>::type>::value,
     ^~~~~~~~~~~~~
src/serialbox/core/CMakeFiles/SerialboxCoreLibraryObjects.dir/build.make:62: recipe for target 'src/serialbox/core/CMakeFiles/SerialboxCoreLibraryObjects.dir/FieldMap.cpp.o' failed
make[5]: *** [src/serialbox/core/CMakeFiles/SerialboxCoreLibraryObjects.dir/FieldMap.cpp.o] Error 1
CMakeFiles/Makefile2:166: recipe for target 'src/serialbox/core/CMakeFiles/SerialboxCoreLibraryObjects.dir/all' failed
make[4]: *** [src/serialbox/core/CMakeFiles/SerialboxCoreLibraryObjects.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/serialbox2.dir/build.make:110: recipe for target '/home/piotrows/propoze/build/serialbox2_GNU_Debug/src/serialbox2-stamp/serialbox2-build' failed
make[2]: *** [/home/piotrows/propoze/build/serialbox2_GNU_Debug/src/serialbox2-stamp/serialbox2-build] Error 2
CMakeFiles/Makefile2:747: recipe for target 'CMakeFiles/serialbox2.dir/all' failed
make[1]: *** [CMakeFiles/serialbox2.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

[serialbox-c] Uninitialized memory

The memset in https://github.com/eth-cscs/serialbox2/blob/master/src/serialbox-c/Utility.h#L126 is incorrect as it only initializes the first array element.

How to get the `serialbox2/examples/fortran/perturbation` compiled?

I think it is disabled in CMake (AFAIK Cmake doesn't even mention this directory), and when I tried to fix it, I spend 2 hours and gave up.

But this example is exactly the thing we would like to have working to use the serialbox2 as a drop-in replacement for SB1.

Backport tools from old serialbox

Tools from old serialbox are not in the current repository. Would be nice to have at leat the compare tool.

https://github.com/MeteoSwiss-APN/serialbox/tree/master/tools

Jenkins: add daint as target

Jenkins: Use pre-compiled pFUnit

Currently we rebuild pFUnit on every Jenkins launch...

Limitation of version number

Currently we have a limitation that X has to be in [0...9] in the version number Y.X.X

Because we do multiplication by 10 inside the library to get a single integer version identifier.
(100*Major+10*Minor+Patch)

Is there support for Netcdf from within the Fortran code?

I want the Fortran code to serialize a field directly into netcdf format. Is that supported/possible?

Conflict of serialbox parsing and PURE keyword

MeteoSwiss-APN/serialbox#75

pitrows

We have noticed that serialization removes INTENT(IN), which is in conflict with PURE concept that demands INTENT(IN). For example, this realization results in compilation error:

    PURE FUNCTION eval_v1(xant,u1,u2,u3,hmx,i,j,k,np,mp,lp,ih)
        INTEGER(KIND=iintegers),INTENT(IN) ::i,j,k,np,mp,lp,ih
#ifdef SERIALIZE
        REAL(KIND=euwp),DIMENSION(1-ih:np+ih, 1-ih:mp+ih, 1-ih:lp+ih) :: &
#else
        REAL(KIND=euwp),DIMENSION(1-ih:np+ih, 1-ih:mp+ih, 1-ih:lp+ih),INTENT(IN) :: &
#endif
            u1, u2, u3, xant

clementval

A workaround would be to only be able to write serialized data in a PURE function and not read. Then, it wouldn't be needed to remove the INTENT(IN).
At the moment, this is a correct behavior of pp_ser.py

adamryczkowski

Can you explain me again reasons for this feature? Why is it there? What is the use case?

piotrows

I think it is not worth to spend time on discussing this issue in present context, at least from my side. As far as I remember, it is about IN/OUT attributes.

clementval

It's because of the read features of serialbox. If you serialize data in a function/subroutine, you have no problem. But when you want to read serialized data back into this function/subroutine serialbox has to modify the INTENT in order to be able to read the data from the serialized archive and write them in their corresponding variable.
The problem occurs with PURE function/subroutine where modifying the INTENT can be a problem. This is not addressed in serialbox yet.

Check for proper error handling if files are missing

Check what happens if one of the archives is broken and add proper error handling if needed.

Examples in the serialbox2 don't compile

Steps to reproduce:

git clone --depth 1 https://github.com/eth-cscs/serialbox2.git
git clone --depth 1 ssh://[email protected]/GridTools/gridtools -b serialbox2_support
cd serialbox2
mkdir build
cd build
cmake ..
cmake-gui .. 

In cmake-gui set SERIALBOX_TESTING_GRIDTOOLS, and assign to GRIDTOOLS_ROOT the path to the cloned gridtools repository

make

I get the error:

Scanning dependencies of target example-01-laplacian-cpp
[ 94%] Building CXX object examples/gridtools/simple/CMakeFiles/example-01-laplacian-cpp.dir/example-01-laplacian.cpp.o
In file included from /usr/include/boost/move/traits.hpp:31:0,
                 from /usr/include/boost/move/utility.hpp:29,
                 from /usr/include/boost/optional/optional.hpp:48,
                 from /usr/include/boost/optional.hpp:15,
                 from /usr/include/boost/format/internals.hpp:20,
                 from /usr/include/boost/format.hpp:38,
                 from /home/Adama-docs/Adam/MyDocs/praca/IMGW/repos/serialbox2/src/serialbox/core/Exception.h:18,
                 from /home/Adama-docs/Adam/MyDocs/praca/IMGW/repos/serialbox2/src/serialbox/core/frontend/gridtools/Exception.h:18,
                 from /home/Adama-docs/Adam/MyDocs/praca/IMGW/repos/serialbox2/src/serialbox/gridtools/serialbox.hpp:28,
                 from /home/Adama-docs/Adam/MyDocs/praca/IMGW/repos/serialbox2/examples/gridtools/simple/example-01-laplacian.cpp:35:
/usr/include/boost/move/detail/type_traits.hpp:725:35: error: ‘bool_’ has not been declared
          static yes_type test(U&, bool_<sizeof(U(source<U>()))>* = 0);
                                   ^
/usr/include/boost/move/detail/type_traits.hpp:725:40: error: expected ‘,’ or ‘...’ before ‘<’ token
          static yes_type test(U&, bool_<sizeof(U(source<U>()))>* = 0);
                                        ^
examples/gridtools/simple/CMakeFiles/example-01-laplacian-cpp.dir/build.make:62: recipe for target 'examples/gridtools/simple/CMakeFiles/example-01-laplacian-cpp.dir/example-01-laplacian.cpp.o' failed
make[2]: *** [examples/gridtools/simple/CMakeFiles/example-01-laplacian-cpp.dir/example-01-laplacian.cpp.o] Error 1
CMakeFiles/Makefile2:981: recipe for target 'examples/gridtools/simple/CMakeFiles/example-01-laplacian-cpp.dir/all' failed
make[1]: *** [examples/gridtools/simple/CMakeFiles/example-01-laplacian-cpp.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

Review linking of libs shared/static

Invalid insertion of USE statement

pp_ser insert USE statement at the wrong place in the following example.

program sub1(dummyarg1,  &
  dummyarg2) 

  use dummy_mod
  implicit none
  real, dimension(10) :: dummyarg1, dummyarg2

  !$ser savepoint dummy
  !$ser data d1=dummyarg1 d2=dummyarg2
end program sub1

#define ACC_PREFIX !$acc
program sub1(dummyarg1,  &

#ifdef SERIALIZE
USE m_serialize, ONLY: &
  fs_add_savepoint_metainfo, &
  fs_read_and_perturb_field, &
  fs_read_field, &
  fs_create_savepoint, &
  fs_write_field
USE utils_ppser, ONLY:  &
  ppser_get_mode, &
  ppser_savepoint, &
  ppser_serializer, &
  ppser_serializer_ref, &
  ppser_intlength, &
  ppser_reallength, &
  ppser_realtype, &
  ppser_zrperturb
#endif

  dummyarg2) 

  use dummy_mod
  implicit none
  real, dimension(10) :: dummyarg1, dummyarg2

#ifdef SERIALIZE
call fs_create_savepoint('dummy', ppser_savepoint)
! file: issue.f90 lineno: #9
SELECT CASE ( ppser_get_mode() )
  CASE(0)
    call fs_write_field(ppser_serializer, ppser_savepoint, 'd1', dummyarg1)
    call fs_write_field(ppser_serializer, ppser_savepoint, 'd2', dummyarg2)
  CASE(1)
    call fs_read_field(ppser_serializer_ref, ppser_savepoint, 'd1', dummyarg1)
    call fs_read_field(ppser_serializer_ref, ppser_savepoint, 'd2', dummyarg2)
  CASE(2)
    call fs_read_and_perturb_field(ppser_serializer_ref, ppser_savepoint, 'd1', dummyarg1, ppser_zrperturb)
    call fs_read_and_perturb_field(ppser_serializer_ref, ppser_savepoint, 'd2', dummyarg2, ppser_zrperturb)
END SELECT
#endif
end program sub1

pp_ser: create files with same privileges as original file

Add check for build failure in build.sh

Currently we have a build problem with pgi on kesch and daint. However all tests pass and the jenkins plan is green, but the library is not installed.

Cleanup CMake

[C] Segmentation fault when reading meta info

In the Fortran unit test everything works fine when it calls fs_get_halos, but in a real world program I'm receiving a segmentation fault from serialbox-c/FieldMetainfo.cpp.

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fe89cc8216a in ???
#1  0x7fe89cc81393 in ???
#2  0x7fe89c69f13f in ???
#3  0x7fe89d1fb1d2 in ???
#4  0x7fe89d1fabb3 in serialboxFieldMetainfoGetMetainfo
        at /home/christian/workspace/serialbox2/src/serialbox-c/FieldMetainfo.cpp:104
#5  0x7fe89d20c28b in serialboxSerializerGetFieldMetainfo2
        at /home/christian/workspace/serialbox2/src/serialbox-c/Serializer.cpp:337
#6  0x7fe89d22128a in serialboxFortranSerializerGetFieldHalos
        at /home/christian/workspace/serialbox2/src/serialbox-c/FortranWrapper.cpp:194
#7  0x7fe89d1e8862 in __m_serialize_MOD_fs_get_halos
        at /home/christian/workspace/serialbox2/src/serialbox-fortran/m_serialize.f90:928
#8  0x558cc6f55501 in test
        at /home/christian/workspace/serialbox2-metainfo-bug/test.f90:46
#9  0x558cc6f55657 in main
        at /home/christian/workspace/serialbox2-metainfo-bug/test.f90:3

I have created an example including the files from the unit test that work fine and the one that causes the crash. I kept the example in Fortran because it's just easier to get a stack trace there, but I have tried to use the C library directly and it also ended up in a seg fault.

serialbox2-metainfo-bug.zip

Fix SerialboxConfig.sh.in

I think my hack in
https://github.com/eth-cscs/serialbox2/blob/f9e7b7abd6dcc25a0bb603fcf9943a9257d7da13/cmake/CMakeLists.txt#L31
is very stupid we should find a proper solution, e.g. generate the library variables in CMakeLists.txt and expose the content to the SerialboxConfig.cmake.in and SerialboxConfig.sh.in.

add ability to break generated line in multiple line with &

Some compiler have limits on the number of characters you can have on a line. Would be nice to be able to specify this in the pp_ser in order to break lines that are generated and have a default at 132 columns.

Compiler Warning: warning: Dynamic exception specifications are deprecated in C++11 [-Wdeprecated]

I'm getting the following warnings when compiling with GCC-7 on OSX:

n file included from /Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/ArchiveFactory.h:19:0,
                 from /Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/ArchiveFactory.cpp:15:
/Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/Archive.h:45:72: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
                         const std::shared_ptr<FieldMetainfoImpl> info) throw(Exception) = 0;
                                                                        ^~~~~
/Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/Archive.h:53:68: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
                     std::shared_ptr<FieldMetainfoImpl> info) const throw(Exception) = 0;
                                                                    ^~~~~
In file included from /Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/ArchiveFactory.cpp:18:0:
/Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/BinaryArchive.h:89:72: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
                         const std::shared_ptr<FieldMetainfoImpl> info) throw(Exception) override;
                                                                        ^~~~~
/Users/pascal/Meteo/Code/serialbox2/src/serialbox/core/archive/BinaryArchive.h:92:68: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
                     std::shared_ptr<FieldMetainfoImpl> info) const throw(Exception) override;
                                                                    ^~~~~

There are currently 85 similar warnings shown on locations such as these: https://github.com/eth-cscs/serialbox2/blob/master/src/serialbox/core/archive/Archive.h#L45

On Stackoverflow is a discussion on what exception specifiers are and why they are bad.
Following the answer:

So in summary, exception specs aren't enforced strongly enough in the language to actually make code any safer, and implementing them as specified was a big performance drain.

I suggest remove all the exception specifiers, especially since they are being deprecated from C++11.

C Interface: C-Files should always have .c at the end of the file

Frontend for FortranTestGenerator

Hi all,

there are some features left that I have added to serialbox-ftg and not yet integrated to serialbox2. serialbox-ftg is based on serialbox1 and used by the FortranTestGenerator.

The features are:

1. Routines for allocating allocatables and pointer variables based on there bounds stored in the halo fields.
2. Bullshit recognition, i.e. do not try to write fields that seem to be uninitialized pointers or the like. Since you cannot tell it for sure (ASSOCIATED only works if the pointer was at least initialized with NULL()), I have implemented a heuristic for that.
3. Do not try to read fields that do not exist (without terminating with error).
4. Always store the address of a written field as meta info.
5. The option to register fields of any type without actually writing the data, just to store the meta data.

That goes all beyond serialbox' core functionality, so I'm not sure what will be the right way to integrate it. I think, there are three options:

• Just add them to m_serialize.f90 as done in serialbox-ftg.
• Create a FortranTestGenerator frontend and put it to src/serialbox-fortran/ or somewhere else.
• Create a FortranTestGenerator frontend and keep it outside of serialbox2.

Each feature could also be handled differently.

What do you think?

Cheers, Christian

Remove dependency on boost::log

@thfabian Will remove the dependency

Does serialbox2 supports serializing field into more than one format?

(Continuation of the closed topic #61 (comment))

This was done to easily convert your data from one archive to another (e.g Binary to NetCDF). All archive specific meta-data are stored in the ArchiveMetaData-*.json while the archive independent meta-data (like the Savepoints etc.) are stored in the normal MetaData-field.json.

It suggests you are preparing to support saving in hetergenous formats, e.g. one field in binary format, and another in NetCDF. Am I right? Is it already supported? We do use Serialbox to serialize into two formats, but we solve it on the level of CMake and conditional compilation.

Why does serialbox2 uses two distinct metadata JSON files?

I am curious, why did you introduce another metadatafile with Serialbox2? The JSON already handles structured data, so you could simply store the additional metadata inside the original JSON file (the one without the Metadata- prefix).

Serialization of unallocated array from Fortran

MeteoSwiss-APN/serialbox#18

clementval:

In some cases, it can happen that an array declared in a serialize directive is not allocated because the chosen path in code was different (because of the input data). In that case, the Fortran wrapper should check weather the array is allocated or not. If not, just output a warning instead of having a segmentation fault would make more sense.

Does serialbox2 detects automatically read format?

We are going to change the serialization format for all our fields from the "binary" to "NetCDF".

When I read the fields using serialbox::gridtools::serializer object, it accepts archive_name argument, where I can specify "NetCDF".

Since you store so much metadata, I wonder whether you can support automatic detection of the input archive format. I suspect you don't, but I ask to be sure.

implement unittests for pp_ser

MeteoSwiss-APN/serialbox#2

pp_ser is not installed with execute right

pp_ser.py should be executable once installed. This is not the case currently.

Replace netcdf support by convert script?

or binary archive by netcdf (introduces a dependency)?

Remove boost::filesystem once std::filesystem is established

Currently we use std::experimental::filesystem if enabled and available with fall back on boost if not.