garrison / eigen3-hdf5

Bare-bones hdf5 serialization of Eigen matrices and vectors

eigen3-hdf5

Easy serialization of C++ Eigen matrices using HDF5.

The library is meant to be bare-bones (at least for now). It gets me 90% of what I want/need in a few hundred lines of code. It may also get you 90% (or even 100%) of what you need.

Requirements

• Eigen3 (tested on 3.1 and 3.2 branches)
• HDF5 C++ wrapper library >= 1.8.12 (yes, this is a very recent version)

Because eigen3-hdf5 is a template library, there is nothing to link against (besides the HDF5 libraries).

API

Supports saving and restoring Eigen matrices and vectors of float, double, long double, int, unsigned int, and std::complex<>.

#include <eigen3-hdf5.hpp>

void save_matrix()
{
    Eigen::Matrix3d mat;
    mat << 1, 2, 3, 4, 5, 6, 7, 8, 9;
    H5::H5File file("filename1.h5", H5F_ACC_TRUNC);
    EigenHDF5::save(file, "MatrixDataSetName", mat);
}

void load_vector()
{
    Eigen::Vector4i vec;
    H5::H5File file("filename2.h5", H5F_ACC_RDONLY);
    EigenHDF5::load(file, "VectorDataSetName", vec);
}

See the unittests directory for more examples.

Unit tests

I am using premake4 and googletest because I am familiar with them.

The unit tests currently write to specific files in the current directory. This could change, eventually.

The GitHub Action can be approximated locally by installing act and running:

act -P ubuntu-20.04=ghcr.io/catthehacker/ubuntu:act-20.04

License

MIT license.

Next steps

• Support more fundamental data types

Thoughts/notes

• Using the HDF5 C++ wrapper library supposedly means it won't work with parallel hdf5. If I were to do it again, I would write eigen3-hdf5 using the regular C HDF5 API, not the C++ wrapper. Patches are welcome. :)

eigen3-hdf5's Issues

header file

How can I use these .hpp files with g++ compiler ?

Extensible dataset

Much of my work involves processing time samples, so I need to be able to save data with an unlimited axis. Do you have thoughts or suggestions on adding this capability to eigen3-hdf5? If I get something reasonable would you want a pull request or do you want to keep your code free of this additional complexity?

Why is stride not needed when calling dataset.write()?

I am impressed by this project. You have a nice clean implementation, and I look forward to trying it out. But in studying the code and comparing it with other code (in particular the opencv mat to eigen conversion function), I am wondering if there is a problem with your save method where you call dataset.write. When using the .data() buffer of an eigen object, I think one needs to take into account the strides, both the inner and outer strides, but particularly the outer. The particular code from opencv I am referring to is at opencv/eigen.hpp at master · Itseez/opencv in the eigen2cv function. Here is the relevant snippet:

    Mat _src(src.rows(), src.cols(), DataType<_Tp>::type,
             (void*)src.data(), src.stride()*sizeof(_Tp));
    _src.copyTo(dst);

I appreciate your time and feedback.

[false issue]can not load from file

I am using the following code to test this package。Notice that I used a different name in EigenHDF5::load, it won't work out and I get an IO exception:

Eigen::Matrix<long double, Eigen::Dynamic, Eigen::Dynamic> mat(3, 4), mat2;
    mat << 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12;
    std::cout << mat << std::endl;
    {
        H5::H5File file("test_MatrixRoundTrip_LongDouble.h5", H5F_ACC_TRUNC);
        EigenHDF5::save(file, "longdouble_matrix", mat);
    }
    {
        H5::H5File file("test_MatrixRoundTrip_LongDouble.h5", H5F_ACC_RDONLY);
        EigenHDF5::load(file, "mat", mat2);
    }

Exception is ：

HDF5-DIAG: Error detected in HDF5 (1.8.19) thread 0:
  #000: H5D.c line 356 in H5Dopen2(): not found
    major: Dataset
    minor: Object not found
  #001: H5Gloc.c line 428 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #002: H5Gtraverse.c line 859 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 639 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #004: H5Gloc.c line 383 in H5G_loc_find_cb(): object 'mat' doesn't exist
    major: Symbol table
    minor: Object not found
terminate called after throwing an instance of 'H5::FileIException'

I think the tests in this package do not really pass because if you use the same longdouble_matrix name when you load from "test_MatrixRoundTrip_LongDouble.h5", then I think you are using the same matrix when you saving to file and the data is never loaded from file.

Changes in HDF5 breaking build

The current HDF5 trunk does not have some functions (like Attribute) in the C++ class. As a result the code is not building. We can revert back to older versions, but apparently those libraries are causing other projects to crash from a severe memory leak! Any thoughts?
Specifically this function is where the code starts failing to compile
template <typename T> void save_scalar_attribute (const H5::H5Location &h5obj, const std::string &name, const T &value) { const H5::DataType * const datatype = DatatypeSpecialization<T>::get(); H5::DataSpace dataspace(H5S_SCALAR); H5::Attribute att = h5obj.createAttribute(name, *datatype, dataspace); att.write(*datatype, &value); }
Apparently, createAttribute is not there in the H5Location

over write issue

Hi, thx for the great library!
I have found out that if there is an existing file with the same name, it will not overwrite the file.
I am wondering if there are method to deal with this?
Or there is some flags that I should select properly..?

thx!

Valgrind: Syscall param write(buf) points to uninitialised byte(s)

When run under valgrind, the MatrixRoundTrip.LongDouble test emits a warning:

==3258== Syscall param write(buf) points to uninitialised byte(s)
==3258==    at 0x4F621E7: write (write.c:26)
==3258==    by 0x4960BAB: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x495A427: H5FD_write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x493985A: H5F__accum_write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4A3C30A: H5PB_write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x49441AC: H5F_block_write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4905602: H5D__flush_sieve_buf (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x48F7F15: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4905706: H5D__flush_real (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4905804: H5D_close (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x49061E5: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x49B4E1F: H5I_dec_ref (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==  Address 0x55ece92 is 18 bytes inside a block of size 200 alloc'd
==3258==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3258==    by 0x4964641: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4964E58: H5FL_blk_malloc (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4965059: H5FL_blk_calloc (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x48F739D: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4AF5910: H5VM_opvv (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x48F6F67: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4911FE8: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x4912849: H5D__select_write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x48F8036: H5D__contig_write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x490BEB0: H5D__write (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258==    by 0x490C69C: H5Dwrite (in /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.103.0.0)
==3258== 

I am not sure if this is a problem with eigen3-hdf5 or is entirely an issue with the underlying library. It might even be benign, in which case we should add a suppression. Once this is resolved, we should run valgrind with --error-exitcode=1 so CI fails if any new error is introduced.