llnl / metall

Persistent memory allocator for data-centric analytics

Metall: A Persistent Memory Allocator for Data-Centric Analytics

• A memory allocator enables applications to transparently allocate data into a file system.

  • From applications, Metall looks like a normal heap allocator.

  • Applications can keep their data beyond single process life cycles and reattach the data in succeeding runs.

  • Leverages the memory-mapped file mechanism (i.e., mmap(2)) to map application data in files to the main memory.

• Employs the API developed by Boost.Interprocess.

  • Useful for allocating C++ data structures (including STL containers).

• Incorporates state-of-the-art allocation algorithms.

• Provides a space-efficient snapshot/versioning, leveraging the reflink copy mechanism in file systems.

• See details: Metall overview slides.

Getting Started

Metall consists of only header files and requires some header files in Boost C++ Libraries.

All core files exist under metall/include/metall/.

Required

• Boost C++ Libraries 1.64 or more.
  • Build is not required; needs only their header files.
  • To use JSON containers in Metall, Boost C++ Libraries 1.75 or more is required.
• C++17 compiler
  • Tested with GCC 8.1 or more; however, 8.3 or more is recommended due to early implementation of the C++ Filesystem library.

Build

To build your program with Metall, all you have to do is just setting include paths such as '-I' or CPLUS_INCLUDE_PATH.

For example,

# Download Boost (Boost C++ Libraries 1.64 or more is required)
# One can skip this step if Boost is already available.
wget https://boostorg.jfrog.io/artifactory/main/release/1.78.0/source/boost_1_78_0.tar.gz
tar xvf boost_1_78_0.tar.gz
export BOOST_ROOT=$PWD/boost_1_78_0

git clone https://github.com/LLNL/metall
export METALL_INCLUDE=$PWD/metall/include

g++ -std=c++17 your_program.cpp -lstdc++fs -I${BOOST_ROOT} -I${METALL_INCLUDE}

Unofficial Support For Clang

Clang can be used instead of GCC to build Metall. However, we haven't tested it intensively. Also, Boost C++ Libraries 1.69 or more may be required if one wants to build Metall with Clang + CUDA.

Package Manager Support

Metall with Spack

Metall package is also available on Spack.

As Metall depends on Boost C++ Libraries, Spack also installs a proper version of Boost C++ Libraries automatically, if needed.

# Install Metall and Boost C++ Libraries
spack install metall

# Sets environment variables: BOOST_ROOT and METALL_ROOT.
# Boost C++ Libraries and Metall are installed at the locations, respectively.
spack load metall

# Build a program that uses Metall
# Please note that one has to put 'include' at the end of BOOST_ROOT and METALL_ROOT
g++ -std=c++17 your_program.cpp -lstdc++fs -I${BOOST_ROOT}/include -I${METALL_ROOT}/include

Metall with Connan

Metall is also available on Conan, thanks to the DICE research group. Conan Metall package information is here.

Use Metall from Another CMake Project

To download and/or link Metall package from a CMake project, see example CMake files placed here.

Build Example Programs

Metall repository contains some example programs under example directory. One can use CMake to build the examples. For more details, see a page here.

Documentations

• Full documentation
• API documentation

Generate API documentation using Doxygen

A Doxygen configuration file is here.

To generate API document:

cd metall
mkdir build_doc
cd build_doc
doxygen ../docs/Doxyfile.in

Publication

Keita Iwabuchi, Karim Youssef, Kaushik Velusamy, Maya Gokhale, Roger Pearce,
Metall: A persistent memory allocator for data-centric analytics,
Parallel Computing, 2022, 102905, ISSN 0167-8191, https://doi.org/10.1016/j.parco.2022.102905.

• Parallel Computing (journal)

• arXiv (preprint)

About

Contact

• GitHub Issues is open.

• Primary contact: Keita Iwabuchi (LLNL).

License

Metall is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.

All new contributions must be made under both the MIT and Apache-2.0 licenses.

See LICENSE-MIT, LICENSE-APACHE, NOTICE, and COPYRIGHT for details.

SPDX-License-Identifier: (Apache-2.0 OR MIT)

Release

LLNL-CODE-768617

metall's Issues

using typedef produces a compilation error in clang

This simple program doesn't compile with clang because of a redefiniton

// clang++ -std=c++17 -I$HOME/github.com/LLNL/metall.git/include/ main.cpp
#include<metall/metall.hpp>

int main(){}

commenting this line (15) in metall.hpp seems to work:

template <typename chunk_no_type, std::size_t chunk_size, typename kernel_allocator_type>
using basic_manager = basic_manager<chunk_no_type, chunk_size, kernel_allocator_type>;

this line doesn't seem to add information anyway because it repeats all the types.

I know the library explicitly requires clang, but it probably does for different reasons.

TSAN Data Races

Hi Keita,

First of all wanted to thank you for this great project, very useful. However, I've been running TSAN with Clang-18 on code that uses metall, and getting lots of data race warnings from the metall allocator. For instance,

WARNING: ThreadSanitizer: data race (pid=23601)
  Read of size 8 at 0x7b4c00022be8 by thread T96 (mutexes: write M0):
    #0 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::size() const metall/include/metall/kernel/chunk_directory.hpp:249:37
    #1 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::all_slots_marked(unsigned int) const metall/include/metall/kernel/chunk_directory.hpp:212:21
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_object_from_global_without_bin_lock(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:409:5
    #3 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_objects_from_global(unsigned char, unsigned long, long*) metall/include/metall/kernel/segment_allocator.hpp:392:13
    #4 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_object(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:375:5
    #5 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:150:31
    #6 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #7 metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::priv_allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:176:51
    #8 metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:120:54
    #9 std::allocator_traits<metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::allocate(metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>&, unsigned long) /include/c++/13.2.0/bits/alloc_traits.h:333:20
    #10 std::_Vector_base<long, metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_allocate(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:378:20
    #11 std::vector<long, metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_default_append(unsigned long) /include/c++/13.2.0/bits/vector.tcc:663:34
    #12 std::vector<long, metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::resize(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:1013:4

  Previous write of size 8 at 0x7b4c00022be8 by thread T57 (mutexes: write M1):
    #0 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::priv_insert_large_chunk(unsigned char) metall/include/metall/kernel/chunk_directory.hpp:605:30
    #1 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::insert(unsigned char) metall/include/metall/kernel/chunk_directory.hpp:106:27
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_large_object(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:483:58
    #3 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:151:31
    #4 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #5 metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::priv_allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:176:51
    #6 metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:120:54
    #7 std::allocator_traits<metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::allocate(metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>&, unsigned long) /include/c++/13.2.0/bits/alloc_traits.h:333:20
    #8 std::_Vector_base<float, metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_allocate(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:378:20
    #9 std::vector<float, metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_default_append(unsigned long) /include/c++/13.2.0/bits/vector.tcc:663:34
    #10 std::vector<float, metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::resize(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:1013:4

SUMMARY: ThreadSanitizer: data race metall/include/metall/kernel/chunk_directory.hpp:249:37 in metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::size() const

WARNING: ThreadSanitizer: data race (pid=23601)
  Write of size 8 at 0x7b4c00022be8 by thread T76 (mutexes: write M0, write M1):
    #0 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::priv_insert_small_chunk(unsigned char) metall/include/metall/kernel/chunk_directory.hpp:562:30
    #1 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::insert(unsigned char) metall/include/metall/kernel/chunk_directory.hpp:104:27
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_insert_new_small_object_chunk(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:471:38
    #3 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_object_from_global_without_bin_lock(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:402:10
    #4 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_objects_from_global(unsigned char, unsigned long, long*) metall/include/metall/kernel/segment_allocator.hpp:392:13
    #5 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_object(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:375:5
    #6 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:150:31
    #7 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #8 metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::priv_allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:176:51
    #9 metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:120:54
    #10 std::allocator_traits<metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::allocate(metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>&, unsigned long) /include/c++/13.2.0/bits/alloc_traits.h:333:20
    #11 std::_Vector_base<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_allocate(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:378:20
    #12 void std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_range_initialize<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>>(std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::forward_iterator_tag) /include/c++/13.2.0/bits/stl_vector.h:1689:14
    #13 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::vector<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, void>(std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&) /include/c++/13.2.0/bits/stl_vector.h:708:4
    #14 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>> std::__make_from_tuple_impl<std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>, 0ul, 1ul, 2ul>(std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>&&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>) /include/c++/13.2.0/tuple:2309:14
    #15 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>> std::make_from_tuple<std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>>(std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>&&) /include/c++/13.2.0/tuple:2324:14
    #16 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>> std::make_obj_using_allocator<std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>>(metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&) /include/c++/13.2.0/bits/uses_allocator_args.h:232:14

  Previous read of size 8 at 0x7b4c00022be8 by thread T44:
    #0 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::size() const metall/include/metall/kernel/chunk_directory.hpp:249:37
    #1 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::size() const metall/include/metall/kernel/segment_allocator.hpp:237:53
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:153:5
    #3 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #4 metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::priv_allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:176:51
    #5 metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:120:54
    #6 std::allocator_traits<metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::allocate(metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>&, unsigned long) /include/c++/13.2.0/bits/alloc_traits.h:333:20
    #7 std::_Vector_base<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_allocate(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:378:20
    #8 void std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_range_initialize<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>>(std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::forward_iterator_tag) /include/c++/13.2.0/bits/stl_vector.h:1689:14
    #9 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::vector<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, void>(std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&) /include/c++/13.2.0/bits/stl_vector.h:708:4
    #10 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>> std::__make_from_tuple_impl<std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>, 0ul, 1ul, 2ul>(std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>&&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>) /include/c++/13.2.0/tuple:2309:14
    #11 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>> std::make_from_tuple<std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>>(std::tuple<std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>&&) /include/c++/13.2.0/tuple:2324:14
    #12 std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>> std::make_obj_using_allocator<std::vector<unsigned long, metall::stl_allocator<unsigned long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>>(metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&, std::move_iterator<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long>>>>&&) /include/c++/13.2.0/bits/uses_allocator_args.h:232:14

SUMMARY: ThreadSanitizer: data race metall/include/metall/kernel/chunk_directory.hpp:562:30 in metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::priv_insert_small_chunk(unsigned char)

WARNING: ThreadSanitizer: data race (pid=23601)
  Write of size 8 at 0x7b4c00022b60 by thread T135 (mutexes: write M0, write M1):
    #0 metall::kernel::mmap_segment_storage<long, unsigned long>::priv_extend(unsigned long) metall/include/metall/kernel/segment_storage/mmap_segment_storage.hpp:424:30
    #1 metall::kernel::mmap_segment_storage<long, unsigned long>::extend(unsigned long) metall/include/metall/kernel/segment_storage/mmap_segment_storage.hpp:172:12
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_extend_segment(unsigned int, unsigned long) metall/include/metall/kernel/segment_allocator.hpp:502:29
    #3 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_insert_new_small_object_chunk(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:472:10
    #4 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_object_from_global_without_bin_lock(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:402:10
    #5 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_objects_from_global(unsigned char, unsigned long, long*) metall/include/metall/kernel/segment_allocator.hpp:392:13
    #6 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_small_object(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:375:5
    #7 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:150:31
    #8 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #9 metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::priv_allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:176:51
    #10 metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:120:54
    #11 std::allocator_traits<metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::allocate(metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>&, unsigned long) /include/c++/13.2.0/bits/alloc_traits.h:333:20
    #12 std::_Vector_base<long, metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_allocate(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:378:20
    #13 std::vector<long, metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_default_append(unsigned long) /include/c++/13.2.0/bits/vector.tcc:663:34
    #14 std::vector<long, metall::stl_allocator<long, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::resize(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:1013:4

  Previous read of size 8 at 0x7b4c00022b60 by main thread (mutexes: write M2):
    #0 metall::kernel::mmap_segment_storage<long, unsigned long>::size() const metall/include/metall/kernel/segment_storage/mmap_segment_storage.hpp:199:35
    #1 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:96:3
    #2 test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>* metall::kernel::manager_kernel<unsigned int, 2097152ul>::priv_generic_construct<test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>>(boost::interprocess::ipcdetail::char_ptr_holder<char>, unsigned long, bool, boost::interprocess::ipcdetail::in_place_interface&) metall/include/metall/kernel/manager_kernel_impl.ipp:834:11
    #3 test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>* metall::kernel::manager_kernel<unsigned int, 2097152ul>::generic_construct<test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>>(boost::interprocess::ipcdetail::char_ptr_holder<char>, unsigned long, bool, bool, boost::interprocess::ipcdetail::in_place_interface&) metall/include/metall/kernel/manager_kernel_impl.ipp:429:10
    #4 test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>* boost::interprocess::ipcdetail::named_proxy<metall::kernel::manager_kernel<unsigned int, 2097152ul>, test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, false>::operator()<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>(metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&) const boost/interprocess/detail/named_proxy.hpp:132:10

SUMMARY: ThreadSanitizer: data race metall/include/metall/kernel/segment_storage/mmap_segment_storage.hpp:424:30 in metall::kernel::mmap_segment_storage<long, unsigned long>::priv_extend(unsigned long)

WARNING: ThreadSanitizer: data race (pid=23601)
  Write of size 8 at 0x7b4c00025ce8 by thread T78 (mutexes: write M0):
    #0 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::priv_insert_large_chunk(unsigned char) metall/include/metall/kernel/chunk_directory.hpp:605:30
    #1 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::insert(unsigned char) metall/include/metall/kernel/chunk_directory.hpp:106:27
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::priv_allocate_large_object(unsigned char) metall/include/metall/kernel/segment_allocator.hpp:483:58
    #3 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:151:31
    #4 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #5 metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::priv_allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:176:51
    #6 metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>::allocate(unsigned long) const metall/include/metall/stl_allocator.hpp:120:54
    #7 std::allocator_traits<metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::allocate(metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>&, unsigned long) /include/c++/13.2.0/bits/alloc_traits.h:333:20
    #8 std::_Vector_base<float, metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_allocate(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:378:20
    #9 std::vector<float, metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::_M_default_append(unsigned long) /include/c++/13.2.0/bits/vector.tcc:663:34
    #10 std::vector<float, metall::stl_allocator<float, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::resize(unsigned long) /include/c++/13.2.0/bits/stl_vector.h:1013:4

  Previous read of size 8 at 0x7b4c00025ce8 by main thread (mutexes: write M1):
    #0 metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::size() const metall/include/metall/kernel/chunk_directory.hpp:249:37
    #1 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::size() const metall/include/metall/kernel/segment_allocator.hpp:237:53
    #2 metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 281474976710656ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::allocate(unsigned long) metall/include/metall/kernel/segment_allocator.hpp:153:5
    #3 metall::kernel::manager_kernel<unsigned int, 2097152ul>::allocate(unsigned long) metall/include/metall/kernel/manager_kernel_impl.ipp:91:50
    #4 test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>* metall::kernel::manager_kernel<unsigned int, 2097152ul>::priv_generic_construct<test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>>(boost::interprocess::ipcdetail::char_ptr_holder<char>, unsigned long, bool, boost::interprocess::ipcdetail::in_place_interface&) metall/include/metall/kernel/manager_kernel_impl.ipp:834:11
    #5 test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>* metall::kernel::manager_kernel<unsigned int, 2097152ul>::generic_construct<test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>>(boost::interprocess::ipcdetail::char_ptr_holder<char>, unsigned long, bool, bool, boost::interprocess::ipcdetail::in_place_interface&) metall/include/metall/kernel/manager_kernel_impl.ipp:429:10
    #6 test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>* boost::interprocess::ipcdetail::named_proxy<metall::kernel::manager_kernel<unsigned int, 2097152ul>, test_object<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>>>, false>::operator()<metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&>(metall::stl_allocator<std::byte, metall::kernel::manager_kernel<unsigned int, 2097152ul>> const&) const boost/interprocess/detail/named_proxy.hpp:132:10

SUMMARY: ThreadSanitizer: data race metall/include/metall/kernel/chunk_directory.hpp:605:30 in metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::priv_insert_large_chunk(unsigned char)

In a multi-threaded context, I'm concurrently calling construct<T>()(args..., metall::stl_allocator const&) and a lot of std::make_obj_using_allocator<T>(metall::stl_allocator const&, args...). There's also obviously the case where an allocator-aware object (e.g. std::vector) needs to perform some internal allocation.

3 out of 4 of the listed data races include metall::kernel::chunk_directory<unsigned int, 2097152ul, 281474976710656ul>::size() const metall/include/metall/kernel/chunk_directory.hpp:249:37 so m_last_used_chunk_no doesn't seem to be accessed in a thread-safe manner. And for the other data race, m_current_segment_size seems to be the issue.

Using metall with STL container

I followed your (really good) example folder to get a feeling for the library. Everything went well while using boost container or std::vector. However with std::string I wasn't so lucky.

using persistent_string =
  std::basic_string<char, std::char_traits<char>,
    metall::manager::allocator_type<char>>;
int main() {
    metall::manager manager(metall::create_only, "/tmp/datastore");
    auto str = manager.construct<persistent_string>("mystring")(
        "Hello, World!", manager.get_allocator<>());
    std::cout << *str << '\n';
}

Do I need to do something different or are some STL container simply not supported?

Object Cache ASan Error - malloc delete[] mismatch

With the latest master, I get the following:

==133200==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete []) on 0x7fef63e6a800
    #0 0x5587af97c308 in operator delete[](void*) clang/compiler-rt/lib/asan/asan_new_delete.cpp:155:3
    #1 0x5587afc589d3 in std::enable_if<is_convertible<metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul> (*) [], metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul> (*) []>::value, void>::type std::default_delete<metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul> []>::operator()<metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul>>(metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul>*) const /include/c++/13.2.0/bits/unique_ptr.h:140:4
    #2 0x5587afc589d3 in std::unique_ptr<metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul> [], std::default_delete<metall::kernel::obcdetail::cache_container<long, unsigned char, 52ul, 2048ul> []>>::~unique_ptr() /include/c++/13.2.0/bits/unique_ptr.h:674:4
    #3 0x5587afc589d3 in metall::kernel::object_cache<unsigned long, long, metall::kernel::bin_number_manager<2097152ul, 140737488355328ul>, metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 140737488355328ul, metall::kernel::mmap_segment_storage<long, unsigned long>>>::~object_cache() /metall/include/metall/kernel/object_cache.hpp:466:36
    #4 0x5587afc589d3 in metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 140737488355328ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::~segment_allocator() /metall/include/metall/kernel/segment_allocator.hpp:131:41
    #5 0x5587afc584d5 in metall::kernel::manager_kernel<unsigned int, 2097152ul>::~manager_kernel() /metall/include/metall/kernel/manager_kernel_impl.ipp:36:1
    #6 0x5587afc0702f in std::default_delete<metall::kernel::manager_kernel<unsigned int, 2097152ul>>::operator()(metall::kernel::manager_kernel<unsigned int, 2097152ul>*) const /include/c++/13.2.0/bits/unique_ptr.h:99:2
    #7 0x5587afc0702f in std::unique_ptr<metall::kernel::manager_kernel<unsigned int, 2097152ul>, std::default_delete<metall::kernel::manager_kernel<unsigned int, 2097152ul>>>::~unique_ptr() /include/c++/13.2.0/bits/unique_ptr.h:404:4
    #8 0x5587afc0702f in metall::basic_manager<unsigned int, 2097152ul>::~basic_manager() /metall/include/metall/basic_manager.hpp:176:37

0x7fef63e6a800 is located 0 bytes inside of 289636352-byte region [0x7fef63e6a800,0x7fef752a2800)
allocated by thread T6 here:
    #0 0x5587af933cf7 in malloc clang/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x5587afc96b1b in metall::kernel::object_cache<unsigned long, long, metall::kernel::bin_number_manager<2097152ul, 140737488355328ul>, metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 140737488355328ul, metall::kernel::mmap_segment_storage<long, unsigned long>>>::priv_allocate_cache() /metall/include/metall/kernel/object_cache.hpp:584:23
    #2 0x5587afc94f18 in metall::kernel::object_cache<unsigned long, long, metall::kernel::bin_number_manager<2097152ul, 140737488355328ul>, metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 140737488355328ul, metall::kernel::mmap_segment_storage<long, unsigned long>>>::object_cache() /metall/include/metall/kernel/object_cache.hpp:463:5
    #3 0x5587afc94f18 in metall::kernel::segment_allocator<unsigned int, unsigned long, long, 2097152ul, 140737488355328ul, metall::kernel::mmap_segment_storage<long, unsigned long>>::segment_allocator(metall::kernel::mmap_segment_storage<long, unsigned long>*) /metall/include/metall/kernel/segment_allocator.hpp:117:9
    #4 0x5587afc947c0 in metall::kernel::manager_kernel<unsigned int, 2097152ul>::manager_kernel() /metall/include/metall/kernel/manager_kernel_impl.ipp:19:7
    #5 0x5587afc4fb03 in std::__detail::_MakeUniq<metall::kernel::manager_kernel<unsigned int, 2097152ul>>::__single_object std::make_unique<metall::kernel::manager_kernel<unsigned int, 2097152ul>>() /include/c++/13.2.0/bits/unique_ptr.h:1070:34
    #6 0x5587afc4fb03 in metall::basic_manager<unsigned int, 2097152ul>::basic_manager(metall::open_read_only_t, char const*) /metall/include/metall/basic_manager.hpp:135:18

SUMMARY: AddressSanitizer: alloc-dealloc-mismatch clang/compiler-rt/lib/asan/asan_new_delete.cpp:155:3 in operator delete[](void*)

You're not supposed to mix std::malloc with the default unique_ptr deleter, namely for object_cache::m_cache. To fix, change std::unique_ptr<cache_storage_type[]> m_cache to std::unique_ptr<cache_storage_type[], decltype([](cache_storage_type* p) { std::free(p); })> m_cache for example.

How to do performance tuning?

Hi and first of all thx for this great piece of software.

We have just implemented persistence for our RDF graph store tentris (still in a private repo, though) at the DICE research group. It seems to scale well up to 8 million edges. After that insertion performance decreases significantly (factor 10+). With std::allocator, so without any persistence, it scales well up to several billion edges.

So I have some questions regarding performance tuning. 😄

First: You wrote in the metall overview slides on slide 21 that you did:

mmap (VM tuned)
• system mmap + Metall
• Tuned page cache behaviors
• Required privileged access
• System-wide change

Could you share those settings? The difference in the plot seems significant.

Second: The umap backend in the above plot looks most promising. But you mention here in the docs that the umap back end does not give any memory back. Neither DRAM nor storage in the backing file. Is this still up-to-date?
Would it need work in metall or umap or both to address these issues?

And Third: What are the largest memory footprints in GB you handled with metall so far?

Question: check if segment already exists

How would I check if a segment at a certain part exists already?

I cannot find a method for that in metall::basic_manager.

Contributions regarding C Interface

Currently metall's C interface is design around the typical malloc usecase, where you have a single global allocator that does all allocations.

This certainly has its uses, particularly if you want to replace malloc. However some applications might want more fine grained control over metall managers and create multiple on their own. This usecase can currently not be served with the C interface.

We have currently implemented our own metall C interface for that purpose (https://github.com/dice-group/metall-ffi).

So question: Would you be interested in integrating this in some way? Either replacing the current interface or (probably preferred) providing something like this as an alternative to the current interface?

CMake's BOOST_ROOT option does not work with Boost 1.70 and more

A possible solution is:

cmake ../ -DBoost_NO_BOOST_CMAKE=ON -DBOOST_ROOT=/path/to/boost/1.70

See boostorg/boost_install#12

flush/snapshot only certain key

One nice to have feature would be to be able to flush or snapshot only a certain key.
We have a situation in a multithreaded application where threads may be writing to a container within a key which is going through a resize. If another thread is doing a flush at the same time, it would cause data corruption or a runtime error. It would be nice to have a key level flush so that we don't need global level locks on the manager.

boost::json::string_view in Boost v1.79.0

From v1.79, boost::json::string_view is boost/core/detail/string_view.hpp instead of boost/utility/string_view.hpp.

The difference between the two implementations are described here:
https://groups.google.com/g/boost-developers-archive/c/8z2EQAPAIVg

This change may cause some build errors in the Metall container JSON applications.
We need to investigate that.

The following code is compilable with Boost v1.78 but not with v1.79:

#include <boost/json/src.hpp>
#include <boost/container/string.hpp>

int main() {
  boost::container::string key("key");
  boost::json::object obj;
  obj[key] = 10; // obj[key] is an issue
  return 0;
}

It looks that there is no implicit conversion from boost::container::string to boost::json::string_view in v1.79.

Could remove select_on_container_copy_construction()

metall/include/metall/stl_allocator.hpp

Line 139 in eec3d07

stl_allocator select_on_container_copy_construction(const stl_allocator &a) {

Datastore is "inconsistent" when the program ends, while the snapshot appears to be working for "complex map"

When a "complex map" is created and populated with data, the data store on the disk appears to be "incomplete" with the following error message:

include/metall/kernel/manager_kernel_impl.ipp at line 941 --- Inconsistent data store — it was not closed properly and might have been collapsed.

If the snapshot is created, it is working properly
This does not happen for "simpler" value types

Ubuntu 22.04.1 LTS
boost 1.74
g++ 17

I attached the code which produces the error

main.cc.zip

Assertion triggered on small allocations with custom chunk size

Hi, we've ran into a problem an wanted some guidance.
So, the metall manager has this k_chunk_size template parameter which is set to 1 << 21 by default.

If we set it to e.g. 1 << 28 metall fails to allocate small objects (as far as I can tell everything below 2*sizeof(void *)). The following assertion is triggered inside metall:

multilayer_bitset.hpp:481: std::size_t metall::kernel::multilayer_bitset::num_all_blocks(std::size_t) const: Assertion `idx < mlbs::k_num_layers_table.size()' failed.

So I'm basically just here to ask if this is expected behaviour or if this is a bug (an explaination of what the parameter does exactly would also be appreciated) 😅

Set default boost version to boost-1.77.0

Minor Issue:

Spack picks up the default boost dependency version from here.
Setting default boost version to boost-1.77.0 avoids all pragma deprecation warnings and gives a clean compilation.
update needed at metall.readthedocs.io, readme, and gitlab-ci.yml.

Metall requires GCC 8.3 or more due to a bug in std::filesystem::create_directories

GCC Bug 87846:
std::filesystem::create_directories with a path with a trailing slash does not create any directory.

Contributions in logging and file handling

We've done overhauls in the two areas logging and file handling, and would like to contribute them back:

• logging: make it more stream lined and easier to integrate by using format strings and allowing the application that uses metall to define the function that actually writes the log.
• file handling: use linux sys calls for file operations on Linux and make proper use of std::fs making it non-optional. Using sys calls instead of external progamms like cp for file handling allows you to use statically linked metall applications in any Linux environment, even in scratch docker images.

Both are breaking changes because interfaces are affected and we had to raise the C++ standard to C++20.

Would you be interested to integrate them?

Cannot immediately see a directory created by another process on NFS?

Suggestion: Make fallback_allocator::stateful_allocatorator_available() public

metall/include/metall/container/fallback_allocator.hpp

Lines 217 to 219 in 1f6c027

	auto priv_stateful_allocatorator_available() const {
	return !!(m_stateful_allocatorator.get_pointer_to_manager_kernel());
	}

We use the fallback_allocator in an algorithm that behaves differently based on weather the stateful allocator is used or not. ATM, we just re-implement the private_stateful_allocatorator_available() in our code.

I would suggest to make a public stateful_allocatorator_available(). Shouldn't hurt because the interfaces to calculate the result are public anyway and might come in handy for others.

PS: Is allocatorator a pun or a typo?

Build error with CMake Latest Release (3.16.4)

Uses Allocator Properly in JSON Container

For example, use select_on_container_copy_construction propagate_on_container_copy_assignment, and propagate_on_container_move_assignment in the container.

cuda compiler complains about defaulted copy assignment

with a miminal example:

clang++ -x cuda -Wall -Wextra -Ofast -Wfatal-errors -Wpedantic -Wmove -std=c++17 --cuda-gpu-arch=sm_60 
In file included from ./llnl_metall.cpp:10:
In file included from github.com/LLNL/metall.git/include/metall/metall.hpp:9:
In file included from /github.com/LLNL/metall.git/include/metall/basic_manager.hpp:16:
In file included from /github.com/LLNL/metall.git/include/metall/kernel/manager_kernel.hpp:21:
In file included from /github.com/LLNL/metall.git/include/metall/kernel/segment_allocator.hpp:18:
/github.com/LLNL/metall.git/include/metall/kernel/bin_directory.hpp:85:18: fatal error: the parameter for this explicitly-defaulted copy assignment operator is const, but a member or base requires it to be non-const
  bin_directory &operator=(const bin_directory &) = default;

I don't know if the member is table_type (aka boost::container::vector), it would be strange that it doesn't have an operator=(T const&){...}.

Fails to extend segment beyond 255GB

We have trouble with Metall on some Debian machines. Segment extension fails beyond 255GB. I've put together a minimal example. It loads up to 300GB of 4MB chunks into a std::list using the Metall allocator. Runs with roughly 1,4GB/s on my laptop with a PCIe4 SSD. You can find it here: https://github.com/dice-group/debug_metall

I've attached the binary from the dockerfile here: debug_metall.zip (fully statically linked with musl)

Adding chunk no 32671 ... done. 
130684 MB = 137032105984 Bytes
Adding chunk no 32672 ... /root/.conan/data/metall/0.21/_/_/package/9f447b31527ba03fcf0419f3224954b77c8a66f0/include/metall/detail/file.hpp at line 216 --- open: No file descriptors available
/root/.conan/data/metall/0.21/_/_/package/9f447b31527ba03fcf0419f3224954b77c8a66f0/include/metall/kernel/segment_storage/mmap_segment_storage.hpp at line 417 --- Failed to extend the segment
/root/.conan/data/metall/0.21/_/_/package/9f447b31527ba03fcf0419f3224954b77c8a66f0/include/metall/kernel/segment_allocator.hpp at line 388 --- Failed to extend the segment to 274074697728 bytes
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

It failed on:

• Debian 11, kernel 5.10.0-16-amd64
• Debian 12, kernel 5.18.0-1-amd64

It succeeded on:

• Ubuntu 22.04, kernel 5.15.0-40-generic
• Debian 11, kernel 5.10.0-15-amd64 (relatively fresh installation, HDD)

My first guess would be that we have to change some kernel parameters on the Debian systems. But I don't really have an idea where to start there.
Have you faced a similar issue before?

Slide Suggestions

Just read through the (excellent!) slides in the NVMW tutorial branch. Here are a few small suggestions to make them read more naturally.

Slide 2:
-"one of key challenges" -> "one of the key challenges" or "a key challenge"
-"multiple analytics to the same data" -> "multiple analytics on the same data"

Slide 4:
-"dismantling and assembling large complex data structures are" -> "dismantling and assembling large complex data structures is"
-"transparent accesses to applications" -> "transparent access to applications"

Slide 8:
-"destructor synchronies" -> "destructor synchronizes"

Slide 11:
-"object is points" -> "object is points to"

Slide 13:
-"improve I/O performance" -> "improves I/O performance"

Slide 17:
-"clash" -> "crash"

Slide 18:
-"Snapshot/versioning" -> "Snapshot/Versioning" (capitalization)

Slide 23:
-Maybe make the highlighted text red to emphasize that those points are negative

FYI: A polymorphic allocator for metall

We have a use-case for metall where we need a common type that can either live in mmaped memory or std::allocator memory. Out of the box, that's only achievable with custom wrapper types because metall doesn't allow us to use std::pmr::polymorphic_allocator (because of its use of dynamic polymorphism).

For that, we implemented a static polymorphic allocator that can, e.g., hold a metall allocator (or boost interprocess allocator), or a std::allocator (actually it's a thin wrapper around std::allocator that uses offset ptr).

We provide a short example here: example polymorphic allocator (uses boost interprocess but I guess you get the idea)

I hope you find it useful, too. License is MIT. So, should likely not be an issue to integrate it into your code base if you want.

FYI: metall is available as conan package now

see: https://conan.io/center/metall

I know, you have your own package manager spack at the LLNL. As long as we use it at DICE, I'll update it regularly when you release new versions. PR for the latest v0.21 is already out: conan-io/conan-center-index#11581

O3 compilation gives different result than O1 in simple example (and fails assertion)

[correaa@cuk llnl_metall]$ c++ -std=c++17 -lstdc++fs simple.cpp -I$HOME/soft/metall/include -o simple.x && ./simple.x
5
10
[correaa@cuk llnl_metall]$ c++ -O3 -std=c++17 -lstdc++fs simple.cpp -I$HOME/soft/metall/include -o simple.x && ./simple.x
simple.x: /home/correaa/soft/metall/include/metall/v0/kernel/manager_kernel.hpp:196: void* metall::v0::kernel::manager_kernel<chunk_no_type, k_chunk_size, allocator_type>::allocate(metall::v0::kernel::manager_kernel<chunk_no_type, k_chunk_size, allocator_type>::size_type) [with chunk_no_type = unsigned int; long unsigned int k_chunk_size = 2097152; allocator_type = std::allocator<char>; metall::v0::kernel::manager_kernel<chunk_no_type, k_chunk_size, allocator_type>::size_type = long unsigned int]: Assertion `priv_initialized()' failed.
Aborted (core dumped)
[correaa@cuk llnl_metall]$ head simple.cpp
// Copyright 2019 Lawrence Livermore National Security, LLC and other Metall Project Developers.
// See the top-level COPYRIGHT file for details.
//
// SPDX-License-Identifier: (Apache-2.0 OR MIT)

#include <iostream>
#include <boost/container/vector.hpp>

#include <metall/metall.hpp> // Only one header file is required to be included
...

Trying to compile with clang gives the same error as gcc -O3,

$ clang++ -std=c++17 -lstdc++fs simple.cpp -I$HOME/soft/metall/include -o simple.x && ./simple.x
simple.x: /home/correaa/soft/metall/include/metall/v0/kernel/manager_kernel.hpp:196: void *metall::v0::kernel::manager_kernel<unsigned int, 2097152, std::allocator<char> >::allocate(const metall::v0::kernel::manager_kernel::size_type) [chunk_no_type = unsigned int, k_chunk_size = 2097152, allocator_type = std::allocator<char>]: Assertion `priv_initialized()' failed.
Aborted (core dumped)

noexcept specification in move constructor is overconstrained or contradictory

I think removing noexcept will make it work and still deduce noexcept by itself if the base classes have noexcept move constructors.

$ clang++ -std=c++17 -lstdc++fs simple.cpp -I$HOME/soft/metall/include -o simple.x && ./simple.x
In file included from simple.cpp:9:
In file included from /home/correaa/soft/metall/include/metall/metall.hpp:9:
In file included from /home/correaa/soft/metall/include/metall/v0/manager_v0.hpp:13:
/home/correaa/soft/metall/include/metall/v0/kernel/manager_kernel.hpp:124:3: error: exception specification of explicitly defaulted move constructor does not match the calculated one
  manager_kernel(manager_kernel &&) noexcept = default;
  ^
/home/correaa/soft/metall/include/metall/v0/manager_v0.hpp:37:33: note: in instantiation of template class 'metall::v0::kernel::manager_kernel<unsigned int, 2097152, std::allocator<char> >' requested here
  using void_pointer = typename kernel_type::void_pointer;
                                ^
/home/correaa/soft/metall/include/metall/detail/base_manager.hpp:27:33: note: in instantiation of template class 'metall::v0::detail::manager_type_holder<unsigned int, 2097152, std::allocator<char> >' requested
      here
  using void_pointer = typename type_holder::void_pointer;
                                ^
/home/correaa/soft/metall/include/metall/v0/manager_v0.hpp:64:27: note: in instantiation of template class 'metall::detail::base_manager<metall::v0::manager_v0<unsigned int, 2097152, std::allocator<char> >,
      metall::v0::detail::manager_type_holder<unsigned int, 2097152, std::allocator<char> > >' requested here
class manager_v0 : public metall::detail::base_manager<manager_v0<chunk_no_type, k_chunk_size, kernel_allocator_type>,
                          ^
simple.cpp:13:56: note: in instantiation of template class 'metall::v0::manager_v0<unsigned int, 2097152, std::allocator<char> >' requested here
using vector_t = boost::container::vector<int, metall::manager::allocator_type<int>>;
                                                       ^
In file included from simple.cpp:9:
In file included from /home/correaa/soft/metall/include/metall/metall.hpp:9:
In file included from /home/correaa/soft/metall/include/metall/v0/manager_v0.hpp:13:
/home/correaa/soft/metall/include/metall/v0/kernel/manager_kernel.hpp:125:19: error: exception specification of explicitly defaulted move assignment operator does not match the calculated one
  manager_kernel &operator=(manager_kernel &&) noexcept = default;
                  ^
In file included from simple.cpp:9:
In file included from /home/correaa/soft/metall/include/metall/metall.hpp:9:
/home/correaa/soft/metall/include/metall/v0/manager_v0.hpp:125:3: error: exception specification of explicitly defaulted move constructor does not match the calculated one
  manager_v0(manager_v0 &&) noexcept = default;
  ^
simple.cpp:13:56: note: in instantiation of template class 'metall::v0::manager_v0<unsigned int, 2097152, std::allocator<char> >' requested here
using vector_t = boost::container::vector<int, metall::manager::allocator_type<int>>;
                                                       ^
In file included from simple.cpp:9:
In file included from /home/correaa/soft/metall/include/metall/metall.hpp:9:
/home/correaa/soft/metall/include/metall/v0/manager_v0.hpp:127:15: error: exception specification of explicitly defaulted move assignment operator does not match the calculated one
  manager_v0 &operator=(manager_v0 &&) noexcept = default;
              ^
4 errors generated.

Some warnings in with -Wall

metall/include/metall/v0/kernel/manager_kernel.hpp:773:34: warning: comparison of integer expressions of different signedness: ‘long int’ and ‘long unsigned int’ [-Wsign-compare]
  773 |       assert(page_no * page_size < m_segment_storage.size());
      |              ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

metall/include/metall/v0/kernel/manager_kernel.hpp:773:34: warning: comparison of integer expressions of different signedness: ‘long int’ and ‘long unsigned int’ [-Wsign-compare]
  773 |       assert(page_no * page_size < m_segment_storage.size());
      |              ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

metall/include/metall/v0/kernel/manager_kernel.hpp:773:34: warning: comparison of integer expressions of different signedness: ‘long int’ and ‘long unsigned int’ [-Wsign-compare]
  773 |       assert(page_no * page_size < m_segment_storage.size());
      |              ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

metall/include/metall/v0/kernel/manager_kernel.hpp:773:34: warning: comparison of integer expressions of different signedness: ‘long int’ and ‘long unsigned int’ [-Wsign-compare]
  773 |       assert(page_no * page_size < m_segment_storage.size());
      |              ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

metall/include/metall/v0/kernel/manager_kernel.hpp:773:34: warning: comparison of integer expressions of different signedness: ‘long int’ and ‘long unsigned int’ [-Wsign-compare]
  773 |       assert(page_no * page_size < m_segment_storage.size());
      |              ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~