andypern / metall

Meta allocator for persistent memory

Metall (memory allocator for persistent memory)

• Provides rich memory allocation interfaces for C++ applications that use persistent memory devices to persistently store heap data on such devices.

• Creates files in persistent memory and maps them into virtual memory space so that users can access the mapped region just as normal memory regions allocated in DRAM.

• Actual persistent memory hardware could be any non-volatile memory (NVM) with filesytem support.

• To provide persistent memory allocation, Metall employs concepts and APIs developed by Boost.Interprocess.

• Supports multi-thread

• Also provides a space-efficient snapshot/versioning, leveraging reflink copy mechanism in filesystem. In case reflink is not supported, Metall automatically falls back to regular copy.

• See details: Metall overview slides.

Getting Started

Required to Build Metall

• GCC 8.1 or more (8.3 or more is recommended due to early implementation of the Filesystem library).
• Boost C++ Libraries 1.64 or more (build is not required; needs only their header files).

Install and Build

Metall consists of only header files and requires some header files in Boost C++ Libraries.

Metall is available at: https://github.com/LLNL/metall.

All core files exist under metall/include/metall/.

To build your program with Metall, all you have to do is just setting include paths such as '-I' or CPLUS_INCLUDE_PATH.

For example,

g++ -std=c++17 your_program.cpp -lstdc++fs -I/path/to/metall/include -I/path/to/boost/include

Note that one might need to link stdc++fs library to use the Filesystem library in C++17.

Install Using Spack

Metall package is also available on Spack.

To install Metall using Spack, type spack install metall.

As Metall requires Boost C++ Libraries, Spack also installs a proper version of Boost C++ Libraries automatically, if needed.

As Spack's load command configures environmental values properly, one can avoid specifying include paths to build a program with Metall. For instance:

spack load metall
g++ -std=c++17 your_program.cpp -lstdc++fs 

Build Examples Using CMake

This repository contains some example programs under example directory. One can use CMake to build the examples. For more details, see a page here.

Unofficial Support For Clang

Clang can be used instead of GCC to build Metall. However, we haven't tested it intensively. Also, Boost C++ Libraries 1.69 or more may be required if one wants to build Metall with Clang + CUDA.

Documentation

Full documentation is available.

Generate API document using Doxygen

A Doxygen configuration file is here.

To generate API document:

cd metall
mkdir build_doc
cd build_doc
doxygen ../docs/Doxyfile.in

Publication

Metall: A Persistent Memory Allocator Enabling Graph Processing

Paper PDF

IEEE Xplore

About

Authors

• Keita Iwabuchi (kiwabuchi at llnl dot gov)
• Roger A Pearce (rpearce at llnl dot gov)
• Maya B Gokhale (gokhale2 at llnl dot gov).

License

Metall is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.

All new contributions must be made under both the MIT and Apache-2.0 licenses.

See LICENSE-MIT, LICENSE-APACHE, NOTICE, and COPYRIGHT for details.

SPDX-License-Identifier: (Apache-2.0 OR MIT)

Release

LLNL-CODE-768617