yhirokawa-ccs / cbsl

CBSL: Compressed Binary Serialization Library

CBSL: Compressed Binary Serialization Library

What's this?

The library provides the binary serialization with the compression by Zstandard. A motivation of the library is to implement the data-compressed checkpoint/restart, which is well-known technique to recover computer failures in high-performance computing. This library aims to simple and lightweight access for users.

The library support C99 or later, and Fortran 2008 or later: perhaps we use the features are all supported by major compilers.

How to build and test

The library uses CMake version 3.3.x or later.

$ cmake --version
cmake version 3.14.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ mkdir build && cd build
$ cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/install/path ..
$ make
$ make test
$ make install

Do you need help to install zstd package?

We can build and install zstd automatically in the build process. Please pass -D INSTALL_ZSTD=on to cmake, we will install zstd package where CMAKE_INSTALL_PREFIX directory before building the library.

$ cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/install/path -D INSTALL_ZSTD=on ..
$ make
$ make test
$ make install

Test environments

1. CMake version 3.14.3
2. GCC version 4.8.5
3. Zstandard version 1.4.0
4. CentOS Linux release 7.5.1804 (Core)

Benchmark

We provide the library performance benchmark with best and worst case. Please be reminded that the benchmark results not indicate the performance of Zstandard, these measure the overhead (use cost) of this library.

1. Best case : all data is zero filled (A compression ratio achieves up to 99%)
2. Worst case : data is generated by rand() (A compression ratio is lower than 1%)

benchmark target executes the benchmarks.

...
$ make benchmark
min data size = 262144.00 [B]
max data size = 134217728.00 [B]
data is zero filled (maximum compression)
<write data [Byte]> <time [seconds]> <speed [MiB/sec]>
262144    0.002397    109.360615
524288    0.002887    181.572393
1048576    0.003440    304.827177
...

Use zstd command

A compressed file by the library can be decompressed by zstd command.

License

Copyright 2019 Yuta Hirokawa (University of Tsukuba, Japan)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Future work

1. API error code
2. Refectoring tests

NOTE

Unsupported binary compatibility

The library does not support the binary compatibility and endianness conversion, which is required on the communication across machine. If you want it, please consider using other serialization libraries such as MessagePack.