llnl / zfp

Compressed numerical arrays that support high-speed random access

ZFP

zfp is a compressed format for representing multidimensional floating-point and integer arrays. zfp provides compressed-array classes that support high throughput read and write random access to individual array elements. zfp also supports serial and parallel (OpenMP and CUDA) compression of whole arrays, e.g., for applications that read and write large data sets to and from disk.

zfp uses lossy but optionally error-bounded compression to achieve high compression ratios. Bit-for-bit lossless compression is also possible through one of zfp's compression modes. zfp works best for 2D, 3D, and 4D arrays that exhibit spatial correlation, such as continuous fields from physics simulations, natural images, regularly sampled terrain surfaces, etc. zfp compression of 1D arrays is possible but generally discouraged.

zfp is freely available as open source and is distributed under a BSD license. zfp is primarily written in C and C++ but also includes Python and Fortran bindings. zfp conforms to various language standards, including C89, C99, C11, C++98, C++11, and C++14, and is supported on Linux, macOS, and Windows.

Quick Start

To download zfp, type:

git clone https://github.com/LLNL/zfp.git

zfp may be built using either CMake or GNU make. To use CMake, type:

cd zfp
mkdir build
cd build
cmake ..
cmake --build . --config Release
ctest

This builds the zfp library in the build/lib directory and the zfp command-line executable in the build/bin directory. It then runs the regression tests. The full test suite may be run by enabling the BUILD_TESTING_FULL CMake option during the build step.

zfp may also be built using GNU make:

cd zfp
make
make test

Note: GNU builds are less flexible and do not support all available features, e.g., CUDA support.

For further configuration and build instructions, please consult the documentation. For examples of how to call the C library and use the C++ array classes, see the examples section.

Documentation

Full HTML documentation is available online. A PDF version is also available.

Further information on the zfp software is included in these files:

• Change log: see CHANGELOG.md.
• Support and additional resources: see SUPPORT.md.
• Code contributions: see CONTRIBUTING.md.

Authors

zfp was originally developed by Peter Lindstrom at Lawrence Livermore National Laboratory. Please see the Contributors Page for a full list of contributors.

Citing zfp

If you use zfp for scholarly research, please cite this paper:

• Peter Lindstrom. Fixed-Rate Compressed Floating-Point Arrays. IEEE Transactions on Visualization and Computer Graphics, 20(12):2674-2683, December 2014. doi:10.1109/TVCG.2014.2346458.

The algorithm implemented in the current version of zfp is described in the documentation and in the following paper:

• James Diffenderfer, Alyson Fox, Jeffrey Hittinger, Geoffrey Sanders, Peter Lindstrom. Error Analysis of ZFP Compression for Floating-Point Data. SIAM Journal on Scientific Computing, 41(3):A1867-A1898, June 2019. doi:10.1137/18M1168832.

License

zfp is distributed under the terms of the BSD 3-Clause license. See LICENSE and NOTICE for details.

SPDX-License-Identifier: BSD-3-Clause

LLNL-CODE-663824

zfp's Issues

zfp python wrapper problems with "big" multidimensional arrays

I experience a problem when compressing some 3d numpy array (using the implementation discussed in #29, using ebe63fa from branch feature/py-bindings).

When compressing an array that has a shape of (4, 4, s), where s > 2^16, the shape of the decompressed array is not correct anymore.

How to reproduce:

shp = (4, 4, 65537)
ar = np.zeros(shp)
_comprsd_3d = zfp.compress_numpy(ar, tolerance=0.2)
decompressed = zfp.decompress_numpy(_comprsd_3d)
print(decompressed.shape) # --> prints (4, 4, 1)

When using a shape of (16, 65537), this does not happen, so it does not seem to be related to the total size of the array.

Expected

Shape of numpy array is the same after decompression for any size. Or, if that is not possible, some kind of error during compression (or at least decompression).

Missing default constructor for zfparray view

In certain cases having a default constructor for the zfparray view class is helpful. For example, when the view is part of another object.

Wrong use of memset

memset in two files had the wrong order of parameters. I uploaded a fix in PR #83

endtoend test fail on Cygwin

endtoend tests fail on Cygwin because CMake no longer defines WIN32 under Cygwin.
To fix this, we have two options:

1. Check if __CYGWIN__ is defined
2. Use __unix__ instead of __linux__. The Unix macro seems to be defined on Cygwin

I uploaded a fix on #31

Missing include to zfp.h in CUDA build

When building the CUDA port in 0.5.4, a compilation error is generated due to missing zfp.h.
See:

-- Generating dependency file: /home/mbamakhr/repo/zfp/build/src/CMakeFiles/zfp.dir/cuda_zfp/zfp_generated_cuZFP.cu.o.NVCC-depend
/hpc/shared/eb_apps/rhel6/cuda/8.0.44-intel-2016a/bin/nvcc -M -D__CUDACC__ /home/mbamakhr/repo/zfp/src/cuda_zfp/cuZFP.cu -o /home/mbamakhr/repo/zfp/build/src/CMakeFiles/zfp.dir/cuda_zfp/zfp_generated_cuZFP.cu.o.NVCC-depend -ccbin /usr/bin/cc -m64 -Xcompiler ,"-fPIC","-O3","-DNDEBUG" -DNVCC -I/hpc/shared/eb_apps/rhel6/cuda/8.0.44-intel-2016a/include
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
In file included from /home/mbamakhr/repo/zfp/src/cuda_zfp/cuZFP.cu:3:
/home/mbamakhr/repo/zfp/src/cuda_zfp/cuZFP.h:4:17: error: zfp.h: No such file or directory
CMake Error at zfp_generated_cuZFP.cu.o.cmake:207 (message):
Error generating
/home/mbamakhr/repo/zfp/build/src/CMakeFiles/zfp.dir/cuda_zfp/./zfp_generated_cuZFP.cu.o
make[2]: *** [src/CMakeFiles/zfp.dir/cuda_zfp/zfp_generated_cuZFP.cu.o] Error 1
make[2]: Leaving directory /home/mbamakhr/repo/zfp/build' make[1]: *** [src/CMakeFiles/zfp.dir/all] Error 2 make[1]: Leaving directory /home/mbamakhr/repo/zfp/build'
make: *** [all] Error 2

I uploaded a pull request to fix this in: #24

CUDA GPU version of zfp

The README mentions a CUDA GPU version of zfp in the works. Is there a public version that can be made available of this CUDA GPU version sometime soon?

Thank you

ZFP-0.5.5 doesn't adhere to user-specified absolute error

ZFP significantly overshoots the specified absolute error for some tests.

An easy way to repro the issue is to use the following test data:
http://www.cs.txstate.edu/~burtscher/research/datasets/FPsingle/msg_bt.sp.spdp
You will have to decompress it using the SPDP decompressor.

To compress I used the compression utility but the bug can be reproduced also using the API.
Command to compress:
./zfp -i msg_bt.sp -z msg_bt.sp.zfp -f -1 33298679 -a 1e-4 -h

Command to decompress:
./zfp -z msg_bt.sp.zfp -o msg_bt.sp.new -h

I wrote the following script to compare two files containing binary floating point data:

import sys
import struct
import math

def printHelp():
    print("Example run:")
    print("python3 verify.py file1 file2 1e-6")

if len(sys.argv) != 4:
    printHelp()
    exit(1)

fname1 = sys.argv[1]
fname2 = sys.argv[2]
absError = float(sys.argv[3])

f1 = open(fname1, "br")
f2 = open(fname2, "br")

u = 0 
while True:
    a = f1.read(4)
    b = f2.read(4)
    if len(a) == len(b) == 0:
        exit()
    if len(a) != len(b) or len(a) != 4:
        print("Couldn't read sufficient amount of bytes from either stream. {}: {} bytes, {}: {} bytes.".format(fname1, len(a), fname2, len(b)))
    a = struct.unpack("f", a)[0]
    b = struct.unpack("f", b)[0]
    dif = math.fabs(a - b)
    if dif > absError:
        print("Error at index {}".format(u))
        print("Value from {}: {}. Value from {}: {}. Delta: {}. Max allowed error: {}".format(fname1, a, fname2, b, dif, absError))
    u += 1

The script outputs the following:

Error at index 8864436
Value from ../../dataset/32bit/msg_bt.sp: 24161450.0. Value from msg_bt.sp.new: 24161424.0. Delta: 26.0. Max allowed error: 0.0001
Error at index 8893252
Value from ../../dataset/32bit/msg_bt.sp: 3437.55908203125. Value from msg_bt.sp.new: 3437.558837890625. Delta: 0.000244140625. Max allowed error: 0.0001
Error at index 8893255
Value from ../../dataset/32bit/msg_bt.sp: 2046.7005615234375. Value from msg_bt.sp.new: 2046.700439453125. Delta: 0.0001220703125. Max allowed error: 0.0001

In particular, the error of 26.0 is significant.
I checked with other test files and for certain inputs the error is beyond the user-specified threshold and occurs way more often than only three times as in this example.

Unchecked mallocs in parallel.c

The function compress_init_par in the file parallel.c contains the following unchecked mallocs

zfp/src/share/parallel.c

Line 51 in 25a554e

bs = (bitstream**)malloc(chunks * sizeof(bitstream*));

zfp/src/share/parallel.c

Line 54 in 25a554e

void* buffer = copy ? malloc(size) : (uchar*)stream_data(stream->stream) + stream_size(stream->stream) + block * stream->maxbits / CHAR_BIT;

I uploaded a fix in #40

CXXLAGS?

cfp/src/Makefile contains the following link rule:

$(LIBDIR)/libcfp.so: $(OBJECTS)
      mkdir -p $(LIBDIR)
      $(CXX) $(CXXLAGS) -shared $(SOFLAGS) $^ -o $@

Is CXXLAGS correct, or should it be CXXFLAGS?

cfp_ or cfp. in docs

In the docs where cfp interface is described, I think there are numerous instances of use of cfp_arrayXx that should really be cfp.arrayXx (e.g. dot and not underscore). I would have submitted PR but honestly wasn't absolutely sure either.

Use https:// instead of git://

In many companies, the git:// protocol is blocked by corporate firewall. To circumvent that, use http:// instead of git://

I uploaded a fix in #33

Compare with grib2 format

grib2 is a format commonly used by NWS for compressing spatially correlated weather forecast data, among other uses. I am curious how grib2 and zfp would compare both in theory and in practice, and if a hybrid would make sense. Thanks.

-h option not working for decompression

zfp -i datafile -z datafile.zfp <options> works perfectly
zfp -z datafile.zfp -o datafile.raw <options> works perfectly
zfp -i datafile -z datafile.zfp -o datafile.raw <options> works perfectly

but,
zfp -i datafile -z datafile.zfp <options> -h works
zfp -z datafile.zfp -o datafile.raw -h doesn't work and reports array size must be nonzero

What is the correct way to use -h option in order to avoid typing <options> in decompression?

PS: datafile is around several MB in byte size

Persisting ZFP compressed arrays

Thought'd be better to capture discussion here instead of email ;)

I don't have a lot of experience designing C++ objects or their interfaces but I am wondering...from the perspective of a ZFP compressed array, what is the proper/ideal interface for persisting it?

Here is an idea I have with the goal of making it easy to integrate ZFP's compressed array implementation with HDF5 without having to introduce HDF5 dependencies to ZFP.

• Add static, private members to these classes, _load_fptr and _store_fptr which are function pointers to two global functions to load or store a ZFP array.
• Add static methods load and store to ZFP compressed array base class. Something like...

int store(char const *path=0, void *storage_id=0)
{
    if (!_store_fptr) return 1;
    return (*_store_ftpr)(path, storage_id);
}
int load(char const *path, void *storage_id=0)
{
    if (!_load_fptr) return 1;
    return (*_load_fptr)(path, storage_id);
}

• By default, the above function pointers point to two default methods implemented in ZFP for writing ZFP arrays to files in the file system (e.g. just write the zfp stream to a file). Something like...

int store(char const *path, void *storage_id)
{
    char const *name = path ? path : <make a name up>;
    FILE *f = fopen(name, "w");
    if (!f) return 1;
    if (fwrite(this->compressed_data(), 1 this->compressed_size, file) =this->compressed_size())
    { fclose(f); return 1;}
    fclose(f);
    return 0;
}

• Define a global function that allows a caller to update the function pointers, above, to those of their own choosing (e.g. ones that will read or write ZFP arrays to an HDF5 file). Something like...

static void set_persistents_methods((*load)(char const *, void*), (*store)(char const *, void*));

• storage_id is interpreted as needed by the implementation of a given load/store pair. For HDF5 it would be a hid_t loc_id
• Now, an application can include a library that supports writing ZFP array classes with HDF5 but the ZFP classes themselves are agnostic to this.
• This design allows caller to wholly switch the persistents methods of all instances, those in existence now or created in the future, in a single, simple call. But, maybe this is too heavy handed? Ok, so do it both ways...allow the ability to set these global properties, but also allow any ZFP array instance to override the global settings. New instances that are constructed inherit initially the globally defined behavior but caller can override.

Link error for endtoend tests due to missing rt library

endtoend tests fail to link giving the following error:

[ 87%] Linking C executable ../../../bin/testZfpOmp3dInt64
/home/mbamakhr/repo/zfp/tests/src/endtoend/zfpEndtoendBase.c:469: error: undefined reference to 'clock_gettime'
/home/mbamakhr/repo/zfp/tests/src/endtoend/zfpEndtoendBase.c:487: error: undefined reference to 'clock_gettime'
/home/mbamakhr/repo/zfp/tests/src/endtoend/zfpEndtoendBase.c:469: error: undefined reference to 'clock_gettime'
/home/mbamakhr/repo/zfp/tests/src/endtoend/zfpEndtoendBase.c:487: error: undefined reference to 'clock_gettime'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/testZfpSerial2dFloat] Error 1
make[1]: *** [tests/src/endtoend/CMakeFiles/testZfpSerial2dFloat.dir/all] Error 2

Adding rt to target_link_libraries in the zfp_add_test function in tests/src/endtoend/CMakeLists.txt fixes the problem.

Unused stores

This is just a FYI issue.

clang-tidy is finding three unused assignments in ZFP. All three are the last shift-assign of meta in zfp_field_set_metadata's cases:

https://github.com/LLNL/zfp/blob/master/src/zfp.c#L299
https://github.com/LLNL/zfp/blob/master/src/zfp.c#L303
https://github.com/LLNL/zfp/blob/master/src/zfp.c#L308

Small error on Compiling in Mac OS X

In mac OS, the ar command will not create the lib library if it doesn't exist. I get the following error:

ar: ../lib/libzfp.a: No such file or directory

Creating the library myself, the compilation works.

Installing "memory.h" conflicts with libc6

The line:

zfp/CMakeLists.txt

Line 240 in 25a554e

install(DIRECTORY array/ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})

installs a memory.h file in <prefix>/include.
On debian, ubuntu and several other linux distributions, the /usr/include/memory.h is installed by libc6-dev.

This causes a conflict:

1. The libc version of the file is overwritten if the prefix is /usr
2. When searching for the libc version of<memory.h>, even for software that does not use zfp, the compiler is likely to find the zfp version before the right one, therefore the compilation is likely to fail.

I suggest to rename the file, or to put it in a subdirectory.

cases where bitstream type is used without including bitstream.h

zfp.h includes bitstream.h and so that kinda sorta hides this issue. But, zfp applications such as the file compressor, zfp or the simple example use bitstream types without a #include <bitstream.h>

Progress on thread safety?

I'm experimenting with using zfp to store terrain (elevation) data. Early results are very encouraging, saving 50% over my current compression solution - thank you.

The project I'm building it into is multi-threaded (for read access only). Before I go any further with integrating it, could I ask if there's any progress on making zfp thread-safe?

Cert expired

The https cert associated with the following site has expired:

https://computation.llnl.gov/projects/floating-point-compression

I would personally use a new cert that expires monthly and have an automated process to renew it. This will exercise some automation skills and will eventually prevent the cert from expiring by surprise. Thanks.

Array size must be nonzero

Hi, zfp -i datafile not working, the message is "array size must be nonzero", the same if i try to decompress, may you help me?

Efficient sub-cube access

I would like to have a fast sub-cube access to a raw buffer from the C++ array classes.
I note that for the whole cube looping over the elements is 2-3x slower than using get(Scalar *p).

The view logic isnt faster as it basically ends up going through the same point-by-point accessors.

I implemented a in function (in zfparray3.h) that has the signature
get(Scalar *p, uint sub_x, unit sub_y, uint sub_z, uint sub_nx, uint sub_ny, uint sub_nz )

This uses the same strategy as get(Scalar *p) and visits each block only one and gets all the juice out of that block. It is a similar speed to the full cube get().

From a style point of view it would be nicer to have a view::get(Scalar *p) interface.
I.e. "just get the uncompressed bytes for this view". However view doesn't really have access to all the cache representation inside the array class.

Does anyone have suggestions for the cleanest way to do this?

make install overwrites googletest

When running make install, the gtest headers are installed to /usr/local/include, overwriting any version of gtest that might already be installed.

Warp divergence in deposit bit plane

The loop writing bit planes was missing a static loop bound.
Adding the loop bound eliminates Warp divergence and speeds up things.
In addition, the unrolling factor was missing from the loop.

I have uploaded a fix in #35

Segenmtation fault when runing

I want to compress 1059810*3 float array with zfp

const void* my_zfp_compression_xyz(unsigned int org_size, float* xyz, size_t& bytes_size){
    zfp_type type;
    zfp_field* field;
    zfp_stream* zfp;
    void* buffer;
    size_t  bufsize;
    bitstream* stream;
    size_t zfpsize;

    type = zfp_type_float;
    field = zfp_field_2d(xyz, type, 3, org_size); //org_size = 1059810

    zfp = zfp_stream_open(NULL);

    zfp_stream_set_accuracy(zfp, 1e-3);

    bufsize = zfp_stream_maximum_size(zfp, field);
    buffer = malloc(bufsize);

    stream = stream_open(buffer, bufsize);
    zfp_stream_set_bit_stream(zfp, stream);
    zfp_stream_rewind(zfp);

    ///compress
    zfpsize = zfp_compress(zfp, field);
    bytes_size = zfpsize;
    if(!zfpsize){
        fprintf(stderr, "compression failed!\n");
        exit(0);
    }
    else return  buffer;
}

and when I debug, it comes out

"Program received signal SIGSEGV, Segmentation fault.
0x00007f4f34f92e3e in zfp_encode_partial_block_strided_float_2 () from /usr/local/lib/libzfp.so.0"

below is partial assembly code, but I cant understand it whats that means(never learn assemble language)

! [0x00007f4f34f92e3e..0x00007f4f34f9313d]:
	movss  (%rdi),%xmm0                                    ! 0x00007f4f34f92e3e
	add    %r10,%rdi                                       ! 0x00007f4f34f92e42
	cmp    %r9d,%eax                                       ! 0x00007f4f34f92e45
	movss  %xmm0,-0x1f0(%rbp,%r8,4)                        ! 0x00007f4f34f92e48
	jne    0x7f4f34f92e38  # <zfp_encode_partial_block_strided_float_2+168> ! 0x00007f4f34f92e52
	add    %r15,%rbx                                       ! 0x00007f4f34f92e54
	cmp    $0x2,%edx                                       ! 0x00007f4f34f92e57
	je     0x7f4f34f92f98  # <zfp_encode_partial_block_strided_float_2+520> ! 0x00007f4f34f92e5a
	cmp    $0x3,%edx                                       ! 0x00007f4f34f92e60
	je     0x7f4f34f92f60  # <zfp_encode_partial_block_strided_float_2+464> ! 0x00007f4f34f92e63
	cmp    $0x1,%edx                                       ! 0x00007f4f34f92e69
	je     0x7f4f34f92f78  # <zfp_encode_partial_block_strided_float_2+488> ! 0x00007f4f34f92e6c
	add    $0x1,%r13d                                      ! 0x00007f4f34f92e72
	add    %r14,%rbx                                       ! 0x00007f4f34f92e76
	add    $0x4,%r11d                                      ! 0x00007f4f34f92e79
	cmp    %r13d,%ecx                                      ! 0x00007f4f34f92e7d

anyone can give some suggestions?

regression testing requires BIT_STREAM_WORD_TYPE=uint64

Any particular reason why regression testing does not support BIT_STREAM_WORD_TYPE=uint8?

bitstream.h is a very generic name

Much like #25, "bitstream.h" is a very generic name and at the very least should be installed in under the include/zfp/.

Would help to set version no in branches targetted for given release

Was using feature/lossless branch to enhance H5Z-ZFP.

I want H5Z-ZFP to conditionally compile against various versions of ZFP.

In this case, I wanted to do something like...

#if ZFP_VERSION_NO >= 0x0055
            case H5Z_ZFP_MODE_REVERSIBLE:
                zfp_stream_set_reversible(dummy_zstr);
                break;
#endif

But, building against ZFP's feature/lossless branch, my tests were all failing because that branch still has version number set at 0.5.4.

Rationale for double negation (!!)

Just curious what the rationale is for the double negation,

zfp/tests/testzfp.cpp

Line 434 in 068f815

pass = !!zfp_decompress(stream, output);

appearing above and in many other places in this same file?

Please add pkg-config .pc file

Something like this:

Name: zfp
Description: Library for compressed numerical arrays that support high throughput read and write random access
Requires:
Version: 0.5.3
Libs:  -L/usr/local/lib -lzfp
Cflags:  -I/usr/local/include

30x less compression rate then zip

I am using tolerance = 0.0 to compress 2d 1000 * 1000 float arrays( a lot of different arrays, so this is not some accident). If for example the array just contains zeros or some value compression rate is high, but when there are several repeated values and some random compression rate is low. it is something like 30%. At the same time, zip library compresses the same array on average 30x better.

Add support for OpenMP decompression

I upload a patch in #38 to add support for OpenMP decompression.
This patch is not compatible with the ZFP header. We need to align on how to combine both of these features.

cfp library not part of install package

pull request upcoming.

Question- Sample test Data (example-seismic ) Info

Question 1:

Is any sample test data(example seismic data) or sample test files Info provide to understand and test on compress and decompress ,to test on OpenMP and ZFP serial stream
example: zfp -f -3 201 201 2412 -r 4 -x serial -i snaps.bin -z test.zfp -h

How to get the number of uncompressed bytes from zfp_decompress

Hello, I'm playing with the ZFP High level API and found that the zfp_decompress function returns the number of compressed bytes found in the stream, while I was expecting to return the number of uncompressed bytes extracted from the compressed stream.

The reason I wish to know the uncompressed byte count is that, when I compress some data, I can store (say on a file) the compressed buffer and its compressed byte size, then I would read back this compressed data, uncompress them in a large enough buffer and find out which is the original dimension in bytes.

Am I missing something? Thank you for any hint or explanation.

Building static library with cmake?

All,

I'm trying to build H5Z-ZFP so first I need to build zfp (0.5.2, because it seems to be the latest supported version). However, when I did:

/ford1/share/gmao_SIteam/Baselibs/ESMA-Baselibs-5.2.0-ZFPTry/x86_64-unknown-linux-gnu/ifort_18.0.3.222-openmpi_3.1.0-gcc_6.3.0/Linux/bin/h5pcc H5Zzfp_plugin.o -shared -o plugin/libh5zzfp.so \
    -Wl,-rpath,/ford1/share/gmao_SIteam/Baselibs/ESMA-Baselibs-5.2.0-ZFPTry/x86_64-unknown-linux-gnu/ifort_18.0.3.222-openmpi_3.1.0-gcc_6.3.0/Linux/lib -Wl,-rpath,/ford1/share/gmao_SIteam/Baselibs/ESMA-Baselibs-5.2.0-ZFPTry/x86_64-unknown-linux-gnu/ifort_18.0.3.222-openmpi_3.1.0-gcc_6.3.0/Linux/zfp/lib \
    -L/ford1/share/gmao_SIteam/Baselibs/ESMA-Baselibs-5.2.0-ZFPTry/x86_64-unknown-linux-gnu/ifort_18.0.3.222-openmpi_3.1.0-gcc_6.3.0/Linux/zfp/lib -L/ford1/share/gmao_SIteam/Baselibs/ESMA-Baselibs-5.2.0-ZFPTry/x86_64-unknown-linux-gnu/ifort_18.0.3.222-openmpi_3.1.0-gcc_6.3.0/Linux/lib -lhdf5 -lzfp 
/usr/bin/ld: cannot find -lzfp
collect2: error: ld returned 1 exit status

So, no libzfp.a, and if I look at my zfp build:

(338)((no branch)) $ tree lib64/ bin/ 
lib64/
|-- libzfp.so -> libzfp.so.0
|-- libzfp.so.0 -> libzfp.so.0.5.2
`-- libzfp.so.0.5.2
bin/
|-- testzfp
`-- zfp

0 directories, 5 files

My cmake line was:
cmake -DCMAKE_INSTALL_PREFIX=$(prefix)/zfp -DZFP_BIT_STREAM_WORD_SIZE=8 ..

Is there an extra option I need to pass to CMake to generate the static library as well as the shared object library?

ZFP header for GPU compression

Dear all,

I am trying to generate a .zfp file with a header using CUDA. Using the zfp util on the command line, a serial test gives:

zfp -f -3 201 201 2412 -r 4 -x serial -i snaps.bin -z test.zfp -h

test.zfp has size 50188912 and can be correctly decompressed using:

zfp -h -z test.zfp -o test.bin

But doing the same with the cuda flag, i.e.:

zfp -f -3 201 201 2412 -r 4 -x cuda -i snaps.bin -z test.zfp -h

Now test.zfp has size 50188896, and

zfp -h -z test.zfp -o test.bin

gives incorrect or missing header. When doing the decompression with all the parameters, it works perfectly. How could I put this header when doing the compression on the GPU?

Thanks in advance,

Victor

AMD GPU-3D Test Cases Failed like testZfpCuda3dDouble,testZfpCuda3dInt32..

Cuda source changes are convert into HIP(Hipify). ZFP source changes run in AMD GPU(vega10).

On "make test" command- test cases are executed.

AMD_ZFP_Trace.zip

97% tests passed | 4 tests failed out of 155
Total Test time (real) = 145.29 sec
The following tests FAILED: |  
92 - testZfpCuda3dInt32 (Failed)
95 - testZfpCuda3dInt64 (Failed)
114 - testZfpCuda3dFloat (Failed)
117 - testZfpCuda3dDouble (Failed)

bin]$./testZfpCuda3dInt32
[Traces]
ERROR: Overall compress/decompress test failure

[ ERROR ] --- [ LINE ] --- /home/taccuser/Desktop/Anil_Bommareddy/zfp_old_org/tests/src/endtoend/zfpEndtoendBase.c:621: error: Failure!
ERROR: Overall compress/decompress test failure

[ ERROR ] --- [ LINE ] --- /home/taccuser/Desktop/Anil_Bommareddy/zfp_old_org/tests/src/endtoend/zfpEndtoendBase.c:621: error: Failure!
ERROR: Overall compress/decompress test failure

[ ERROR ] --- [ LINE ] --- /home/taccuser/Desktop/Anil_Bommareddy/zfp_old_org/tests/src/endtoend/zfpEndtoendBase.c:621: error: Failure!
[ PASSED ] 3 test(s).
[ FAILED ] 3 test(s), listed below:
[ FAILED ] given_Cuda_3dInt32ReversedArray_when_ZfpCompressDecompressFixedRate_expect_BitstreamAndArrayChecksumsMatch
[ FAILED ] given_Cuda_3dInt32PermutedArray_when_ZfpCompressDecompressFixedRate_expect_BitstreamAndArrayChecksumsMatch
[ FAILED ] given_Cuda_3dInt32Array_when_ZfpCompressDecompressFixedRate_expect_BitstreamAndArrayChecksumsMatch

zfpEndtoendBase.c:621: static void
runCompressDecompressTests(void** state, zfp_mode mode, int numCompressParams)

Please find attached traces

What is the purpose of this code?

looking at the code I don't see the purpose of following lines

    /* read compressed input file in increasingly large chunks */
    FILE* file = !strcmp(zfppath, "-") ? stdin : fopen(zfppath, "rb");
    if (!file) {
      fprintf(stderr, "cannot open compressed file\n");
      return EXIT_FAILURE;
    }
    bufsize = 0x100;
    do {
      bufsize *= 2;
      buffer = realloc(buffer, bufsize);
      if (!buffer) {
        fprintf(stderr, "cannot allocate memory\n");
        return EXIT_FAILURE;
      }
      zfpsize += fread((uchar*)buffer + zfpsize, 1, bufsize - zfpsize, file);
    } while (zfpsize == bufsize);
    if (ferror(file)) {
      fprintf(stderr, "cannot read compressed file\n");
      return EXIT_FAILURE;
    }
    fclose(file);
    /* associate bit stream with buffer */
    stream = stream_open(buffer, bufsize);
    if (!stream) {

if we have 1tb of compressed data, and available to us only 32gb of RAM.
how you are going to read this file in ram?

shouldn't stream implement reading in chunks from file, instead from ram?

zfp/types.h header is not C++ safe

When trying to use the zfp library for a C++ project, we have encountered a warning regarding the use of the __STDC_VERSION__ macro at the very beginning of the zfp/types.h header file. The problem is that the g++ compiler does not define this macro, because it considers itself to be a C++ and not a C compiler, as pointed out in the documentation in https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html:

This macro is not defined if the -traditional-cpp option is used, nor when compiling C++ or Objective-C.

According to the C++ standards, it is perfectly legit for a C++ compiler to leave __STDC_VERSION__ undefined.

Therefore, you should at least check whether __STDC_VERSION__ is defined first. Also, you might consider preceding this by an #if block that checks for __cplusplus >= 201103L, which would indicate a C++11 conforming implementation that offers the cstdint header along with the necessary typedefs, which you could use in analogy to the C99-conforming case.

And furthermore, in lines 62 and 67 of zfp/types.h, you are using #elif ZFP_LP64 instead of #elif defined(ZFP_LP64), which also leads to at least one warning regarding undefined macros.

stream_word_bits declared outside extern "C"

in include/bitstream.h, shouldn't the declaration for stream_word_bits be put inside the extern "C" block?

/* forward declaration of opaque type */
typedef struct bitstream bitstream;

extern_ const size_t stream_word_bits; /* bit stream granularity */

#ifndef inline_
#ifdef __cplusplus
extern "C" {
#endif

mmap-backed persistent ZFP arrays

It would be nice to be able to save to/restore compressed ZFP arrays from mmapped files. At present a crude workaround has been applied to zfparray.h:

added three extra variables to the array class:
std::string storage; //persistent storage via mmap
bool is_mmapped;
size_t storage_size;

and then alloc() / free() have been changed too:

// allocate memory for compressed data
void alloc(bool clear = true)
{
bytes = blocks * blkbits / CHAR_BIT;
printf("zfp::array3::alloc<clear:%d, storage:%s, bytes:%zu>\n", clear, storage.c_str(), bytes);
if (bytes > 1 && storage != "")
{
//try mmap first
int fd = open(storage.c_str(), O_RDWR | O_CREAT, (mode_t)0600);
if (fd != -1)
{
int stat = ftruncate64(fd, bytes);
if (!stat)
{
void *buffer = mmap(0, bytes, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

      if (buffer != MAP_FAILED)
      {
        data = (uchar *)buffer;
        is_mmapped = true;
        storage_size = bytes;
        printf("zfp::array3::alloc<mmap OK>.\n");
      }
      else
        perror("mmap");
    }
    else
    {
      perror("ftruncate64");
    }
    close(fd);
  }
}

if (!is_mmapped)
{
  reallocate(data, bytes, 0x100u);
  if (clear)
    std::fill(data, data + bytes, 0);
}
stream_close(zfp->stream);
zfp_stream_set_bit_stream(zfp, stream_open(data, bytes));
clear_cache();

}

// free memory associated with compressed data
void free()
{
nx = ny = nz = 0;
bx = by = bz = 0;
blocks = 0;
stream_close(zfp->stream);
zfp_stream_set_bit_stream(zfp, 0);
bytes = 0;
if (!is_mmapped)
deallocate(data);
else
{
if (data != NULL)
{
if (munmap(data, storage_size) == -1)
perror("un-mapping error");
else
printf("zfp::array3::free.\n");

    storage_size = 0;
    is_mmapped = false;
  };
};
data = 0;
deallocate(shape);
shape = 0;

}

This is a "quick&dirty" workaround. A more elegant ZFP-wide solution would be much nicer.

the array3 constructor in zfparray3.h has been modified slightly too:
array3(uint nx, uint ny, uint nz, double rate, const Scalar* p = 0, size_t csize = 0, std::string _storage = std::string("")) :
array(3, Codec::type),
cache(lines(csize, nx, ny, nz))
{
storage = _storage;
is_mmapped = false;
storage_size = 0;
set_rate(rate);
resize(nx, ny, nz, p == 0);
if (p)
set(p);
}

Half-precision (16-bit) floating-point compression

Is there any subsequent plan to support f16 (half-precision) floating-point value compression? At this point, we can only select -f for single- and -d for double-precision compression.

roundtrip impossible

Hi,

I am playing with zfp to compress some cryoEM data. I am struggling to perform a roundtrip, so I am doing the following with the current master:

/path/to/zfp -x omp=40 -f -i Tomo147_odd_topCrop.raw -o Tomo147_odd_topCrop_rt_a0.0001.raw -3 1092 3977 2580 -a 0.0001

I was hoping that the output file would match in size to the input, which it doesn't. The output is 4x smaller than the input. Which suprises me? The same happens if I do:

$ /path/to/zfp -i test.raw -z test.zfp <other options>
$ /path/to/zfp -z test.zfp -o roundtrip.raw <other options>

Am I overseeing something?
Best,
Peter

Compression segfaults for z-smallest strides

The following code using strides of {4*5, 5, 1} on data of size (3,4,5) and fails with a segfault on linux, but using strides of {1, 3, 3*4 } works just fine.

#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include "zfp.h"

int main() {
    static const int nx = 3;
    static const int ny = 4;
    static const int nz = 5;
    int strides[3] = { 4*5, 5, 1};
    //int strides[3] = { 1, 3, 3*4 };
    float data[3*4*5] = {0};

    // Fill the data
    for(int iz=0; iz<nz; ++iz) {
        for(int iy=0; iy<ny; ++iy) {
            for(int ix=0; ix<nx; ++ix) {
                int idx = ix*strides[0] + iy * strides[1] + iz * strides[2];
                data[idx] = iz + sin(0.01*ix + 0.02*iy + 0.03*iz + 0.00002*(ix*iy));
            }
        }
    }

    // Compress and write to stdout
    double bits_per_value = 12;
    zfp_field * field = zfp_field_alloc();
    zfp_field_set_size_3d(field, nx, ny, nz);
    zfp_field_set_pointer(field, data);
    zfp_field_set_stride_3d(field, strides[0], strides[1], strides[2]);
    zfp_field_set_type(field, zfp_type_float);
    zfp_stream * zfp = zfp_stream_open(NULL);
    zfp_stream_set_rate(zfp, bits_per_value, zfp_type_float, 3, 0);
    size_t bufsize = zfp_stream_maximum_size(zfp, field);
    void* buffer = malloc(bufsize);    
    bitstream * stream = stream_open(buffer, bufsize);
    zfp_stream_set_bit_stream(zfp, stream);
    size_t zfpsize = zfp_compress(zfp, field);
    fwrite(buffer, zfpsize, 1, stdout);
}

The error I get is:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bc5f60 in gather_partial_float_3 (q=0x7fffffffae00, p=0x8003ffffb334, nx=3, ny=4, nz=4, sx=20, sy=5, sz=1)
    at /xxx/zfp/src/template/encode3.c:22
22              q[16 * z + 4 * y + x] = *p;

The stack trace I get is :

#0  0x00007ffff7bc5f60 in gather_partial_float_3 (q=0x7fffffffae00, p=0x8003ffffb334, nx=3, ny=4, nz=4, sx=20, sy=5, sz=1)
    at /xxxzfp/src/template/encode3.c:22
#1  0x00007ffff7bc6368 in zfp_encode_partial_block_strided_float_3 (stream=0x603040, p=0x7fffffffb320, nx=3, ny=4, nz=4, sx=20, sy=5, sz=1)
    at /xxx/zfp/src/template/encode3.c:71
#2  0x00007ffff7baeb54 in compress_strided_float_3 (stream=0x603040, field=0x603010)
    at /xxx/zfp/src/template/compress.c:80
#3  0x00007ffff7bb693a in zfp_compress (zfp=0x603040, field=0x603010) at /xxx/zfp/src/zfp.c:746
#4  0x0000000000400e46 in main () at zfp_test.c:44

And gdb shows:

buffer=0x603070
data=0x7fffffffb320
*stream = {bits = 0, buffer = 0, ptr = 0x603070, begin = 0x603070, end = 0x603148}
*zfp = {minbits = 768, maxbits = 768, maxprec = 64, minexp = -1074, stream = 0x603150, exec = {policy = zfp_exec_serial, params = {omp = {threads = 0,
        chunk_size = 0}}}}
*field = {type = zfp_type_float, nx = 3, ny = 4, nz = 5, sx = 20, sy = 5, sz = 1, data = 0x7fffffffb320}

GPU issue ?

Dear all,

I want to use zfp on GPU and for that, I am doing some bench.
I have uncommented the line #define CUDA_ZFP_RATE_PRINT 1 in shared.h file and did some runs.
The code is working well for 3D cubes of size of 100, 200, 300, 400 and 500.
However, we have noticed that for 3D cube of size 600, 700, or even 550 the run fails.

Let A be a 3D cude of size 600.
It turns out that the decompression of A, say A_d, seems to not working.
More precisely, the returned value of zfp_decompress is not 0, and the error ||A - A_d||_2 is of order 1e3. Note that in order to ease the computation of the error norm, we consider A and A_d as 1D vector.

Here is attached the test driver, the output obtained on a V100 GPU, and the needed additional compilation lines for a CMakeList.txt.
Please remove the .log extension from the source file, and add at the end of the examples/CMakeList.txt file the content of zfp_cmakelist_update.txt.

zfp_cmakelist_update.txt
report_gpu_zfp_issue.c.log
zfp_exec_issue.log

It turns out that the decode rate of A is inf and the next call of zfp on GPU fails to allocate GPU memory.

Let me know if you need more details.
Sébastien

Provide pip and/or conda installable Python package

It would really help to have a pip installable Python 3.7 compatible package published on https://pypi.org/ . This could then be updated whenever there is a new release of zfp. It will lower the barrier to using zfp. Thanks.

How do I know what's new in each release?

I am updating H5Z-ZFP filter to use latest and greates ZFP library. But, I honestly don't know what is new in the library? Is it just bug fixes or other internal refactoring? Or, are there new features I should be aware of? Where might I go looking to get this information if it already exists? And, if it already exists, any chance future releases might include it in the releases section of this repo?

Feature request: Compute bound in error

In this paper, equation 5.1 and 5.2, the uniform norm error of the compression scheme is bound in terms of a constant K_b. Could this be computed in code, for example, if I parametrize zfp in some way, can I test those parameters against K_b before I do any compression?