race condition when using private_const_view from inside oneapi::tbb:parallel_for lambda about zfp

[lindstro]

I'm afraid I'm unfamiliar with oneAPI and TBB, but I can make some general recommendations.

Before using a private_const_view you should flush the cache of the underlying array to ensure cache consistency. See the documentation and this code example:

zfp/examples/diffusion.cpp

Lines 95 to 103 in c4f1756

	// flush shared cache to ensure cache consistency across threads
	u.flush_cache();
	// zero-initialize du
	du.set(0);
	// compute du/dt in parallel
	#pragma omp parallel
	{
	// create read-only private view of entire array u
	zfp::array2d::private_const_view myu(&u);

It might also be instructive to capture the indices through which the private_const_view is accessed.

Finally, zfp's private views were written for use with OpenMP. I suspect that view reference counting, as needed to enforce thread safety, is being circumvented:

zfp/include/zfp/internal/array/store.hpp

Lines 126 to 136 in c4f1756

	// increment private view reference count (for thread safety)
	void reference()
	{
	#ifdef _OPENMP
	#pragma omp critical(references)
	{
	references++;
	codec.set_thread_safety(references > 1);
	}
	#endif
	}

We'd have to do something similar to support TBB.

[aavbsouza]

Hello @lindstro I have added the flush_cache without success. The error happens even if I only try to access a fixed index, for instance (0,0). I thought that compiling the zfp library with OpenMP enabled and also linking the program with OpenMP would be sufficient to use the OpenMP part of the code. I also have tried to create the views using the enumerable_thread_specific from the tbb library. I will try a similar arrangement using only OpenMP.

[lindstro]

I thought that compiling the zfp library with OpenMP enabled and also linking the program with OpenMP would be sufficient to use the OpenMP part of the code.

I'm afraid not as the zfp arrays are implemented in headers, and hence the OpenMP pragmas needs to be compiled together with the application code (i.e., your example above). Is it possible for TBB and OpenMP to coexist in the same application, e.g., by adding -fopenmp when compiling your code? If not, do you know the TBB equivalent to an OpenMP critical section?

[aavbsouza]

Hello @lindstro the oneapi tbb documentation states that is possible to mix the use of the library with other threading packages. The library supports many kinds of mutexes. I also believe that would be possible to use standard c++ mutexes.

Regarding the compilation I am linking with the openMP runtime

[1/2] /usr/bin/g++  -I/host/build-gcc/_deps/tbb-src/src/tbb/../../include -I/host/build-gcc/_deps/zfp-src/include -O2 -g -DNDEBUG -fPIE -fsanitize=thread -Wextra -Wall -pedantic -rdynamic -fno-omit-frame-pointer -g -ggdb -fdiagnostics-color=always -mtune=native -O3 -fopenmp -std=c++20 -MD -MT tests/CMakeFiles/zfp_tbb.dir/zfp_tbb.cpp.o -MF tests/CMakeFiles/zfp_tbb.dir/zfp_tbb.cpp.o.d -o tests/CMakeFiles/zfp_tbb.dir/zfp_tbb.cpp.o -c /workspaces/cpp-zfp/tests/zfp_tbb.cpp
[2/2] : && /usr/bin/g++ -O2 -g -DNDEBUG -fsanitize=thread tests/CMakeFiles/zfp_tbb.dir/zfp_tbb.cpp.o -o tests/zfp_tbb  -Wl,-rpath,/host/build-gcc/gnu_11.3_cxx20_64_relwithdebinfo:/host/build-gcc/_deps/zfp-build/lib64  gnu_11.3_cxx20_64_relwithdebinfo/libtbb.so.12.7  _deps/zfp-build/lib64/libzfp.so.1.0.0  /usr/lib/gcc/x86_64-redhat-linux/11/libgomp.so  /usr/lib64/libpthread.a && :

[lindstro]

Yes, but linking is not sufficient--you also need to compile your code with -fopenmp so that _OPENMP is defined and the pragmas in the zfp headers are processed. Even then, I'm not entirely convinced that non-OpenMP threads entering the OpenMP critical region will do the right thing, but it's worth a shot.

[aavbsouza]

Hello @lindstro I am compiling with -fopenmp is on the far right of the command. I also have tried to implement using openMP directly:

#include <numeric>
#include <vector>
#include <zfp/constarray2.hpp>

void test() {
    const std::size_t n1 = 2000;
    const std::size_t n2 = 60000;
    const std::size_t nindex = 10000;
    std::vector<float> buf(n1 * n2);
    zfp::const_array2<float> comp_mat(n1, n2, zfp_config_precision(10), buf.data());
    std::vector<std::size_t> indexes(nindex);
    std::vector<double> results(nindex);
    std::iota(indexes.begin(), indexes.end(), 0);

#pragma omp parallel default(none) shared(results, indexes, comp_mat)
    {
        zfp::const_array2<float>::private_const_view pv(&comp_mat);
#pragma omp for schedule(dynamic)
        for (std::size_t i = 0; i < indexes.size(); ++i) {
            double sum = 0.0;
            for (std::size_t i1 = 0; i1 < n1; ++i1) {
                sum += pv(i1, i);
            }
            results[i] = sum ;
        }
    }
}
int main() {
    test();
    return 0;
}

However I still got a data race:

WARNING: ThreadSanitizer: data race (pid=40760)
  Read of size 8 at 0x7ffdedbdd8e8 by thread T78:
    #0 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::reference() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:132 (zfp_omp+0x40374f)
    #1 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::private_const_view(zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>*, unsigned long) /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:352 (zfp_omp+0x40374f)
    #2 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:19 (zfp_omp+0x40374f)
    #3 gomp_thread_start <null> (libgomp.so.1+0x1d245)

  Previous write of size 8 at 0x7ffdedbdd8e8 by main thread:
    #0 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::reference() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:132 (zfp_omp+0x40375f)
    #1 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::private_const_view(zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>*, unsigned long) /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:352 (zfp_omp+0x40375f)
    #2 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:19 (zfp_omp+0x40375f)
    #3 GOMP_parallel <null> (libgomp.so.1+0x14575)
    #4 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Location is stack of main thread.

  Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f8e8)

  Thread T78 (tid=41286, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:132 in zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::reference()
==================
==================
WARNING: ThreadSanitizer: data race (pid=40760)
  Write of size 1 at 0x7ffdedbdd940 by thread T78:
    #0 zfp::codec::zfp_base<2u, float>::set_thread_safety(bool) /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:102 (zfp_omp+0x403779)
    #1 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::reference() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:133 (zfp_omp+0x403779)
    #2 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::private_const_view(zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>*, unsigned long) /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:352 (zfp_omp+0x403779)
    #3 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:19 (zfp_omp+0x403779)
    #4 gomp_thread_start <null> (libgomp.so.1+0x1d245)

  Previous write of size 1 at 0x7ffdedbdd940 by main thread:
    [failed to restore the stack]

  Location is stack of main thread.

  Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f940)

  Thread T78 (tid=41286, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:102 in zfp::codec::zfp_base<2u, float>::set_thread_safety(bool)
==================
==================
WARNING: ThreadSanitizer: data race (pid=40760)
  Read of size 8 at 0x7b0c00000030 by thread T78:
    #0 stream_clone /host/build-gcc/_deps/zfp-src/include/zfp/bitstream.inl:469 (libzfp.so.1+0x17e9e)
    #1 zfp::codec::zfp_base<2u, float>::clone_stream() const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:142 (zfp_omp+0x403f80)
    #2 zfp::codec::zfp_base<2u, float>::decode_block(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:165 (zfp_omp+0x403f80)
    #3 zfp::codec::zfp2<float>::decode_block(unsigned long, unsigned int, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:299 (zfp_omp+0x403f80)
    #4 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::decode(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:107 (zfp_omp+0x403f80)
    #5 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::line(unsigned long, unsigned long, bool) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:176 (zfp_omp+0x403f80)
    #6 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:59 (zfp_omp+0x403f80)
    #7 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:405 (zfp_omp+0x403f80)
    #8 zfp::internal::dim2::const_handle<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::get() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/handle2.hpp:28 (zfp_omp+0x403f80)
    #9 zfp::internal::dim2::const_reference<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::operator float() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/reference2.hpp:19 (zfp_omp+0x403f80)
    #10 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:24 (zfp_omp+0x403f80)
    #11 gomp_thread_start <null> (libgomp.so.1+0x1d245)

  Previous write of size 8 at 0x7b0c00000030 by main thread:
    [failed to restore the stack]

  Location is heap block of size 40 at 0x7b0c00000030 allocated by main thread:
    #0 malloc <null> (libtsan.so.0+0x32dd7)
    #1 stream_open /host/build-gcc/_deps/zfp-src/include/zfp/bitstream.inl:444 (libzfp.so.1+0x17dd5)
    #2 zfp::codec::zfp_base<2u, float>::open(void*, unsigned long) /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:60 (zfp_omp+0x4061db)
    #3 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::compact() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:122 (zfp_omp+0x4061db)
    #4 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::set(float const*, bool) /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:224 (zfp_omp+0x4061db)
    #5 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::const_array2(unsigned long, unsigned long, zfp_config const&, float const*, unsigned long) /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:59 (zfp_omp+0x4061db)
    #6 test() /workspaces/cpp-zfp/tests/zfp_omp.cpp:12 (zfp_omp+0x4061db)
    #7 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Thread T78 (tid=41286, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race /host/build-gcc/_deps/zfp-src/include/zfp/bitstream.inl:469 in stream_clone
==================
==================
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=40760)
  Write of size 8 at 0x7ffdedbdd8d0 by main thread:
    #0 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:179 (zfp_omp+0x405fa9)
    #1 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore2() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:11 (zfp_omp+0x405fa9)
    #2 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~const_array2() /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:70 (zfp_omp+0x405fa9)
    #3 test() /workspaces/cpp-zfp/tests/zfp_omp.cpp:29 (zfp_omp+0x405fa9)
    #4 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Previous read of size 8 at 0x7ffdedbdd8d0 by thread T70:
    [failed to restore the stack]

  Location is stack of main thread.

  Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f8d0)

  Thread T70 (tid=41278, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:179 in zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore()
==================
==================
WARNING: ThreadSanitizer: data race (pid=40760)
  Write of size 8 at 0x7b0c00000048 by main thread:
    #0 free <null> (libtsan.so.0+0x38c38)
    #1 stream_close /host/build-gcc/_deps/zfp-src/include/zfp/bitstream.inl:460 (libzfp.so.1+0x17e4d)
    #2 zfp::codec::zfp_base<2u, float>::close() /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:66 (zfp_omp+0x406024)
    #3 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::free() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:227 (zfp_omp+0x406024)
    #4 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:179 (zfp_omp+0x406024)
    #5 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore2() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:11 (zfp_omp+0x406024)
    #6 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~const_array2() /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:70 (zfp_omp+0x406024)
    #7 test() /workspaces/cpp-zfp/tests/zfp_omp.cpp:29 (zfp_omp+0x406024)
    #8 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Previous read of size 8 at 0x7b0c00000048 by thread T43:
    #0 stream_clone /host/build-gcc/_deps/zfp-src/include/zfp/bitstream.inl:469 (libzfp.so.1+0x17e9e)
    #1 zfp::codec::zfp_base<2u, float>::clone_stream() const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:142 (zfp_omp+0x403f80)
    #2 zfp::codec::zfp_base<2u, float>::decode_block(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:165 (zfp_omp+0x403f80)
    #3 zfp::codec::zfp2<float>::decode_block(unsigned long, unsigned int, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:299 (zfp_omp+0x403f80)
    #4 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::decode(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:107 (zfp_omp+0x403f80)
    #5 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::line(unsigned long, unsigned long, bool) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:176 (zfp_omp+0x403f80)
    #6 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:59 (zfp_omp+0x403f80)
    #7 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:405 (zfp_omp+0x403f80)
    #8 zfp::internal::dim2::const_handle<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::get() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/handle2.hpp:28 (zfp_omp+0x403f80)
    #9 zfp::internal::dim2::const_reference<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::operator float() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/reference2.hpp:19 (zfp_omp+0x403f80)
    #10 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:24 (zfp_omp+0x403f80)
    #11 gomp_thread_start <null> (libgomp.so.1+0x1d245)

  Thread T43 (tid=41251, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race (/lib64/libtsan.so.0+0x38c38) in free
==================
==================
WARNING: ThreadSanitizer: data race (pid=40760)
  Write of size 8 at 0x7b0c00000010 by main thread:
    #0 zfp_stream_set_bit_stream /host/build-gcc/_deps/zfp-src/src/zfp.c:747 (libzfp.so.1+0xf3de)
    #1 zfp::codec::zfp_base<2u, float>::close() /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:67 (zfp_omp+0x40603a)
    #2 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::free() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:227 (zfp_omp+0x40603a)
    #3 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:179 (zfp_omp+0x40603a)
    #4 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore2() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:11 (zfp_omp+0x40603a)
    #5 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~const_array2() /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:70 (zfp_omp+0x40603a)
    #6 test() /workspaces/cpp-zfp/tests/zfp_omp.cpp:29 (zfp_omp+0x40603a)
    #7 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Previous read of size 8 at 0x7b0c00000010 by thread T29:
    #0 zfp::codec::zfp_base<2u, float>::clone_stream() const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:141 (zfp_omp+0x403f4d)
    #1 zfp::codec::zfp_base<2u, float>::decode_block(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:165 (zfp_omp+0x403f4d)
    #2 zfp::codec::zfp2<float>::decode_block(unsigned long, unsigned int, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:299 (zfp_omp+0x403f4d)
    #3 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::decode(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:107 (zfp_omp+0x403f4d)
    #4 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::line(unsigned long, unsigned long, bool) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:176 (zfp_omp+0x403f4d)
    #5 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:59 (zfp_omp+0x403f4d)
    #6 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:405 (zfp_omp+0x403f4d)
    #7 zfp::internal::dim2::const_handle<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::get() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/handle2.hpp:28 (zfp_omp+0x403f4d)
    #8 zfp::internal::dim2::const_reference<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::operator float() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/reference2.hpp:19 (zfp_omp+0x403f4d)
    #9 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:24 (zfp_omp+0x403f4d)
    #10 gomp_thread_start <null> (libgomp.so.1+0x1d245)

  Location is heap block of size 40 at 0x7b0c00000000 allocated by main thread:
    #0 malloc <null> (libtsan.so.0+0x32dd7)
    #1 zfp_stream_open /host/build-gcc/_deps/zfp-src/src/zfp.c:539 (libzfp.so.1+0xe58c)
    #2 zfp::codec::zfp_base<2u, float>::zfp_base() /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:21 (zfp_omp+0x40466a)
    #3 zfp::codec::zfp2<float>::zfp2() /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:286 (zfp_omp+0x40466a)
    #4 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::BlockStore() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:175 (zfp_omp+0x40466a)
    #5 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::BlockStore2(unsigned long, unsigned long, zfp_config const&) /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:21 (zfp_omp+0x40466a)
    #6 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::const_array2(unsigned long, unsigned long, zfp_config const&, float const*, unsigned long) /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:54 (zfp_omp+0x40466a)
    #7 test() /workspaces/cpp-zfp/tests/zfp_omp.cpp:12 (zfp_omp+0x40466a)
    #8 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Thread T29 (tid=41237, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race /host/build-gcc/_deps/zfp-src/src/zfp.c:747 in zfp_stream_set_bit_stream
==================
==================
WARNING: ThreadSanitizer: data race (pid=40760)
  Write of size 8 at 0x7b0c00000000 by main thread:
    #0 free <null> (libtsan.so.0+0x38c38)
    #1 zfp_stream_close /host/build-gcc/_deps/zfp-src/src/zfp.c:557 (libzfp.so.1+0xe656)
    #2 zfp::codec::zfp_base<2u, float>::~zfp_base() /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:31 (zfp_omp+0x406087)
    #3 zfp::codec::zfp2<float>::~zfp2() /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:286 (zfp_omp+0x406087)
    #4 zfp::internal::BlockStore<zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store.hpp:179 (zfp_omp+0x406087)
    #5 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~BlockStore2() /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:11 (zfp_omp+0x406087)
    #6 zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::~const_array2() /host/build-gcc/_deps/zfp-src/include/zfp/constarray2.hpp:70 (zfp_omp+0x406087)
    #7 test() /workspaces/cpp-zfp/tests/zfp_omp.cpp:29 (zfp_omp+0x406087)
    #8 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

  Previous read of size 8 at 0x7b0c00000000 by thread T29:
    #0 zfp::codec::zfp_base<2u, float>::clone_stream() const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:141 (zfp_omp+0x403f4d)
    #1 zfp::codec::zfp_base<2u, float>::decode_block(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:165 (zfp_omp+0x403f4d)
    #2 zfp::codec::zfp2<float>::decode_block(unsigned long, unsigned int, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/codec/zfpcodec.hpp:299 (zfp_omp+0x403f4d)
    #3 zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4>::decode(unsigned long, float*) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/store2.hpp:107 (zfp_omp+0x403f4d)
    #4 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::line(unsigned long, unsigned long, bool) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:176 (zfp_omp+0x403f4d)
    #5 zfp::internal::BlockCache2<float, zfp::internal::BlockStore2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/cache2.hpp:59 (zfp_omp+0x403f4d)
    #6 zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> >::get(unsigned long, unsigned long) const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/view2.hpp:405 (zfp_omp+0x403f4d)
    #7 zfp::internal::dim2::const_handle<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::get() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/handle2.hpp:28 (zfp_omp+0x403f4d)
    #8 zfp::internal::dim2::const_reference<zfp::internal::dim2::private_const_view<zfp::const_array2<float, zfp::codec::zfp2<float>, zfp::index::hybrid4> > >::operator float() const /host/build-gcc/_deps/zfp-src/include/zfp/internal/array/reference2.hpp:19 (zfp_omp+0x403f4d)
    #9 test() [clone ._omp_fn.0] /workspaces/cpp-zfp/tests/zfp_omp.cpp:24 (zfp_omp+0x403f4d)
    #10 gomp_thread_start <null> (libgomp.so.1+0x1d245)

  Thread T29 (tid=41237, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x61748)
    #1 gomp_team_start <null> (libgomp.so.1+0x1d8a3)
    #2 main /workspaces/cpp-zfp/tests/zfp_omp.cpp:31 (zfp_omp+0x4034c7)

SUMMARY: ThreadSanitizer: data race (/lib64/libtsan.so.0+0x38c38) in free
==================
ThreadSanitizer: reported 7 warnings

[lindstro]

Your OpenMP example is expected to fail as you're spawning multiple threads in the inner foor loop that are accessing the same private view. zfp's private views allow you to access the same array on N threads by creating N private views of the array, with one view per thread. See the example I provided in #198.

Returning to your TBB example, I missed the -c at the end of the line and thought this was the link line when I saw the references to .o files. Now, I guess we don't know to what extent TBB and OpenMP threads can be mixed; maybe what you're trying to do simply isn't supported. One guarantee we need is that each TBB thread gets assigned a thread-local private view, and I don't know enough about TBB to tell if that's the case (though I'd be surprised if that wasn't the case). From a performance standpoint, I see no reason to create a new private view in the outer loop; I would just pull that outside the i loop. Still, that should have no impact on the data race.

I think for now, you're stuck with OpenMP until we can research more whether and how to mix thread libraries. But it would surely be cleaner to avoid any such mixing and to use TBB primitives for any critical regions in zfp entered by TBB threads.

[lindstro]

Your OpenMP example is expected to fail as you're spawning multiple threads in the inner foor loop

My bad--I missed that the inner loop does not have a parallel directive, so your code is fine. I can confirm a reported data race with -fsanitize=thread as well as with my example from #198, so I believe more investigation is needed on our side. I will mark this as a bug for now.

[lindstro]

Doing some more research, it appears that ThreadSanitizer is to blame. It throws false positives on OpenMP code that clearly is thread safe. For instance, on the example from #198, ThreadSanitizer flags a line a code within the critical region as a data race, which obviously is nonsensical.

See also these links for even simpler examples of false positives:
https://stackoverflow.com/questions/33004809/can-i-use-thread-sanitizer-for-openmp-programs
https://stackoverflow.com/questions/59913536/race-condition-in-openmp-outside-parallel-block-threadsanitizer-false-positiv

Until it can be shown that there truly are some issues with zfp thread safety (e.g., segmentation faults, memory corruption, unexpected/inconsistent values when accessing arrays through private views, etc.), I will assume that the issue is with ThreadSanitizer and not with zfp.

[aavbsouza]

Hello @lindstro the original motivation for the issue was a segmentation fault on a code when running in parallel (with oneapi::tbb), the snippet used on this thread is based on a pattern that appear on this code. Others sections of this same code I was able to run with thread sanitize enabled without generating false positives. However the section in that code that is similar to the above snippet when executed with the thread sanitize enabled presents the same kind of errors show on this thread. And the problem only manifests if the parallelism is enabled.

[lindstro]

From the discussion above, we do expect issues when using zfp with TBB if the code is not compiled with OpenMP enabled. Do the segmentation faults persist if you compile with -fopenmp? Given that ThreadSanitizer gives false positives for the most basic OpenMP example, I think we cannot rely on it to diagnose data races. To make further progress, I would need a complete--but minimal--code example that invokes provably erroneous behavior.

[lindstro]

@aavbsouza Are you still having issues with zfp's private views and OpenMP? From our previous discussion, I think TBB is off the table, while the views appear to be working correctly when compiled with OpenMP support. If this is no longer an issue, then I'd like to go ahead and close it.

[lindstro]

I'm being assured by colleagues at Intel that it's safe to mix TBB and OpenMP, so all should work as long as the application is built with -fopenmp. I'm closing this issue for now, but feel free to reopen in case this issue is not resolved.