Comments (4)
luben
commented on July 27, 2024
Hm, it could have be interplay of GC and locking issue. I have to test something.
from zstd.
Cyan4973
commented on July 27, 2024
Hi @luben .
zstd streaming interface is supposed to be thread safe.
It only uses information from local context, so as long as you have one context per thread, there is no race-condition to worry about.
from zstd.
luben
commented on July 27, 2024
Yes, I looked over the code and there is no global state. I suspect that the JVM is not resprecing the buffer pinning and it shows under GC pressure only - it moves them while compression/decompression is in progress. I have to test this and possibly allocate off-heap memory for the buffers or use the buffered version that you provide already.
from zstd.
luben
commented on July 27, 2024
Yes, it is a problem with JVM not pinning the buffers, I have confirmed. I still have to find a proper solution. Closing it now as it is not a problem of the library.
from zstd.
Related Issues (20)
- Is it safe to move compression and decompression contexts between threads? HOT 1
- ZDICT_trainFromBuffer_cover is not thread safe HOT 17
- zstd compression output differens with the same options between 1.5.5 and 1.5.6 HOT 5
- Warning message for `zstd -v --train` is missing line breaks
- How to accelerate the process of dictionary training in zstd? HOT 5
- tests/cli-tests/cltools/zstdless.sh fails with newer version of less HOT 3
- Please promote thread pools from experimental to stable HOT 1
- The CMake build script breaks check_ipo_supported
- Dynamic decompression HOT 3
- Change `dictionary_compression.c` example to use API for dictionary creation
- Enable weak symbol support for Risc-V? HOT 1
- Possibly missing check for truncated initial states in Huffman weight block HOT 4
- Poor compressor behavior on interleaved data HOT 2
- zstd 1.5.5+ has worse performance on Graviton2 nodes than v1.4.4 HOT 4
- [Not a bug] Dictionary building strategy HOT 7
- CLI: Hang bomb with with crafted circular symbolic link causes "zstd -d -r -f" to infinitely loop. "pigz -d-r -f" skips symbolic links with non compressed suffix
- ZSTD Decompression Speed vs Compression Level HOT 3
- is there a pre-trained version for "english" dictionary for web use? HOT 1
- patch-from in single-thread mode unexpected worse performing than multithread HOT 2
- Consider new release, with #4019 applied
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zstd.