Comments (8)
Bonjour Christophe
Is the default level still intended to be level 1?
Yes.
I have (unfulfilled) long term plans for even faster modes,
that's what negative levels will be used for.
or is the whole zstd vs zstd_hc a thing of the past?
Well, I realized that having 2 separate functions, with zstd meaning the same as zstd_hc with a fixed level 1, did not helped API readability.
So decided to merge everything into a single ZSTD_compress()
What about level 0? The comment in zstd_static.h says that level 0 is "never used", but quick tests show that using compression level 0 works fine (and compress about the same as level 1)
All levels <= 0 are simply remapped to 1 internally. for now.
Are there plans to have standardized names for some compression levels, like "fast", "high", "ultra" and so on?
Nope.
Could change in the future if there is reasons to. But every wrapper should be able to select whichever value it wants.
For example, folly
selected fast : 1
, default : 1
, best : 19
Right now max level is 20 but it seems that it can change. If yes, do you plan to have some way to, at runtime, probe for the range of supported compression levels?
Yes.
The value ZSTD_MAX_CLEVEL
within zstd_static.h
is meant for this usage.
from zstd.
For example, folly selected fast : 1 , default : 1 , best : 19
Do you mean this? https://github.com/facebook/folly/blob/master/folly/io/Compression.h#L152
The idea of having values -1
, -2
, -3
remapped to whatever the min/default/max points in the range supported by the loaded library seems nice, but if you intend to indroduce negative levels, this would need be moved to some other values...
The value ZSTD_MAX_CLEVEL within zstd_static.h is meant for this usage.
It is currently a #define
which is not visible when you are consuming a dll
(at least from a managed language such as .NET). Or maybe there is a way but I'm not familiar with it. Currently, I have a constant in my .NET code which I synchronize with the latest version of the .h
when I rebuild the dll internally, but this is brittle. I don't have this issue with ZSTD_VERSION_...
because there is a ZSTD_versionNumber()
method.
from zstd.
It is currently a
#define
which is not visible when you are consuming adll
Good point.
from zstd.
Within latest "dev" branch update d608088, there is a new method ZSTD_maxCLevel()
to retrieve that value.
from zstd.
I have changed my code to query for ZSTD_maxCLevel()
on start, and I'm currently using an enum with 4 levels:
Default
= 0, but is converted to 1 internally before calling zstd.Fastest
= 1 (but could change?)Medium
= (1 + MAX) / 2Highest
= MAX
This gives Medium = 10 and Highest = 20 in the current version.
Having 0 for the default level helps in that it is also the default value for uninitialized variables and optional parameters in .NET, which works well in the API. This could work if you NEVER use 0 as a compression level (even if you start using negative values). If not then I could simply use a nullable integer in the API, which would also be acceptable.
If the minimum compression level can be less than 1 in the future, we would need a ZSTD_minCLevel()
method to query the value. Or maybe a single method that returns the bounds?
from zstd.
Having 0 for the default level helps
Agreed. This is the current convention, and is expected to remain like this. LZ4
uses the same convention.
If the minimum compression level can be less than 1 in the future, we would need a ZSTD_minCLevel() method to query the value.
Agreed, although this is just a "long term intention", with no precise plan.
Even if I introduce negative compression levels later on (which is not yet guaranteed), this won't change anything for "normal" existing compression levels, hence interface break.
So we'll have time to handle this issue if it ever becomes a reality.
from zstd.
Is there anything else to add, or could we close the topic ?
from zstd.
I think it's OK to close.
To sum up:
- I'm using 0 to mean "the default" which currently is 1
- I'm calling ZSTD_maxCLevel to get the max supported by the version of dll loaded at runtime.
- I provide a set of Fastest/Medium/High/Highest helpers that can be used optionally, but with guidance in the comments that app code should rely on testing for the best level and use the numerical value instead... or use default if they don't really care.
- I'm probably going to validate levels at the wrapper level, and throw if they are above this, instead of capping it silently at the MAX, because it's more inline with what a .NET developer would expect (most API validate enum-like parameters and throw).
- In the future if compression level can get negative, then maybe a ZSTD_minCLevel could be added to get the lower bound (but sill keep 0 as the default).
from zstd.
Related Issues (20)
- Add library and cli flags for file format with embedded dictionary
- Question about ZSTD protocole HOT 2
- Building on MacOS 13 and targeting MacOS 11 and SDK 11.3 (or any other MacOS version) does not work HOT 2
- Integrating the library with an external thread pool HOT 2
- Is it safe to move compression and decompression contexts between threads? HOT 1
- ZDICT_trainFromBuffer_cover is not thread safe HOT 17
- zstd compression output differens with the same options between 1.5.5 and 1.5.6 HOT 5
- Warning message for `zstd -v --train` is missing line breaks
- How to accelerate the process of dictionary training in zstd? HOT 5
- tests/cli-tests/cltools/zstdless.sh fails with newer version of less HOT 3
- Please promote thread pools from experimental to stable HOT 1
- The CMake build script breaks check_ipo_supported
- Dynamic decompression HOT 3
- Change `dictionary_compression.c` example to use API for dictionary creation
- Enable weak symbol support for Risc-V? HOT 1
- Possibly missing check for truncated initial states in Huffman weight block HOT 4
- Poor compressor behavior on interleaved data HOT 2
- zstd 1.5.5+ has worse performance on Graviton2 nodes than v1.4.4 HOT 4
- [Not a bug] Dictionary building strategy HOT 7
- CLI: Hang bomb with with crafted circular symbolic link causes "zstd -d -r -f" to infinitely loop. "pigz -d-r -f" skips symbolic links with non compressed suffix
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zstd.