Comments (10)
Correct. (and excellent debugging by the way)
Zstd is not yet protected against invalid input.
This is an objective, obviously.
Part of the work is done, since FSE seems to properly handle invalid input.
Now, Zstd proper must receive the same attention.
from zstd.
I found a variation of this crash that is most likely exploitable by people with malicious intent. Let me know where I can send the details, or I can publish the details here.
from zstd.
Hi Brian
It's just a bit too soon.
Zstd is not yet protected against such scenario, it's that simple.
Investigating specific situations will be useful later on, when protection code will be settled.
I'll keep this issue opened to track progresses.
from zstd.
#22 contains some interesting testing code initiated by @luben
from zstd.
Latest version of Zstd is now supposed to be safe against invalid input.
from zstd.
Not completely though. Here's another example file:
00000000 fd 2f b5 1e 00 00 31 40 00 20 31 32 32 32 32 32
00000010 32 31 33 31 32 33 31 31 31 32 32 32 32 32 33 31
00000020 32 32 31 31 32 32 32 32 32 0a 03 00 20 00 00 00
00000030 00 a1 00 10 00 08 42 10 c0 00 00
If you'll run zstd -d
on it (as of commit 79d3b1e), it will segfault with this backtrace:
#0 0x000000000040c9ac in FSE_read64 (memPtr=0x70101103b) at fse.c:202
#1 0x000000000040c8f5 in FSE_readLE64 (memPtr=0x70101103b) at fse.c:277
#2 0x000000000040474b in FSE_readLEST (memPtr=0x70101103b) at fse.c:311
#3 0x0000000000404a1e in FSE_reloadDStream (bitD=0x7fffffffe470) at fse.c:1594
#4 0x0000000000404a7c in FSE_initDState (DStatePtr=0x7fffffffe490, bitD=0x7fffffffe470, dt=0x801006000) at fse.c:1604
#5 0x000000000040a5d9 in ZSTD_decompressSequences (ctx=0x801006000, dst=0x801032000, maxDstSize=524288, seqStart=0x801011023, seqSize=14, litStart=0x801011003 "1222222131231112222231221122222\n\003", litSize=32) at ../lib/zstd.c:1543
#6 0x000000000040a387 in ZSTD_decompressBlock (ctx=0x801006000, dst=0x801032000, maxDstSize=524288, src=0x801011000, srcSize=14) at ../lib/zstd.c:1591
#7 0x000000000040a1df in ZSTD_decompressContinue (dctx=0x801006000, dst=0x801032000, maxDstSize=524288, src=0x801011000, srcSize=49) at ../lib/zstd.c:1734
#8 0x00000000004167d5 in FIO_decompressFilename (output_filename=0x4181d6 "-", input_filename=0x7fffffffec50 "--file-path-here--") at fileio.c:362
#9 0x00000000004174ee in main (argc=3, argv=0x7fffffffe998) at zstdcli.c:334
BTW, I'm using this fuzzer to find these crashes; I fully recommend you to use it too.
from zstd.
Thanks @magv, this is exactly the kind of feedback I'm looking for right now.
Thanks also for the link, I did not know this tool, and I'll certainly have a look at it !
from zstd.
This specific error fixed into fee8e24
from zstd.
Thanks for the link. AFL is really an excellent fuzzer tool.
It managed to find 2 more bugs that internal fuzzer tools did not triggered yet.
What's more, it did it without knowing the internal data format, just "guessing" the edges, and progressing along promising test cases. This is truly impressive.
After a few hours, it did not found any new bug, so I guess we are starting to see a relatively robust release ...
from zstd.
Yeah, AFL is magical. High yield with low effort? Yes, please.
I also see no crashes in the most recent commit, which is great, but I'd like to leave the fuzzer running for a while longer (a few more days would be good, I think).
from zstd.
Related Issues (20)
- ZDICT_trainFromBuffer_cover is not thread safe HOT 17
- zstd compression output differens with the same options between 1.5.5 and 1.5.6 HOT 5
- Warning message for `zstd -v --train` is missing line breaks
- How to accelerate the process of dictionary training in zstd? HOT 5
- tests/cli-tests/cltools/zstdless.sh fails with newer version of less HOT 3
- Please promote thread pools from experimental to stable HOT 1
- The CMake build script breaks check_ipo_supported
- Dynamic decompression HOT 3
- Change `dictionary_compression.c` example to use API for dictionary creation
- Enable weak symbol support for Risc-V? HOT 1
- Possibly missing check for truncated initial states in Huffman weight block HOT 4
- Poor compressor behavior on interleaved data HOT 2
- zstd 1.5.5+ has worse performance on Graviton2 nodes than v1.4.4 HOT 4
- [Not a bug] Dictionary building strategy HOT 7
- CLI: Hang bomb with with crafted circular symbolic link causes "zstd -d -r -f" to infinitely loop. "pigz -d-r -f" skips symbolic links with non compressed suffix
- ZSTD Decompression Speed vs Compression Level HOT 3
- is there a pre-trained version for "english" dictionary for web use? HOT 1
- patch-from in single-thread mode unexpected worse performing than multithread HOT 2
- Consider new release, with #4019 applied
- [reproducibility] Zstd varies compressed output based on the presence of SSE / NEON in some cases HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zstd.