Comments (6)
Thanks Hanno, this is exactly the kind of report which is needed right now. This issue requires a pretty impressive set of conditions to trigger (it must successfully pass a lot of prior checks to get there).
I'm a bit surprised because I ran afl for almost 24h before release, without finding any issue.
I initially thought it was a matter of randomness. Digging deeper though, I note that when introducing your test case into the initial sef of afl, afl just runs normally, it doesn't find any crash.
Impressively enough, I also note that this bug pass completely unnoticed by valgrind memory tester, which was my tool of choice up to now to grab memory issues. Only clang + address sanitizer can catch it. So I'll reconsider its usage into my CI environment.
I suspect this might be because the faulty operation happens a few bytes beyond the limit of the intended table, but still within the limit of the local stack (there are some other tables after this one).
Anyway, bottom line : only address sanitizer seems able to catch this issue.
I therefore tried to combine afl with address sanitizer (which comes with its own set of difficulties, due to virtual addressing), but afl stops during initialization phase, while trying to decode zstd-oob-stack-HUF_readStats. So it doesn't even start fuzzing.
So the question is : how did you find this issue ?
from zstd.
Okay, a bit of background:
It doesn't surprise me that valgrind can't find these issues, because that is virtually impossible, it's just a valid access to another valid memory address. A stack overflow can't be found by any runtime tool, because the bare assembly code has no knowledge of what is valid and what not, a tool needs to change the memory layout at compile time to be able to detect these issues.
The "secret" of running afl+asan combined is that asan requires lots (terabytes) of virtual memory, so this only works if you disable the afl memory limit (-m none). Also see my tutorial:
https://fuzzing-project.org/tutorial3.html
There are more advanced tricks to make afl+asan work together using cgroups, but I never bothered to test them because disabling the memory limit works just fine for me (although this carries a small risk of making your system unstable if the fuzzer finds inputs that will make your tested app run wild and allocate lots of memory).
from zstd.
Thanks for the link, this is a great read.
Indeed, using -m none
is what I attempted to make afl
work with asan, but another condition is that none of the initial seed files should make asan crash, otherwise fuzzing doesn't even start.
After that condition is fulfilled, fuzzing does proceed accordingly, just quite a bit slower due to asan checks. I'm just curious about what happens when asan detects a problem later on : does that make afl
quit immediately, or does it just add the sample to the crashes
directory as usual ?
from zstd.
The issue should be fixed into latest update of the "dev" branch
from zstd.
afl doesn't treat asan-crashes different from other crashes. So it will just collect crashes in the [outdir]/crashes directory. You'll usually get a couple of duplicates for the issues it finds.
from zstd.
also can confirm fix works, now testing with the new dev branch to see if it finds anything more.
from zstd.
Related Issues (20)
- Add library and cli flags for file format with embedded dictionary
- Question about ZSTD protocole HOT 2
- Building on MacOS 13 and targeting MacOS 11 and SDK 11.3 (or any other MacOS version) does not work HOT 2
- Integrating the library with an external thread pool HOT 2
- Is it safe to move compression and decompression contexts between threads? HOT 1
- ZDICT_trainFromBuffer_cover is not thread safe HOT 17
- zstd compression output differens with the same options between 1.5.5 and 1.5.6 HOT 5
- Warning message for `zstd -v --train` is missing line breaks
- How to accelerate the process of dictionary training in zstd? HOT 5
- tests/cli-tests/cltools/zstdless.sh fails with newer version of less HOT 3
- Please promote thread pools from experimental to stable HOT 1
- The CMake build script breaks check_ipo_supported
- Dynamic decompression HOT 3
- Change `dictionary_compression.c` example to use API for dictionary creation
- Enable weak symbol support for Risc-V? HOT 1
- Possibly missing check for truncated initial states in Huffman weight block HOT 4
- Poor compressor behavior on interleaved data HOT 2
- zstd 1.5.5+ has worse performance on Graviton2 nodes than v1.4.4 HOT 4
- [Not a bug] Dictionary building strategy HOT 7
- CLI: Hang bomb with with crafted circular symbolic link causes "zstd -d -r -f" to infinitely loop. "pigz -d-r -f" skips symbolic links with non compressed suffix
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zstd.