Comments (8)
Thanks for the bug report. You're correct in the strict sense, if the realloc fails, pixz will crash. Unfortunately, since pixz doesn't do any checking for out-of-memory conditions, ANY malloc/realloc failure will cause a crash.
I guess this could be 'fixed' by just going through the code and calling die() if any malloc fails, but that doesn't really help. Ideally we'd preallocate all memory, but that would be a huge and invasive change. Even more-ideally, we'd also try to prevent the system from swapping.
from pixz.
I think, it is good practice to use a tool like the abovementioned cppcheck. I am working on adding this to travis.
from pixz.
@vasi I have prepared a pull request that both adds a cppcheck test to the source directory (good to have this anyway) and fixes this issue by implementing a check on the result of the realloc function call.
If you are OK with this, please let me know or merge the pull request #42 yourself. I am asking you specifically because you have shown hesitance on the issue in a previous comment.
IMHO, in the end, it is always better to give the end-user a reasonable error message than letting pixz error out by itself while dereferencing NULL.
from pixz.
@vasi ping
from pixz.
Sorry, I didn't see this earlier. I think it's useful, you're certainly right that it's better to provide an error than just crash. Feel free to commit!
I do want to still push for a real solution, though:
- We can try to predict when memory will become a problem. The amount of memory pixz uses is roughly proportional to queue-size * buffer-size. If that number exceeds available physical memory, we should print a warning, and maybe scale down one of these factors.
- We can also attempt to do real recovery from OOM. For the index, we could flush what we have to a temp flle.
from pixz.
I have added a TODO comment so we can address this later. We should probably open a new issue to address OOM prevention and recovery as a future feature. I will do this.
from pixz.
SO pixz still has OOM issue?
from pixz.
OOM is usually considered to be a fatal error, because, in most cases, there is nothing you can do about it but crash. Often, applications do not even handle this error and let it fall through to the application caller, e.g. the terminal / shell process.
Being able to prevent OOM or even recover from it highly depends on the application. Whether or not we can do this at all in a sane way must be researched. We will keep you up-to-date on this progress in #44.
What I did here to solve this issue, is just checking the realloc
call and fail-fast with an appropriate error. This is a common C / C++ practice that you should apply as was reported initially by @bremner with the cppcheck tool.
from pixz.
Related Issues (20)
- configure: error: AsciiDoc not found, not able to generate the man page. HOT 3
- Error decoding stream footer when trying to decompress a 3.1 TiB .tpxz file HOT 8
- cppcheck 2.8 warnings about uninitialized variables
- Crash when using -x option
- What is the default level of compression? HOT 2
- Can't compile on Fedora 38 HOT 2
- msys2 build failure HOT 18
- Indexes HOT 2
- Clarify README section on differences with xz HOT 5
- any plans for another release soon? HOT 1
- Server mode HOT 1
- -k should be the default HOT 2
- concatenation of *xz files and then decompression using pixz HOT 2
- Building On Windows HOT 2
- manpage not installed if building from release tarball HOT 2
- build env question not package liblzma HOT 1
- Questions about tpxz / file index format HOT 6
- Error creating block encoder HOT 3
- Syntax for converting existing tar.xz archive to indexed pixz file? HOT 1
- Random failures when compressing large directories HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pixz.