Comments (12)
Out of curiosity, try running as root or sudo. May be a permissions thing.
from lrzip.
Funny enough I actually has elavated the terminal with sudo permission because it is needed for the Proxmox vzdump tool which I am piping to lrzip.
So I should already have sudo permissions.
Thank you for looking into it.
from lrzip.
from lrzip.
I tried not piping just to get an idea if this would make a difference.
Might as well try i suppose, unfortunately it did not work.
I do not know how else to make this example then lrztar.
lrzip requires to pipe through tar and the idea was not to pipe it.
vzdump has no other way than piping the output.
And with both vzdump... | lrzip...
it seemed to not use the p4 threads as my fist post.
In this example with lrztar
lrztar -L 9 -p2 -o /home/proxmox/Compress\ example/Test1-lrzip-L9-p4.lrzip /home/proxmox/Compress\ example/Syncthing/
As the image shows it did not use only 2 threads and 200% CPU usage.
It takes anything that is available to it.
This is done on two seperate debian 12 systems.
To make the test more interesting.
That is why it is with 4 and 2 threads.
Is there anything here that makes sence to you?
from lrzip.
from lrzip.
I can see making a tar actually makes it respect the -p4 threads.
But i am fairly certain that is not gonne be possible for Proxmox.
I had dealy hoped so.
They dont create a tar, then compress it.
It seems they are using vzdump to copy the data to their compression module on the fly.
Would of course never have been certain they would have implemented it.
But i do feel that proxmox is lacking some space conserving compression and not a "fast as possible" compression.
from lrzip.
vzdump 100 --compress 0 --stdout | tar -O -cf - | lrzip -L 9 -p4 -o /Proxmox-BackUp/Manual-vzdump/vzdump-tar-lrzip-L9-p4.lrzip
You completely missed the point and my directions. In order to restrain threads, I advised you to NOT pipe to lrzip. You continue to do so. What you need to do is this:
vzdump 100 --compress 0 --stdout | tar -O -cf vzdump.tar
then
lrzip -L 9 -p4 -o /Proxmox-BackUp/Manual-vzdump/vzdump-tar-lrzip-L9-p4.lrz vzdump.tar
If you do not understand pipes, then linux administration may not be right for you.
from lrzip.
Ah, I just remembered, looking at the lzma code. Each backend in lrzip
may use threads as well. lzma may use up to 32 threads. However, the -p
option in lrzip
will control how large the block of data sent to the backend will be. Example:
If a chunk of data is 90:
- And the number of threads is 9, then each data block sent to the backend will be 10 (90/9).
- And the number of threads is 4, then each data block sent to the backend will be 22.5 (90/4).
- And the number of threads is 2, then each data block sent to the backend will be 45 (90/2).
This, of course, if memory and sliding window permits. lrzip or lrzip-next cannot control the backend use of threads over and above what lrzip uses.
The larger the data block, the better backend compression may be because it has more data to look ahead at.
from lrzip.
I know I started by making a mistake by misunderstanding how to pipe it the right way.
But I then edited the original thread yesterday after my mistake.
I did as you just wrote made an vzdump to a .tar file.
And the used lrzip to compress the tar file.
And it actually worked.
It respected the 4 threads as you made a theory about.
Plz. Have a look at the edited post above your last post.
I didn't write the command I used when I edited, maybe I should have.
But your idea worked.
I appreciate your time.
I just do not believe I could ever convince the proxmox team to implement it in this way.
But I still find lrzip as an excellent and useful compression tool.
Bedt regards,
Darkyere
from lrzip.
Funny we wrote at the same time.
I will have a look at what u just posted later.
I am unable to sit down ATM. Since I am on a phone.
from lrzip.
Yes. it appears that threads are uncontrolled when passed to a backend. This will be an interesting problem since we do not want to constrain the backend from doing its work. I am not sure if this is controllable atm. Good luck, I am done here.
from lrzip.
Yes, I was right. If you really want to constrain threads while using lzma, you must use the make option -DZ7_ST (for lzma sdk 23.01 and lrzip-next), -D_7ZIP_ST for lrzip. This will force lzma to NOT use multi threading. So, lrzip/lrzip-next is performing as designed. However, if you do constrain backend threads, you will a) hinder performance, and b) maybe reduce compression.
from lrzip.
Related Issues (20)
- How to read the output HOT 1
- lrzip -t file.lrz fails when run from write-protected dir
- Streaming issue "No space left on device" in lrzip 0.651 HOT 1
- "Warning, low memory for chosen compression settings" for small target files HOT 12
- lrzip 0.651 test issue HOT 3
- Use of uninitialized memory bug HOT 1
- 41e8014 Add a -Q/--very-quiet option: Suppresses INFO. It shouldn't
- memory error in fill_buffer lrzip/stream.c HOT 1
- ZPAQ Segfault with incompressible blocks
- lrztar shows spurious "illegal option" with long options HOT 4
- heap-buffer-overflow in libzpaq/libzpaq.cpp:1208:25 libzpaq::PostProcessor::write(int) HOT 3
- Incomplete fix of Issue #206 makes use-after-free still possible HOT 4
- "Unable to allocate enough memory for operation" on x86 when 10 threads are used. HOT 1
- Unable to stat file error on broken symlinks HOT 2
- Suggestion to support encrypted Stdin/Stdout when password provided on command line
- CPU detection does not account for CPU affinity HOT 8
- autoconf generates warnings on deprecated macros
- autoreconf: not found
- Makefile.am:34: error: Libtool library used but 'LIBTOOL' is undefined HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lrzip.