weltkante / managed-lzma Goto Github PK

The line
"writer.InitializeLzma2Encoder();" not able compiled.

I change it to
"writer.ConnectEncoder(new ArchiveWriter.Lzma2Encoder(1, null));"
then works fine.

As mentioned in issue #12 the current implementation of ArchiveFileModelMetadataReader is incomplete. It stores the file metadata while reading it but does not build the corresponding ArchiveFileModel to expose it.

Hello,
I am trying to use your library for 7z file reading in my project.
Here is the code:

            ArchiveFileModelMetadataReader mdReader = new ArchiveFileModelMetadataReader();
            ArchiveFileModel mdModel = mdReader.ReadMetadata(stream);
            DecodedSectionReader dsReader = new DecodedSectionReader(stream, mdModel.Metadata, 0, PasswordStorage.Create(streamname));
            while (dsReader.CurrentStreamIndex < dsReader.StreamCount)
            {
                    // . . .
                    dsReader.NextStream();
            }

But it throws InvalidDataException in PrepareMetadata(StreamScope scope).

I have attached the sample archive (7z is in ZIP archive
RIH3.zip
).

Even though the webpage at http://www.7-zip.de/sdk.html is still at v9.22 (the version I'm currently using) there are newer versions on the project site http://sourceforge.net/projects/sevenzip/files/

Things to do:

• Update the copy of the LZMA SDK which is included in the project for running side-by-side tests
• Create diffs of the intermediate versions and check them for bugfixes and other relevant changes
• Check the relevant files in the 7z implementation, in case anything changed in the file format

Compared with the C# version of the LZMA SDK the Decoder class is apparently slower. Should investigate the reason, it shouldn't have to be slower, it's using the same algorithm after all. (Originally reported in #3)

If 7z archive already contains folder/directory it fails while reading metadata

• throws InvalidDataException exception in ArchiveMetadataReader.

Class: ArchiveMetadataReader
Method: ReadSectionDetails()
System.IO.InvalidDataException: 'Found invalid data while decoding.'

I created folder & archive using managed-lzma/sandbox-7z/Program.cs
Just added *ABCfolder* (before line 116) while creating archive, 7z file created successfully
I checked 7z by unpacking using other tools.

If you have a strict limit on archive size you need a way to decide whether you can add another file without exceeding your limit. Since the LZMA buffers can be very large this is very hard to predict without assistance of the library because you have to assume the worst case scenario, which is uncompressible files. If the files did actually compress then the worst case estimate means closing the archive way before reaching your strict limit.

It would be nice if the library could expose a way to inspect the amount of data in the buffers and give lower and upper bounds on the output length under the assumption that no further input follows.

Currently the library surfaces the internal implementation of the LZMA port as public classes and requires to use them directly. This should be replaced by a public API which fits more into the C# world. The currently visible classes will be made internal.

Same goes for the 7z implementation, it currently is just a prototype and as such the public API has been a mess so far.

Easily reproducible, decompressing a zero lenght file causes an exception. (no password file, LZMA2, 7zip archive)

While System.IO.Stream uses long data type for determining the length of the stream and seeking in it, which allows processing streams larger than 2GB, BufferedStream, which is a wrapper over System.IO.Stream uses int for its private fields mOffset, mEnding, and mLength. Due to using checked keyword in the ctor when casting long to int, processing of streams larger than 2GB ends with an OverflowException.

There is apparently no reason for using int and, therefore, it should be changed. The changes in BufferedStream are as follows.

1. Lines 18-20:
   private int mOffset;
   private int mEnding;
   private int mLength;
   =>
   private long mOffset;
   private long mEnding;
   private long mLength;

2. Line 31
   mLength = checked((int)stream.Length);
   =>
   mLength = stream.Length;

3. Line 79
   mOffset = (int)value;
   =>
   mOffset = value;

4. Line 121
   count = mBuffer.Read(buffer, offset, Math.Min(count, mEnding - mOffset));
   =>
   count = mBuffer.Read(buffer, offset, (int)Math.Min((long)count, mEnding - mOffset));

Modify the sandbox-7z project and add a file more than 20mb to the debug folder

Zip file with LZMA2 and AesEncoder
node1 = encoder.CreateEncoder(new ManagedLzma.SevenZip.Writer.Lzma2EncoderSettings(new ManagedLzma.LZMA2.EncoderSettings() { })); node2 = encoder.CreateEncoder(new ManagedLzma.SevenZip.Writer.AesEncoderSettings(ManagedLzma.PasswordStorage.Create("test")));

But I got exception when I try to readArchive
Non-negative number required. Parameter name: count

trace:
at System.Buffer.BlockCopy(Array src, Int32 srcOffset, Array dst, Int32 dstOffset, Int32 count) at ManagedLzma.LZMA2.Decoder.ReadOutputData(Byte[] buffer, Int32 offset, Int32 length) in C:\Project\Code\OpenSource\managed-lzma-master\shared\Compression\Lzma2\Decoder.cs:line 122 at ManagedLzma.SevenZip.Reader.Lzma2ArchiveDecoder.Read(Byte[] buffer, Int32 offset, Int32 count) in C:\Project\Code\OpenSource\managed-lzma-master\shared\SevenZip\Decoders\Lzma2Decoder.cs:line 120 at ManagedLzma.SevenZip.Reader.Lzma2ArchiveDecoder.OutputStream.Read(Byte[] buffer, Int32 offset, Int32 count) in C:\Project\Code\OpenSource\managed-lzma-master\shared\SevenZip\Decoders\Lzma2Decoder.cs:line 19 at ManagedLzma.SevenZip.Reader.ArchiveSectionDecoder.Read(Byte[] buffer, Int32 offset, Int32 count) in C:\Project\Code\OpenSource\managed-lzma-master\shared\SevenZip\ArchiveDecoder.cs:line 207 at ManagedLzma.SevenZip.Reader.DecodedStream.Read(Byte[] buffer, Int32 offset, Int32 count) in C:\Project\Code\OpenSource\managed-lzma-master\shared\SevenZip\ArchiveReader.cs:line 272 at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize) at System.IO.Stream.CopyTo(Stream destination) at sandbox_7z.Program.Main() in C:\Project\Code\OpenSource\managed-lzma-master\sandbox-7z\Program.cs:line 169 at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args) at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args) at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()

Someone posted an archive over at adamhathcock/sharpcompress#73 which can't be unpacked, neither by sharpcompress nor my own managed-lzma implementation. The official 7z application seems to be able to unpack it.

Should investigate at what point my implementation diverges from the official source.

As the subject states - I have a data block which is 0x100043 in length that decompresses to 0x100000 fine in C++ (code below), but exceptions with a bad data error in Lzma2Dec_DecodeToDic() line 323 when using managed lzma.

It could be possible I am using the API incorrectly, that would be the first thing to verify.

The Lzma2Dec_DecodeToDic functions in managed lzma and C++ lzma 1900 are just different enough that there's no easy change to make things work. It looks like the C# is a direct port of a different version of the C++.

C++ decode (which works fine)

TEST_METHOD_CATEGORY( SimpleDecompressBlob, "SevenZip" )
	{
		Log( "7-zip decompress blob test" );
		SetWorkingDirectoryForTests();

		size_t source_length = 0;
		uint8* compressed_memory = LoadFileToMemory( "TestData/Binary.dat", &source_length );

		size_t dest_length = 1024 * 1024;

		uint8* dest = static_cast< uint8* >( calloc( 1, dest_length ) );

		ISzAlloc alloc = { SevenZipMalloc, SevenZipFree };
		ELzmaStatus status = LZMA_STATUS_NOT_SPECIFIED;
		SRes result = Lzma2Decode( dest, &dest_length, compressed_memory, &source_length, 12, ELzmaFinishMode::LZMA_FINISH_END, &status, &alloc );
		Assert::IsTrue( result == SZ_OK, L"Lzma2Decode returned error" );

		free( dest );
	}

C# code - which exceptions (for testing, I put this at the top of sandbox-7z Program.cs)

        public class FLzma2Data
        {
	        public byte[] SourceData = ( byte[] )null;
	        public int SourceOffset = -1;
	        public int SourceLength = -1;
	        public byte[] DestinationData = ( byte[] )null;
	        public int DestinationOffset = -1;
	        public int DestinationLength = -1;
	        public byte DictionarySize = 40;
        }

	[STAThread]
        static void Main()
        {
	        FLzma2Data data = new FLzma2Data();

	        FileInfo Info = new FileInfo( "d:/Temp/Binary.dat" );
	        BinaryReader reader = new BinaryReader( Info.OpenRead() );
	        data.SourceData = reader.ReadBytes( ( int )Info.Length );
	        data.SourceLength = ( int )Info.Length;
	        data.SourceOffset = 0;
	        reader.Close();

	        data.DestinationData = new byte[1024 * 1024];
	        data.DestinationLength = 1024 * 1024;
	        data.DestinationOffset = 0;

	        data.DictionarySize = 12;

	        ManagedLzma.LZMA2.Decoder decoder = new ManagedLzma.LZMA2.Decoder( new ManagedLzma.LZMA2.DecoderSettings( 12 ) );
	        decoder.Decode( data.SourceData, data.SourceOffset, data.SourceLength, data.DestinationLength, true );
	        decoder.ReadOutputData( data.DestinationData, data.DestinationOffset, data.DestinationLength );

I could send you the file, but I don't want to bog down the forum with a meg of uncompressable data =)

What are your recommendations?

Cheers
John

Any chance of a NuGet package for this library?

Now I found this code can compress files into a 7z file:
foreach (var file in directory.EnumerateFiles())
{
using (var fileStream = file.OpenRead())
{
var result = await session.AppendStream(fileStream, true);
metadata.AppendFile(file.Name, result.Length, result.Checksum, file.Attributes, file.CreationTimeUtc, file.LastWriteTimeUtc, file.LastAccessTimeUtc);
}
}

Now what I want is add folders, and files under the folder. How to do that? AppendDirectory? It will broken the 7z file.

Pleas help, thanks a lot.

While prototyping an AES encoder for issue #1 I noticed that the currently generated 7z archives don't include checksums of their files. That's ok because they are optional and for the previous use case (lzma support in duplicati) these weren't necessary because our callers were maintaining their own checksums. However for standalone 7z archives they are important to verify the integrity of the archive, in particular when streams are encrypted you need it to verify that the password was correct.

I would be happy to implement multithreaded extraction if you can point me in the right direction and perhaps there is a reason its not possible?

Dan

As noted in Issue #12 some decoders throw a NotImplementedException. Besides the PPMD decoder covered in said issue there are also decoders for DELTA and BZIP2 formats missing.

Since there is no concrete demand for these I'll delay them a bit, but they'll be included before the nuget packages is considered final.

After "fixing" issue #2 the sample deadlocks and does not terminate because the encoder threads are waiting for something. It has been a while since I worked on that part so the fix may have been wrong, or this may be a bug after all. Investigate.

The PPMD decoder is the only remaining code which requires 'unsafe' pointers for its implementation. Fixing this allows to run on platforms which don't support unsafe code, which is desireable for the portable platform builds.

An archive was created with 7-zip.exe 9.20 and 18.01
with pw and multiple files.

A NotSupportedException is thrown on the first file stream:
at ManagedLzma.SevenZip.Reader.BcjArchiveDecoder.DecodeInto(Byte[] buffer, Int32 offset, Int32 count)
at ManagedLzma.SevenZip.Reader.BcjArchiveDecoder.Read(Byte[] buffer, Int32 offset, Int32 count)
at ManagedLzma.SevenZip.Reader.DecodedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)

In issue #17 it was noted that there exists an unofficial extension of the 7z archive format which stores posix file attributes in metadata bits which are usually unused. Support for this was added by setting a flag of whether this extension should be enabled when parsing 7z metadata.

Feature request: there should be a way to extract posix file attributes when they are available.

Would be great to have some example code, thought I'd mention it..

Thanks for the lib!!

Windows 10 supports a new type of assembly, universal libraries. The advantage of these is that they can contain natively compiled code and are still usable for store apps of all kinds.

Since the C/C++ implementation of the LZMA encoders and decoders has a major speed advantage it may be useful to have an universal library implementation of them instead of using a managed implementation.

Now that .NET Standard 2.0 is finished I want to cover it with the nuget library.

The c# language now supports ref return as a language feature. I want to examine if this feature can be used to simplify code or improve performance.

I can see you support AesDecoderStream, do you support AesEecoderStream?
How can I user it if supported?

Thanks!