Zstd io.ReadCloser made with ZipDecompressor() doesn't consistently consume all compressed data about compress

[klauspost]

I wonder how does it know to consume that final empty block without peeking ahead (or does it?)

TBH I haven't looked into it. It could be that it is trying to always have data decompressed, so it decodes a block as soon as there is no more data available.

[klauspost]

The main issue is that ReadFull doesn't read until EOF if the array can be satisfied.

If you change the last part to be...

	decomp, err := io.ReadAll(zstdReader)
	if err != nil && err != io.ErrUnexpectedEOF {
		panic(err)
	}
	if sampleLen != len(decomp) {
		panic(fmt.Sprintf("Zstd Length mismatch %d != %d", sampleLen, len(decomp)))
	}
	if !bytes.Equal(decompGozstd[:sampleLen], decomp[:sampleLen]) {
		panic("Decoded data doesn't match")
	}

Everything will be read from the input.

My guess is that "valyala" adds an empty block with a Last_Block identifier. This matches with the 3 bytes you see. I try to add this to the actual last block to not waste 3 bytes.

Since there is enough data to satisfy your request without decoding more, it will not decode more blocks until the data is actually needed.

You can "reintroduce" the issue by doing decomp, err := io.ReadAll(&io.LimitedReader{R: zstdReader, N: int64(sampleLen)}) - since the LimitedReader will stop reading upstream once N has been reached.

It is however not my impression the ZipDecompressor forces this and creates the problem by itself.

[vsivsi]

That all makes perfect sense, thanks so much for quickly responding. Since for my application I know the exact upper bound length of the decompressed data, I can easily work around this by just ensuring the read buffer is strictly longer than that upper bound.

Given your explanation, the most surprising thing is that the gozstd generated io.Reader does consistently consume the input stream to EOF when used with io.ReadFull(). I wonder how does it know to consume that final empty block without peeking ahead (or does it?)