Comments (9)
If it returns io.ErrUnexpectedEOF
the data is NOT good.
from compress.
@klauspost Thanks for your timely repsonse.
Well, actually the data is really simple:
var s = "hello, i am zstd!"
input = enc.EncodeAll([]byte(s), nil)
So far, I use err != nil && err != io.ErrUnexpectedEOF
to check to work around this problem.
By the way, dec.DecodeAll(input, nil)
, I read this: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#frame_content_size,
it means the compressed data may contain the original size frame_content_size
, it's optional.
Does it mean I could handle off the memory allocation task to the decoder if I do compress and decompress using this same package.
from compress.
package main
import (
"fmt"
"github.com/klauspost/compress/zstd"
)
func main() {
var s = "hello, i am zstd!"
enc, err := zstd.NewWriter(nil)
if err != nil {
panic(err)
}
input := enc.EncodeAll([]byte(s), nil)
dec, err := zstd.NewReader(nil)
if err != nil {
panic(err)
}
output, err := dec.DecodeAll(input, nil)
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
from compress.
Please provide an actual reproducer.
from compress.
I know what happended now.
Actually, the data compressed is generated by following:
input := []byte("hello! my name is zstd!")
enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest))
require.Nil(t, err)
buf := bytes.NewBuffer(nil)
enc.Reset(buf)
_, err = enc.Write(input)
require.Nil(t, err)
err = enc.Flush() ===> enc.Close()
require.Nil(t, err)
bb := buf.Bytes()
println(len(bb))
I'm sorry I wrote several tests and forgot it's this one caused this unexpected behavior yesterday.
I made a mistake here, change enc.Flush()
to enc.Close()
solved my problem. I misunderstood the streaming and block mode.
Thanks.
from compress.
Yes. Flush does not indicate EOF - it merely pushes out what you have already sent.
from compress.
I realise this is closed but I see the same and I reuse encoders using the Reset() method. How do avoid this issue without calling Close()? Wouldn't calling Close() remove the advantages of using Reset()?
from compress.
No. Encoders are kept after Close as noted in the docs
Please open a QA topic in https://github.com/klauspost/compress/discussions if you need more details.
from compress.
No. Encoders are kept after Close as noted in the docs
Please open a QA topic in https://github.com/klauspost/compress/discussions if you need more details.
Yes, thanks, I realised I was reading the reader docs :) Appologies!
from compress.
Related Issues (20)
- s2: Read skippable block at the beginning of the stream
- all goroutines are asleep - deadlock HOT 4
- zstd: Another data corruption on SpeedBestCompression level HOT 7
- Memory Leak HOT 1
- s2 compression panic: index out of range [0] with length 0 HOT 3
- zstd: make use of RLE_Block
- matchLen redeclared in this block HOT 1
- Support go 1.22? HOT 2
- zstd: compression at concurrency 1 produces 2x bigger compressed data HOT 2
- investigate lz4 conversion crash
- BuildZStdDict Testing HOT 6
- zstd: Reuse single encoder/decoder with many dictionaries
- Zstd io.ReadCloser made with ZipDecompressor() doesn't consistently consume all compressed data HOT 3
- Add LZ4 support HOT 1
- is it ok to not close zstd reader?
- Entries being sorted after reading a zip file HOT 1
- zstd EncodeAll cost smaller cpu than zstd.NewWriter.Write() HOT 1
- build error HOT 1
- zstd decoder sometimes thinks a good file is corrupted HOT 6
- zip: support different compression types
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from compress.