status-im / nim-snappy

Nim implementation of Snappy compression algorithm

Snappy

Compression and decompression utilities for the snappy compression algorithm:

• Overview
• Format description

The main module, snappy, contains in-memory encoders and decoders:

• compress/uncompress work with caller-allocated buffers
  • No dynamic memory allocation (the functions require ~20kb stack space)
  • Exception-free
• encode/decode are convenience wrappers for the above that take care of memory allocation
  • Simplified error reporting
  • Suitable for small buffers mainly

Framed encodings are also supported via functions carrying the Framed suffix.

• Framing format

Stream support

The library supports compression and decompression for the following libraries

• faststreams
  • import snappy/faststreams
• std/streams
  • import snappy/streams (incomplete)

API

In-memory

import snappy

func compress*(
  input: openArray[byte],
  output: var openArray[byte]): Result[int, CodecError]
func encode*(input: openArray[byte]): seq[byte]
func uncompress*(input: openArray[byte], output: var openArray[byte]):
  Result[int, CodecError]
func decode*(input: openArray[byte], maxSize = maxUncompressedLen): seq[byte]

faststreams

⚠️ BETA API, subject to change

When using faststreams, errors are reported via exceptions.

Uncompressing raw snappy is not covered in streaming mode due to the requirement that full uncompressed data must be available during decompression.

import snappy/faststreams

proc compress*(input: InputStream, output: OutputStream)
proc compressFramed*(input: InputStream, output: OutputStream)
proc uncompressFramed*(input: InputStream, output: OutputStream)

std/streams

⚠️ BETA API, subject to change

import snappy/streams

proc compress*(input: Stream, inputLen: int, output: Stream)
# TODO compressFramed
# TODO uncompressFramed

Examples

import snappy
var source = readFile("readme.md")
var encoded = snappy.encode(toOpenArrayByte(source, 0, source.len-1))
var decoded = snappy.decode(encoded)
assert equalMem(decoded[0].addr, source[0].addr, source.len)

Performance

Generally, performance is on par with the C++ implementation, shown as cppLib.

Framed encoding is slower due to the extra CRC32C processing.

The table shows average time to compress data in ms on x86_64. Lower is better.

        inMemory,      fastStreams,       nimStreams,           cppLib,      Samples,         Size,         Test
  0.086 /  0.056,   0.087 /  0.000,   0.112 /  0.000,   0.088 /  0.029,          100,       102400, html
  0.117 /  0.093,   0.118 /  0.094,   0.000 /  0.000,   0.000 /  0.000,          100,       102400, html(framed)
  1.052 /  0.480,   1.073 /  0.000,   1.322 /  0.000,   1.005 /  0.335,          100,       702087, urls.10K
  1.260 /  0.775,   1.286 /  0.785,   0.000 /  0.000,   0.000 /  0.000,          100,       702087, urls.10K(framed)
  0.008 /  0.005,   0.022 /  0.000,   0.092 /  0.000,   0.008 /  0.005,          100,       123093, fireworks.jpeg
  0.051 /  0.047,   0.067 /  0.057,   0.000 /  0.000,   0.000 /  0.000,          100,       123093, fireworks.jpeg(framed)
  0.010 /  0.006,   0.021 /  0.000,   0.066 /  0.000,   0.009 /  0.005,          100,       102400, paper-100k.pdf
  0.046 /  0.050,   0.057 /  0.054,   0.000 /  0.000,   0.000 /  0.000,          100,       102400, paper-100k.pdf(framed)
  0.374 /  0.218,   0.378 /  0.000,   0.451 /  0.000,   0.357 /  0.118,          100,       409600, html_x_4
  0.491 /  0.386,   0.498 /  0.392,   0.000 /  0.000,   0.000 /  0.000,          100,       409600, html_x_4(framed)
  0.334 /  0.186,   0.345 /  0.000,   0.399 /  0.000,   0.331 /  0.126,          100,       152089, alice29.txt
  0.382 /  0.251,   0.392 /  0.251,   0.000 /  0.000,   0.000 /  0.000,          100,       152089, alice29.txt(framed)
  0.300 /  0.165,   0.311 /  0.000,   0.354 /  0.000,   0.300 /  0.114,          100,       129301, asyoulik.txt
  0.343 /  0.220,   0.352 /  0.222,   0.000 /  0.000,   0.000 /  0.000,          100,       129301, asyoulik.txt(framed)
  0.907 /  0.483,   0.932 /  0.000,   1.086 /  0.000,   0.880 /  0.327,          100,       426754, lcet10.txt
  1.053 /  0.675,   1.075 /  0.680,   0.000 /  0.000,   0.000 /  0.000,          100,       426754, lcet10.txt(framed)
  1.241 /  0.646,   1.272 /  0.000,   1.477 /  0.000,   1.201 /  0.466,          100,       481861, plrabn12.txt
  1.387 /  0.856,   1.425 /  0.861,   0.000 /  0.000,   0.000 /  0.000,          100,       481861, plrabn12.txt(framed)
  0.076 /  0.050,   0.075 /  0.000,   0.096 /  0.000,   0.076 /  0.025,          100,       118588, geo.protodata
  0.110 /  0.095,   0.112 /  0.098,   0.000 /  0.000,   0.000 /  0.000,          100,       118588, geo.protodata(framed)
  0.279 /  0.183,   0.287 /  0.000,   0.338 /  0.000,   0.273 /  0.121,          100,       184320, kppkn.gtb
  0.346 /  0.261,   0.354 /  0.263,   0.000 /  0.000,   0.000 /  0.000,          100,       184320, kppkn.gtb(framed)
  0.024 /  0.018,   0.026 /  0.000,   0.032 /  0.000,   0.024 /  0.014,          100,        14564, Mark.Twain-Tom.Sawyer.txt
  0.030 /  0.021,   0.031 /  0.021,   0.000 /  0.000,   0.000 /  0.000,          100,        14564, Mark.Twain-Tom.Sawyer.txt(framed)
 23.814 /  8.608,  27.362 /  0.000,  48.342 /  0.000,  22.157 /  6.958,           50,     38942424, state-2560000-114a593d-0d5e08e8.ssz
 36.075 / 25.389,  39.979 / 28.497,   0.000 /  0.000,   0.000 /  0.000,           50,     38942424, state-2560000-114a593d-0d5e08e8.ssz(framed)```

Installation via nimble

nimble install snappy

nim-snappy's Issues

Investigate oss-fuzz

Google runs infrastructure that automatically fuzzes libraries, and snappy has fuzzing set up - investigate whether snappy can be added as a project assuming they can treat nim like C.

https://google.github.io/oss-fuzz/getting-started/new-project-guide/