image-rs / jpeg-decoder Goto Github PK

Rayon introduces a lot of unsafe code to the dependency graph - at about 2000 unsafe expressions according to cargo-geiger, accounting for over 80% of all unsafe code. This undermines the value proposition of the safe Rust decoder: the entire point of it is implementing a decoder in safe code; if you're OK with depending on unsafe code that's probably fine, you should just use libjpeg-turbo.

It would be nice to experiment and see if we can match the performance of Rayon without pulling in all of its dependency tree.

Right now jpeg-decoder uses Rayon for just one line of code:

It should be fairly trivial to replicate what Rayon does here using manually spawned threads and an MPMC channel to distribute work items to the threads. flume is a 100% safe code implementation of an MPMC channel that's on par with crossbeam in the use cases that are relevant here. I'm not sure if that's the right approach, but it's probably worth a shot.

For progressive jpegs the caller should be able to get new data after each scan, until the last scan has been decoded and the final data is available.

http://www.filedropper.com/brokenjpeg3

decoder panics as: no marker found where RST was expected

url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAQAAAAECAQAAAAD+Fb1AAAAGUlEQVR4AWP8zwADxgzG/xmA3P9AAGLApQC0NgpfF9g/mgAAAABJRU5ErkJggg==")

Calling decode on the same decoder twice returns an error. I think this should just return the already decoded data because multiple calls to read_info does not create any errors. Besides being a minor inconsistency, I believe that anyone calling decoder again would naturally just expect it to give the decoded data and not return an error.

Hi,

I try to make use of this library to convert MJPG encoded frames to a format (RGB, RGBA or YUV) that can suite my needs for image processing . This is because on my Windows computer (and maybe others) the only webcam encoding that can provide (somewhat) high resolution with decent framerate is the MJPG one.

Anyway, I am using a modified version of Escapi which is a Rust library around some Windows API for Webcam capture (among other things). I managed to get the MJPG encoded frames buffer but while trying to decode them I got the following error:

Error is invalid JPEG format: scan makes use of unset dc huffman table

After dumping a frame into a file on looking at it in an hex editor I can confirm that the file has no huffman table section. The Windows image viewer cannot open the file but GIMP manage to do it. Plus, inserting some huffman tables before the "start of scan" section from another random jpeg (found on the web) allow me to see it in the Windows image viewer and to decode it using this library. So I am no expert in JPG nor data compression/decompression but it seems that while my buffers are not strictly valid jpeg files because they lack huffman tables, there is a way to decode nonetheless them. Do you think you will be able to add support for this case?

By the way, I can provide some files if you want to take a look at the MJPG buffers.

• walkdir (1 → 2)
• png (0.5 → 0.14)
• docopt (0.7 → 1)

I would appreciate if you'd update them. Thanks!

I let this fuzzer run for a while (> 250k iterations), and it discovered:

thread '<unnamed>' panicked at 'attempt to add with overflow'
[…]
jpeg_decoder::huffman::derive_huffman_codes in src/huffman.rs:277

You can find the full log as well as the used input here: https://gist.github.com/killercup/d836f174582ee5d90403e9920d893be2

Unable to decode this image:

I think the error is misleading, because the image has 8-bit precision (decodes fine with the default build of libjpeg-turbo).

use std::io::{self, Read, Cursor};
use std::{intrinsics};
use std::panic;

extern crate jpeg_decoder;

use jpeg_decoder::Decoder;

fn main() {
    let mut decoder = Decoder::new(b"\xff\xd8\xff\xc2\x00\x11\x08\x30\x30\x30\x30\x03\x01\x11\x00\x30\x11\x01\x03\x11\x01" as &[u8]);
    let a = decoder.read_info();
}

cargo run --release < out2
     Running `target/release/image-fuzz`
fatal runtime error: out of memory
error: An unknown error occurred

To learn more, run the command again with --verbose.

Found using afl.rs

This is a tracking issue for the potential improvements brought up in this comment. This requires use of slice::copy_within which is available only since 1.37.

Using image: https://i.imgur.com/nO4Jdc5.jpg (somewhat NSFW)

Code:

use std::fs::File;
use std::io::BufReader;

fn main() {
    let file = File::open("nO4Jdc5.jpg").expect("failed to open file");
    let mut decoder = jpeg_decoder::Decoder::new(BufReader::new(file));
    let pixels = decoder.decode().expect("failed to decode image");
    let metadata = decoder.info().unwrap();
}

Works with version =0.1.16, crashes with version =0.1.17. Error is thread 'main' panicked at 'index out of bounds: the len is 1010816 but the index is 1015928'. Seems to be a bug in single-channel decoding.

Here's a case that I believe is different from #133: Running this 504 byte input file through the decode function takes ~30 seconds in release mode (at about 170% CPU load according to htop) until it fails with Io(Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" }).

id:000010,sig:06,src:000016,op:flip32,pos:163.tar.gz

To reproduce, in the afl branch (#131), run:

$ time cargo run --release --bin reproduce_decode out/crashes/id:000010,sig:06,src:000016,op:flip32,pos:163
    Finished release [optimized] target(s) in 0.01s
     Running `target/release/reproduce_decode 'out/crashes/id:000010,sig:06,src:000016,op:flip32,pos:163'`
Decoder returned an error: Io(Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" })
Note: Not a panic, this is fine.

real	0m29.794s
user	0m50.215s
sys	0m2.310s

While running, according to htop, the process does not fill my system memory, that's why I think it's not limited to the "unchecked pre-allocation" issue.

(I hope I don't annoy you with all those issues. Without knowledge of the internals it's hard to judge which cases are duplicates and which cases aren't 🙂 Thanks for developing and maintaining jpeg-decoder!)

It would be useful if one could reuse memory between frames for decoding MJPEG from a webcam at FullHD, 30 fps.
Also it would be great if one could reuse the rayon threads across frames. Not restart them for every frame.
Currently I always have to compile in release mode to get realtime decoding..
In debug mode I don't even get 2 fps :(

At the moment a worker thread is used while decoding. However the emscripten targets (asm.js and WebAssembly) don't support threads, so a synchronous alternative implementation should be used for those targets. The worker thread could possibly just be executed on the main thread for example.

On a side note: rayon is used in other places, but that works just fine with emscripten targets, so supporting emscripten isn't all too big of a change.

https://boygeniusreport.files.wordpress.com/2012/08/iphone-5-mockup.jpeg fails to decode with

thread '<main>' panicked at 'decode failed: Format("did not find marker where expected")', ../src/libcore/result.rs:760

I'm using this library to detect valid jpeg images for generating an afl-rs logo, basically fuzzing whether an input stream decodes or not. While doing this I came across one image that passed the fuzzer in release mode, but then panicked when I tried loading in debug mode, testing with the provided decode example gives the same result:

$ cargo run --release --example decode -- crash.jpg
    Finished release [optimized] target(s) in 0.0 secs
     Running `target/release/examples/decode crash.jpg`

$ cargo run --example decode -- crash.jpg
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/examples/decode crash.jpg`
thread 'main' panicked at 'attempt to add with overflow', src/decoder.rs:551
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: Process didn't exit successfully: `target/debug/examples/decode crash.jpg` (exit code: 101)

This file does load (albeit with different output) in Preview and Safari.

I've been doing some testing with this library, which seems great, and have a couple questions:

• I've made a couple hacks locally to identify exif data - would you be interested in a PR adding this functionality and, if so, how would you prefer this to be structured? My default was to simply add a Vec<u8> to the decoder, and allow users to process it outside the library as they please.
• ultimately I would like to edit some of the exif and write the jpeg back out with the updated exif. i haven't been able to immediately find any projects that support encoding a jpeg with exif data - don't suppose you have any recommendations? :D

• I encountered spectral selection is not allowed in non-progressive scan Error.
• convert command in Linux displayed Invalid SOS parameters for sequential JPEG warning but the image was generated successfully.
• The images are generated by these hardwere in my case
  • samsung SCV36
  • samsung SC-02G
  • samsung SC-05G

may be relevant

• image-rs/image#738
• https://stackoverflow.com/questions/48653584/cannot-load-jpeg-with-java-created-by-samsung-phone

http://www.filedropper.com/test_17

thread 'main' panicked at 'failed to decode image: Format("no data found")', src/libcore/result.rs:997:5

Hi,

while decoding the attached picture (warning: atrocious child torture inside😄), I got this panic:

RUST_BACKTRACE=1 cargo run --example decode /home/xav/IMG_20160204_195730.jpg
     Running `target/debug/examples/decode /home/xav/IMG_20160204_195730.jpg`
thread 'main' panicked at 'decode failed: Format("RST(4) marker found where not allowed")', src/libcore/result.rs:788
stack backtrace:
   1:     0x557f362055ef - std::sys::backtrace::tracing::imp::write::h46e546df6e4e4fe6
   2:     0x557f36208a9b - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h077deeda8b799591
   3:     0x557f3620871f - std::panicking::default_hook::heb8b6fd640571a4f
   4:     0x557f361fe82e - std::panicking::rust_panic_with_hook::hd7b83626099d3416
   5:     0x557f36208ce1 - std::panicking::begin_panic::h941ea76fc945d925
   6:     0x557f361ff0fa - std::panicking::begin_panic_fmt::h30280d4dd3f149f5
   7:     0x557f36208c7e - rust_begin_unwind
   8:     0x557f3623b04f - core::panicking::panic_fmt::h2d3cc8234dde51b4
   9:     0x557f3614fe4d - core::result::unwrap_failed::h946c0fb8477dc56c
                        at /build/rustc-5moGoL/rustc-1.11.0+dfsg1/src/libcore/macros.rs:29
  10:     0x557f3614fc39 - _<core..result..Result<T, E>>::expect::h65575c6cf47ed0a8
                        at /build/rustc-5moGoL/rustc-1.11.0+dfsg1/src/libcore/result.rs:750
  11:     0x557f36149bc4 - decode::main::h25b6bb75384c0f31
                        at /home/xav/rust/jpeg-decoder/examples/decode.rs:57
  12:     0x557f36208328 - std::panicking::try::call::hca715a47aa047c49
  13:     0x557f3621018b - __rust_try
  14:     0x557f3621012e - __rust_maybe_catch_panic
  15:     0x557f36207dce - std::rt::lang_start::h162055cb2e4b9fe7
  16:     0x557f361973b9 - main
  17:     0x7fc5433902b0 - __libc_start_main
  18:     0x557f36148ca9 - _start
  19:                0x0 - <unknown>
error: Process didn't exit successfully: `target/debug/examples/decode /home/xav/IMG_20160204_195730.jpg` (exit code: 101)

img_20160204_195730

Please release a new version with at least commit 32f365a.
Without I can't build Thinkofname/steven-rust

Found this via fuzzing (see #131).

First of all, I'm not totally sure how to interpret these results.

For the following test input (1.4 KiB):

id:000004,sig:06,src:000000,op:flip32,pos:874.tar.gz

...the afl fuzzer reported a crash. When running the reproducer on it, however, it runs fine:

$ cargo run --bin reproduce_decode out/crashes/id\:000004\,sig\:06\,src\:000000\,op\:flip32\,pos\:874
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
     Running `target/debug/reproduce_decode 'out/crashes/id:000004,sig:06,src:000000,op:flip32,pos:874'`
Decoder returned an error: Format("failed to decode huffman code")
Note: Not a panic, this is fine.

$ cargo run --release --bin reproduce_decode out/crashes/id\:000004\,sig\:06\,src\:000000\,op\:flip32\,pos\:874
   Compiling fuzz-target-jpeg-decoder v0.1.0 (/home/danilo/Projects/jpeg-decoder/fuzz-afl)
    Finished release [optimized] target(s) in 2.25s
     Running `target/release/reproduce_decode 'out/crashes/id:000004,sig:06,src:000000,op:flip32,pos:874'`
Decoder returned an error: Format("failed to decode huffman code")
Note: Not a panic, this is fine.

By default, AFL uses a 50 MiB memory limit, so this was likely exceeded.

When running the target through valgrind with --pages-as-heap=no, the program quickly fills my system memory (roughly 20 GiB free) and is then killed by the OOM killer.

$ valgrind --tool=massif --pages-as-heap=no --massif-out-file=massif.out target/release/reproduce_decode out/crashes/id\:000004\,sig\:06\,src\:000000\,op\:flip32\,pos\:874
==1046640== Massif, a heap profiler
==1046640== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==1046640== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1046640== Command: target/release/reproduce_decode out/crashes/id:000004,sig:06,src:000000,op:flip32,pos:874
==1046640==
Killed

When running the target through valgrind with --pages-as-heap=yes, I get the following massif report: massif.out.tar.gz When opening it with massif-visualizer, it reports a peak virtual memory usage of 24 GiB.

I'm not entirely sure if that is a false positive by valgrind, since just running the target without it returns an error without filing system memory, however something is probably still not right with the parser. I hope this issue is still helpful, maybe someone else can figure out what's going on 🙂

This SO question may be helpful: https://stackoverflow.com/questions/52532893/understanding-linux-virtual-memory-valgrinds-massif-output-shows-major-differe

The following reftests are disabled because they don't pass. Fix them and remove them from tests/reftest/disabled.list

ycck.jpg
mozilla/jpg-size-3x3.jpg
mozilla/jpg-size-4x4.jpg

This image is corrupt and will not decode in most image decoders (convert; opencv imread), but does not produce a decoding error in jpeg-decoder. Whether this is a problem is debatable, of course - for an image viewer it's good to be able to display the partial, corrupted image - but it's also nice for a library to be able to report that there's a glitch.

A simple program to validate the lack of error:

use anyhow::{anyhow, Result};

fn doit() -> Result<()> {
    let f = "839442a6aa3dcca42f4962e8ec87b999.jpg";
    let of = std::fs::File::open(f)?;
    let mut decoder = jpeg_decoder::Decoder::new(std::io::BufReader::new(of));
    decoder.read_info()?;
    let md = decoder.info().unwrap();
    println!("Metadata: {:#?} h/w: {} {}", md, md.height, md.width);
    let pixels = decoder.decode()?;
    println!("Pixel count: {}", pixels.len());
    println!("Happy image by jpeg crate");
    Ok(())
}
fn main() {
    if let Err(e) = doit() {
        println!("Error reading jpeg: {}", e);
    }
}

Which, when run, produces:

/target/debug/jpegcheck
Metadata: ImageInfo {
    width: 2560,
    height: 1920,
    pixel_format: RGB24,
} h/w: 1920 2560
Pixel count: 14745600
Happy image by jpeg crate

I would imagine that with IDCT accelerated, Jpeg->YCbCr unscaled planar performance should meet/exceed libjpeg-turbo (it hasn't merged these yet - I think these handle multiple blocks).

https://github.com/huaqiangwang/libjpeg-turbo/blob/avx2-dev/simd/jidctflt-avx2-64.asm
https://github.com/huaqiangwang/libjpeg-turbo/blob/avx2-dev/simd/jfdctflt-avx2-64.asm
https://github.com/huaqiangwang/libjpeg-turbo/blob/avx2-dev/simd/jfdctmflt-avx2-64.asm

I cannot compile jpeg-decoder on my system. The compile throws the following error during build.

   Compiling jpeg-decoder v0.1.2
/Users/arsalanahmad/.cargo/registry/src/github.com-88ac128001ac3a9a/jpeg-decoder-0.1.2/src/decoder.rs:465:38: 465:136 error: use of unstable library feature 'clone_from_slice' (see issue #27750)
/Users/arsalanahmad/.cargo/registry/src/github.com-88ac128001ac3a9a/jpeg-decoder-0.1.2/src/decoder.rs:465                                     .clone_from_slice(&self.coefficients[scan.component_indices[i]][block_offset .. block_offset + 64]);

I'm on OS X and running rustc 1.6.0.

cargo build on windows results in:

src\error.rs:63:69: 63:95 error: no associated item named 'UnexpectedEof' found for type 'std::io::error::ErrorKind' in the current scope

src\error.rs:63 ByteorderError::UnexpectedEOF => Error::Io(IoError::new(IoErrorKind::UnexpectedEof, err)),

This library fails to parse all JPEG images produced by my phone camera (I have a Galaxy S8) image with FormatError(APP(5) with invalid length 2).

I found the byte string 0xFF 0xE5 0x00 0x02 in the image file, and from some brief reading about EXIF data it seems like that should be legal - the length includes the length descriptor itself, so this is just an APP(5) EXIF tag that contains no data. It looks like the issue is that, in parser.rs, read_length requires the length to be > 2, when it should require it to be >= 2.

Other tools (I tried GIMP, ImageGlass, MS Paint, and the default Windows photo viewer) don't have any problems with the images.

I tried to attach an example image file to this issue, but Github re-encodes the image, so I had to gzip it first.

example_image.jpg.gz

At the end of crashtest() the result is not looked at. If I unwrap() the result a lot of the tests fail. I'm not sure whether or not they should fail, but I'd assume they should pass since the test would be expecting an Err otherwise.

let file = File::open(path).unwrap();
let mut decoder = jpeg::Decoder::new(BufReader::new(file));
let _ = decoder.decode();

While trying to open a valid JPG image, I got this error:
Error: FormatError("encountered DHT with invalid length 2")

I tracked the error back to this line:

https://github.com/kaksmet/jpeg-decoder/blob/c8e25252ec6c6888db8eb866dec47131c34949a3/src/parser.rs#L88

In version 0.1.0, this part of the program used to be like this:

if length <= 2 {
    return Err(Error::Format(format!("encountered {:?} with invalid length {}", marker, length)));
}

The <= was changed to <. If it was still the way it used to be, the program would have (probably) functioned correctly.

For reference, this was the image I tried to load

I have a bunch of pictures from an old smartphone, the Sony Ericsson P990i, which jpeg-decoder fails to decode most of (some are decoded without problem, but most fail with a Format("did not find marker where expected") error.

I don't know enough about decoding jpegs to know if these images are actually broken according to the standard, but many other programs (e.g. gimp and eog) reads them just fine. I'm afraid I also don't know enough about jpeg decoding to be any real help in finding the actual problem, but at least I've forked to https://github.com/kaj/jpeg-decoder and added a failing jpeg and a png copy of it (created with gimp) in a branch.

To reproduce:

git clone [email protected]:kaj/jpeg-decoder.git
cd jpeg-decoder
git checkout failing-p990
cargo test

(The image sonyericsson-p990i.jpg is by me and hereby released as CC0)

jpeg_decoder::decoder::Decoder::decode_internal seems to take 50% of the decoding time, or over 75% if using Rayon because this part is not parallelized. This part alone takes more time than libjpeg-turbo takes to decode the entire image.

It appears that jpeg-decoder reads one byte at a time from the input stream and executes some complex logic for every byte, e.g. in HuffmanDecoder::read_bits and a number of other functions called from decode_internal. I suspect performing a single large read (a few Kb in size), then using something that lowers to memchr calls to find marker boundaries would be much faster.

Profiled using this file: https://commons.wikimedia.org/wiki/File:Sun_over_Lake_Hawea,_New_Zealand.jpg via image crate, jpeg-decoder v0.1.19

Single-treaded profile: https://share.firefox.dev/30ZTmks
Parallel profile: https://share.firefox.dev/3dqzE49

The version on crates.io does not include (merged) PR #94, which fixed a decoding issue that some people seem to be still running into.

cc: image/#1020

Hello,
Would you be interested in moving the CI for this project from travis to github actions ? Currently, the CI is super slow, and this makes the management of pull requests less smooth than it could be.
If you are interested, I'm ready to make a PR that creates the necessary github action workflow files.

use std::io::{self, Read, Cursor};
use std::{intrinsics};
use std::panic;

extern crate jpeg_decoder;

use jpeg_decoder::Decoder;

fn main() {
    let mut decoder = Decoder::new(b"\xff\xd8\xff\xee\x30\x30\xff\xd9" as &[u8]);
    let a = decoder.read_info();
}

ubuntu@ip-172-31-32-38:~/image/image-fuzz$ cargo run --release
     Running `target/release/image-fuzz`
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:325
stack backtrace:
   1:     0x7fafb74dd5b4 - std::sys::backtrace::tracing::imp::write::h9fb600083204ae7f
   2:     0x7fafb74e09bb - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hca543c34f11229ac
   3:     0x7fafb74e0643 - std::panicking::default_hook::hc2c969e7453d080c
   4:     0x7fafb74d899d - std::panicking::rust_panic_with_hook::hfe203e3083c2b544
   5:     0x7fafb74e0c51 - std::panicking::begin_panic::h4889569716505182
   6:     0x7fafb74d952a - std::panicking::begin_panic_fmt::h484cd47786497f03
   7:     0x7fafb74e0be1 - rust_begin_unwind
   8:     0x7fafb751615f - core::panicking::panic_fmt::h257ceb0aa351d801
   9:     0x7fafb7516438 - core::panicking::panic::h4bb1497076d04ab9
  10:     0x7fafb7488286 - _<jpeg_decoder..decoder..Decoder<R>>::decode_internal::h5b87959ffb22e82d
  11:     0x7fafb74762c4 - image_fuzz::main::hdbde171b5ac48c88
  12:     0x7fafb74e0258 - std::panicking::try::call::hc5e1f5b484ec7f0e
  13:     0x7fafb74ea36b - __rust_try
  14:     0x7fafb74ea30e - __rust_maybe_catch_panic
  15:     0x7fafb74dfc8c - std::rt::lang_start::h61f4934e780b4dfc
  16:     0x7fafb686ef44 - __libc_start_main
  17:     0x7fafb7476068 - <unknown>
  18:                0x0 - <unknown>
error: Process didn't exit successfully: `target/release/image-fuzz` (exit code: 101)

Found using afl.rs.

Why does reftest allow a pixel component to be off by 2 from the reference? Changing it to 0 makes the test fail, indicating the output does not exactly match the reference.

I noticed this because I was trying to check the output of my own program against a reference, but the difference introduced by jpeg-decoder caused it to fail.

It looks like this quite fundamental crate is maintained by a single person (@kaksmet). Maybe it should be moved to the image-rs organization ? It would give more people push access, and should lower the risk of this crate being abandoned.

Trying to figure out how to open this

Some images like this one:
https://ant-photo.eu/wp-content/uploads/2020/07/Cataglyphis-iberica-worker2.jpg
end up looking like this:
https://i.ibb.co/tp2d6Wy/out.png
after simply decoding the image and saving the output in a human readable format.

Example program:

[dependencies]
bmp = "0.5.0"
jpeg-decoder = "0.1"

extern crate jpeg_decoder as jpeg;
#[macro_use]
extern crate bmp;

use std::fs::File;
use std::io;
use bmp::*;

fn main() {
    let file = File::open("broken.jpg").unwrap();
    let mut dec = jpeg::Decoder::new(io::BufReader::new(file));
    let pixels = dec.decode().unwrap();
    let mut img = Image::new(1920, 1080);
    for s in 0..1920 * 1080 {
        img.set_pixel(s % 1920, s / 1920, px!(pixels[s as usize * 3], pixels[s as usize * 3 + 1], pixels[s as usize * 3 + 2]));
    }
    img.save("out.bmp");
}

This happens to some of the other images at https://ant-photo.eu/galeria/zdjecia-mrowek

When trying to decode the image, it fails with

thread 'main' panicked at 'invalid dimensions', src/parser.rs:245:5
[...]
  11: std::panicking::begin_panic
             at /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libstd/panicking.rs:397
  12: jpeg_decoder::parser::ceil_div
             at src/parser.rs:245
  13: jpeg_decoder::parser::update_component_sizes
             at src/parser.rs:255
  14: jpeg_decoder::parser::parse_sof
             at ./src/parser.rs:228

I was reading the code, and I realised that the planes of progressive JPEGs are iterated upon twice to check whether any of them is empty. I thought I would open an issue:

https://github.com/image-rs/jpeg-decoder/blob/master/src/decoder.rs#L344-L346

and

https://github.com/image-rs/jpeg-decoder/blob/master/src/decoder.rs#L773-L775

We may want to remove the second redundant check. compute_image is called only from decode_internal.

This library is continuously tested with rust 1.24 : https://github.com/kaksmet/jpeg-decoder/blob/master/.travis.yml

However, it currently does not compile with rust 1.24, because of a transitive dependency of rayon: https://travis-ci.org/kaksmet/jpeg-decoder/jobs/569851819

This causes all new pull requests to fail.

The problem is that Cargo.toml requires rayon= "1.0", which means ">=1.0<2.0.0" according to semver. And the current latest version of rayon to meet this requirement is 1.1.0, which requires rustc 1.28.0 or greater.

So there are two solutions:

1. declare this library as requiring rustc >= 1.28.0 and updating .travis.yml (See PR #104)
2. declare this library as requiring rustc >= 1.24.0 and updating Cargo.toml to specify ~1.0 instead of 1.0 as the version of rayon. (See PR #103)

I would be in favor of the first option, because a lot of the ecosystem already requires later rust versions.

Decoding the following input file crashes in debug mode (but not in release mode, because overflows silently wrap in release mode).

id:000000,sig:06,src:000000,op:flip2,pos:1027.tar.gz

$ RUST_BACKTRACE=1 cargo run --bin reproduce_decode out/crashes/id\:000000\,sig\:06\,src\:000000\,op\:flip2\,pos\:1027
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/reproduce_decode 'out/crashes/id:000000,sig:06,src:000000,op:flip2,pos:1027'`
thread 'main' panicked at 'attempt to subtract with overflow', /home/danilo/Projects/jpeg-decoder/src/decoder.rs:760:17
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1028
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:188
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:205
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:464
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:373
  12: rust_begin_unwind
             at src/libstd/panicking.rs:302
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:139
  14: core::panicking::panic
             at src/libcore/panicking.rs:70
  15: jpeg_decoder::decoder::refine_non_zeroes
             at /home/danilo/Projects/jpeg-decoder/src/decoder.rs:760
  16: jpeg_decoder::decoder::decode_block_successive_approximation
             at /home/danilo/Projects/jpeg-decoder/src/decoder.rs:721
  17: jpeg_decoder::decoder::Decoder<R>::decode_scan
             at /home/danilo/Projects/jpeg-decoder/src/decoder.rs:484
  18: jpeg_decoder::decoder::Decoder<R>::decode_internal
             at /home/danilo/Projects/jpeg-decoder/src/decoder.rs:235
  19: jpeg_decoder::decoder::Decoder<R>::decode
             at /home/danilo/Projects/jpeg-decoder/src/decoder.rs:138
  20: reproduce_decode::decode
             at src/reproduce_decode.rs:6
  21: reproduce_decode::main
             at src/reproduce_decode.rs:17
  22: std::rt::lang_start::{{closure}}
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/rt.rs:61
  23: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:48
  24: std::panicking::try::do_call
             at src/libstd/panicking.rs:287
  25: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  26: std::panicking::try
             at src/libstd/panicking.rs:265
  27: std::panic::catch_unwind
             at src/libstd/panic.rs:396
  28: std::rt::lang_start_internal
             at src/libstd/rt.rs:47
  29: std::rt::lang_start
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/rt.rs:61
  30: main
  31: __libc_start_main
  32: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Found using afl (see #131).

Potentially related to #63.

I'm experiencing a memory leak when loading JPEG images.

I discovered growing memory usage when rapidly loading JPEG images through the image crate (image-rs/image#1055).

valgrind confirms memory isn't fully cleared.

Reproduction steps

The following simple program confirms this:

use std::fs::File;
use jpeg_decoder::Decoder;

pub fn main() {
    let file = File::open("/home/timvisee/Pictures/Avatar/256x256/Avatar.jpg")
        .expect("failed to open file");

    Decoder::new(file).decode();
}

This produces the following valgrind log.

# valgrind ./bin

==24980== Memcheck, a memory error detector
==24980== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24980== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==24980== Command: ./target/debug/examples/mem_test
==24980==
==24980==
==24980== HEAP SUMMARY:
==24980==     in use at exit: 1,502 bytes in 19 blocks
==24980==   total heap usage: 391 allocs, 372 frees, 1,034,351 bytes allocated
==24980==
==24980== LEAK SUMMARY:
==24980==    definitely lost: 0 bytes in 0 blocks
==24980==    indirectly lost: 0 bytes in 0 blocks
==24980==      possibly lost: 288 bytes in 1 blocks
==24980==    still reachable: 1,214 bytes in 18 blocks
==24980==         suppressed: 0 bytes in 0 blocks
==24980== Rerun with --leak-check=full to see details of leaked memory
==24980==
==24980== For counts of detected and suppressed errors, rerun with: -v
==24980== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Not everything is cleared. It seems to be originating form the decode() function.

I'm not too familiar with valgrind, but I'm quite sure this isn't right. I'll attempt to investigate this now.

Trying to compile the latest master locally fails

$ cargo --version
cargo 1.42.0 (86334295e 2020-01-31)

$ cargo test
   Compiling jpeg-decoder v0.1.18 (/Users/olojkine/Developpement/jpeg-decoder)
error[E0432]: unresolved import `png::HasParameters`
 --> examples/decode.rs:4:5
  |
4 | use png::HasParameters;
  |     ^^^^^^^^^^^^^^^^^^ no `HasParameters` in the root

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: could not compile `jpeg-decoder`.

The attached image fails to load with the following error.

Decoding(DecodingError { format: Exact(Jpeg), underlying: Some(Format("use of unset quantization table")) })

desktop.zip

This was orignially posted to https://github.com/ArturKovacs/emulsion/issues/35

Implementing a generic integer upsampler will allow us to support most chroma subsampling ratios.

This issue was originally posted to https://github.com/ArturKovacs/emulsion/issues/6

When opening the following image with image::open() the error seen below is returned.

azerty_mac.zip

assertion failed: `(left == right)`
  left: `224000`,
 right: `230400`: destination and source slices have different lengths

The standard says that when a thread is non-interleaved the MCU is just on block. But decode_scan still treats the MCU as H*V blocks when looking for RST markers. Is this a bug?