dropbox / lepton

Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.

⚠️  Deprecation Notice for the Dropbox Lepton project

Dear Lepton users and developers,

Thank you so much for using and contributing to Lepton over the years! We’ve recently realized that we no longer have sufficient resources to adequately support this project. This includes providing timely fixes to bugs and vulnerabilities such as those that were recently reported to us. While we did ensure that the reported vulnerabilities don’t affect our internal use of Lepton, we unfortunately don’t have the capacity to properly fix these and future issues in this public repo. Therefore, after some consideration, we’ve decided to deprecate and archive the project. We apologize for any inconvenience and appreciate your understanding.

If you would like to continue using Lepton, please check out Microsoft’s implementation of Lepton in Rust or alternatively consider switching to other lossless compression methods such as Brotli or to modern image formats such as WebP or JPEG XL.

Lepton

Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.

This can be used to archive large photo collections, or to serve images live and save 22% bandwidth.

Build directions

Using a single core

./autogen.sh
./configure
make
make check

For multiprocessor machines:

./autogen.sh
./configure
make -j8
make check -j8

Using CMAKE:

mkdir -p build
cd build
cmake ..
make -j8

On Windows

mkdir -p build
cd build
"c:\Program Files\CMake\bin\cmake" ..
start .
REM Double click the Visual Studio project

Usage

To roundtrip (compress and decompress) an image, original.jpg, do the following:

./lepton original.jpg compressed.lep
./lepton compressed.lep restored_original.jpg

Or all at once:

./lepton original.jpg compressed.lep && ./lepton compressed.lep restored_original.jpg && diff restored_original.jpg original.jpg && echo no differences

Lepton may also be used with pipes -- be sure to check the exit code when using pipes as if compression fails lepton will produce 0 bytes and return a nonzero exit code (failure). In this case do not assume the 0 byte file is representative of the original.

./lepton - < original.jpg > compressed.lep
./lepton - < compressed.lep > restored_original.jpg

You may specify higher memory limits than the default for lepton to handle bigger images:

./lepton -memory=1024M -threadmemory=128M input_file output_file

Additionally you can configure lepton to process progressive jpegs. Warning: these take more memory to decode than normal JPEGs since the entire framebuffer must be kept in memory for the duration of the decompression, instead of just 2 rows of blocks.

./lepton -allowprogressive -memory=1024M -threadmemory=128M progressive.jpg compressedprogressive.lep

Submitting pull requests to lepton

Please begin by filling out the contributor form and asserting that

The code I'm contributing is mine, and I have the right to license it.
I'm granting you a license to distribute said code under the terms of this agreement.

at this page: https://opensource.dropbox.com/cla/

Then create a new pull request through the github interface

Debugging

Lepton is designed to be easy to debug, but a command line requirement is necessary to prevent the standard forks that let it do a secure verification in a separate process.

To avoid setting follow fork flags, please pass -skipverify to the command line. This will stop verification and let you debug the app as a single process application. If the bug happens in single threaded mode, also you can pass -singlethread which makes it easier to step through the code without other threads hitting breakpoints.

Bindings for other languages

• NodeJS: https://github.com/whitef0x0/node-lepton
• PHP: https://github.com/gtuk/php-lepton

Acknowledgements

Many thanks to Matthias Stirner and HTW Aalen University for the development of the uncmpJPG library.

Thanks to the VPX authors for their highly tuned bool reader and bool writer implementations.

Related Work

Be sure to read the groundbreaking research done by Matthias Stirner, Gehard Seelman and HTW Aalen University in their 2007 paper and check out their excellent PackJPG repositories for compression of JPEG MP3 BMP and PMN formats.

Also the paq algorithms combine for a highly compressed JPEG result when decompression time is less critical.

When the compressed JPEG needs to also be a JPEG, there is mozjpg to explore as well.

License

Unless otherwise noted:

Copyright (c) 2016 Dropbox, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

lepton's Issues

compression reduces as jpeg quality increases

I'm very interested in the lossless side of lepton. It is a legitimate requirement in dropbox's particular case, but in general, it doesn't make point insisting on lossless while jpeg is already lossy.
I'm not sure if lepton can potentially compress a lossless format, so I set jpeg quality to 100, just to approximate a lossless input. My input image is a 16-bit depth lossless TIFF image of mammogram. I convert it to jpeg with " -auto-level -depth 8", and with -quality value 100, 80 and 20. The file sizes are listed below:

• TIFF: 18M (6.6M when converted to TIFF with -auto-level -depth 8)
• jpeg/100: 5.3M ==> lep: 4.8M, 91.17%
• jpeg/80: 636K ==> lep: 519K, 81.63%
• jpeg/20: 82K ==> lep: 43K, 52.55%

This result is very counter-intuitive to me, as I believe less being done by jpeg means more space of optimization for lepton, yet lepton performs worse. My suspicion is that lepton is only compressing the compression artifacts of jpeg, and if we somehow inject it with a true lossless input, there will probably be little compression. Unfortunately I'm not able to provide the image due to the license, but I strongly suspect that this is a general behavior of the lepton algorithm.

Unit test fails in 32 bits platform

See Debian build log here, the unit test fails at Debian i386 architecture.

please add wildcard support for Windows

please add wildcard support for windows so that whole directories or many files at once can be processed. Using Drag & Drop a folder on the executables did not work.

Lepton on mobile phones

I'm hoping to experiment with use Lepton on mobile clients Android (+iOS). I know there are previous and ongoing threads about this.

question about used in mobile devices (android)
Compilation fails on Linux/armhf

From these discussions It looks like lepton is currently not compatible on ARM chipsets. So the question is what it would involve in-order to make lepton something that can run on mobile devices? Ideally I would want to incorporate this as a c++ library into a Android project through Android NDK. As it is, lepton looks to be in its early phases and offers CLI only, I would likely need to propose a solution to Lepton as a shared library to make it into a library.

This question comes from someone who is unfamiliar with the complexities involved, but is willing to spend the time to learn and contribute towards these improvements.

License?

There is a license file in the repository, but I had to spend a moment searching for it. Ideally, this should be aligned with open source convention by naming the license file LICENSE and including a link to it in the readme.

Mix use of UPPERCASE and lowercase cmake commands

Currently CMakeLists.txt is mixed use of UPPER and lower case for cmake commands.

As lowercase is recommended (see http://stackoverflow.com/questions/22839458/tools-for-upper-lower-case-consistency-in-cmake-source)

Shall we all use lowercase?

Build failure on Ubuntu 16.04

make says :

In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/smmintrin.h:32:0,
from src/lepton/idct.cc:3:
/usr/lib/gcc/x86_64-linux-gnu/5/include/tmmintrin.h: In function ‘__m128i epi32l_to_epi16(__m128i)’:
/usr/lib/gcc/x86_64-linux-gnu/5/include/tmmintrin.h:136:1: error: inlining failed in call to always_inline ‘__m128i _mm_shuffle_epi8(__m128i, __m128i)’: target specific option mismatch
_mm_shuffle_epi8 (__m128i __X, __m128i __Y)
^
src/lepton/idct.cc:168:89: error: called from here
0xd, 0xc, 0x9, 0x8, 0x5, 0x4, 0x1, 0x0));

Confusing name.

The Lepton is a thermal imaging camera module released in 2014.

Having a image compression algorithm that has the same name as a (relatively) well established camera sensor module is rather confusing, and I confess I spent the first paragraph of the press release confused as to why dropbox cared about compressing thermal images.

It'd probably be wise to name the project something that doesn't already have decent mind-share in the imaging field. The first commit in this repo was 9-10 months after the Lepton was released.

Progressive images - Usage to be documented

For progressive images, if -allowprogressive is not explicitly added in commandline, will throw error

"
lepton v1.0-
Failed To start subprocess with command line OS_ERROR
"

Works fine with -allowprogressive option.

• Appropriate error has to be thrown (Minor fix)
• Check can be added in code and the right code path taken without returning error.

Platform : Win10, Visual Studio 14 2015

Output format stability

Is the lepton output format stable? For example packJPG says "Compressed PJG files are not compatible between different packJPG versions." What is the status in lepton? Can this be documented in README please?

Thanks.

Lepton on Linux < 3.1 & OOM with big image

Hello, I am trying to build this application on

Linux centos6 2.6.32-358.el6.x86_64

But I have next one error

src/lepton/jpgcoder.cc:52:27: fatal error: linux/seccomp.h: No such file or directory

Okey, I've installed a 'kernel-devel' package on the machine and compile with next one flags

-I /usr/src/kernels/2.6.32-642.3.1.el6.x86_64/include

And I've got next one errors

src/io/seccomp-bpf.hh:69:29: error: ‘SECCOMP_RET_KILL’ was not declared in this scope
src/io/seccomp-bpf.hh:76:29: error: ‘SECCOMP_RET_ALLOW’ was not declared in this scope
src/io/Seccomp.cc:83:31: error: ‘SECCOMP_MODE_STRICT’ was not declared in this scope
src/io/Seccomp.cc:97:35: error: ‘SECCOMP_MODE_FILTER’ was not declared in this scope

I've found that kernel's headers have no this define

fgrep "SECCOMP_RET_KILL" /usr/src/kernels/2.6.32-642.3.1.el6.x86_64 -R

As I've found, maybe it's wrong, this define was appeared in v3.1+ kernel

It would be nice if I can fully disable the SECCOMP on a build phase

Okey, next one question:

In your article

https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/

You said that all operations with image are under 24 megabytes.
So, what is it?

./lepton -singlethread -skipvalidation /tmp/world.topo.bathy.200407.3x21600x10800.jpg -memory=4096M
image size: 27201049 bytes

If I run without '-memory=4096M' I have next one assert

Assertion failed: (false && "Out of memory error"), function custom_calloc

In 'Release' Lepton version I have same problem with memory usage

Image is from NASA site

http://eoimages.gsfc.nasa.gov/images/imagerecords/73000/73751/world.topo.bathy.200407.3x21600x10800.jpg

Consider moving images into a submodule

The images directory is only used by the test suite, and it single-handedly makes the source tarball unnecessarily large (~50M). What about moving it off into a submodule? This way users won't need to download all the images only to discard them after installation (assuming the majority of users won't bother to run the test suite).

A process left after running unit test

It is from Debian bug report #831897. After running unit test, the following process is in system.

% ps auxww | grep lepton
czchen    9926  0.0  0.0 1328568 3620 pts/4    S    10:55   0:00 ./lepton -socket=/tmp/876b9068-b337-469d-b7ce-e31456b5b634 -timebound=10ms -preload

C++11 requires a space between literal and identifier

Getting below error while running command: make or make -j8 on Amazon Linux x86_64-redhat-linux
gcc version 4.8.2 20140120 (Red Hat 4.8.2-15) (GCC)
Please help.

src/lepton/jpgcoder.cc:52:27: fatal error: linux/seccomp.h: No such file or directory
#include <linux/seccomp.h>
^
compilation terminated.
src/lepton/vp8_encoder.cc:52:52: warning: invalid suffix on literal; C++11 requires a space between literal and identifier [-Wliteral-suffix]
case SIGN8:nam = "SI#include "emmintrin.h"GN8";break;
^
make[1]: *** [src/lepton/jpgcoder.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/mnt/pankajg/lepton'
make: *** [all] Error 2

How can I build a static lepton binary

Is there a way to build static lepton binary to avoid library dependencies issues across machines?

Executing in Windows 7

lepton v1.0-08c52d9280df3d409d9246df7ff166dd94628730
Failed To start subprocess with command line OS_ERROR

"version" make target needs to be a dependency

Steps to reproduce:

• git clone
• mkdir build
• cd build
• cmake -DCMAKE_BUILD_TYPE=release -DUSE_SYSTEM_DEPENDENCIES=ON -DCMAKE_INSTALL_PREFIX="/usr" ..
• make

Actual result:

[ 11%] Building CXX object CMakeFiles/lepton-avx.dir/src/lepton/jpgcoder.cc.o
/home/me/lepton/src/lepton/src/lepton/jpgcoder.cc:80:22: fatal error: version.hh: No such file or directory
 #include "version.hh"
                      ^
compilation terminated.
make[2]: *** [CMakeFiles/lepton-avx.dir/build.make:159: CMakeFiles/lepton-avx.dir/src/lepton/jpgcoder.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:105: CMakeFiles/lepton-avx.dir/all] Error 2
make: *** [Makefile:128: all] Error 2

Expected result: Completed build

Simply running make version before running make results in a proper build.

Compilation fails on Linux/armhf

I tried out compilation an a dual-core A20 board (Cortex-A7, armhf, Ubuntu 16.04, kernel 4.6.4) but make -j2 stops with

In file included from src/vp8/decoder/boolreader.hh:33:0,
                 from src/vp8/decoder/boolreader.cc:15:
src/vp8/decoder/../model/numeric.hh:11:23: fatal error: smmintrin.h: No such file or directory
compilation terminated.
Makefile:1814: recipe for target 'src/vp8/decoder/boolreader.o' failed
make[1]: *** [src/vp8/decoder/boolreader.o] Error 1

I did not further investigate but wanted to ask first whether the tool is supposed to be used on different platforms than x86/x86-64?

Printing size_t correctly

Throughout the code variables of type size_t are printed as long (%ld). This is incorrect in two ways:

• long is signed, whereas size_t is not
• long may not be the same width as size_t -- in particular, on i386 (32-bit) systems long is usually a 64-bit integer, while size_t is 32 bit

The correct format specification for size_t is %zu. For ssize_t it is %zd. Pull requests are messy, so I'll just leave an old-fashioned patch-file here. Please, merge it in. Thank you!

patch-warnings

question about used in mobile devices (android)

Is it possible to run this tool on native mobile apps?
I want to use it to compress user photos before sending them to the server ..

Build failures on OS X

This is a continuation of #11. I'm opening a new issue because

1. The old failures due to SSE 4.1 appear to have been fixed in 1.1, while a new one surfaced;
2. The old thread was slightly polluted by pointless arguments.

Again, the failures occur only on Homebrew's CI server, not locally. Builds on OS X 10.9 and 10.10 now pass, but there is still a problem on 10.11, log here:

/usr/local/Library/Homebrew/shims/super/clang++    -I/tmp/lepton-20160718-65941-texea9/lepton-1.2 -I/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/util -I/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model -I/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/encoder -I/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/decoder  -std=c++11 -fno-exceptions -fno-rtti -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.11   -msse4.2   -DDEFAULT_ALLOW_PROGRESSIVE -DHIGH_MEMORY -o CMakeFiles/lepton.dir/src/lepton/jpgcoder.cc.o -c /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/jpgcoder.cc
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/jpgcoder.cc:64:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/vp8_decoder.hh:4:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/lepton_codec.hh:4:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/model.hh:10:
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/numeric.hh:295:32: error: call to '_mm_mullo_epi32' is ambiguous
    __m128i t = _mm_srli_epi32(_mm_mullo_epi32(m, abs_num), log_max_numerator);
                               ^~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/include/smmintrin.h:130:1: note: candidate function
_mm_mullo_epi32 (__m128i __V1, __m128i __V2)
^
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/../util/mm_mullo_epi32.hh:38:1: note: candidate function
_mm_mullo_epi32(const __m128i &a, const __m128i &b)
^
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/jpgcoder.cc:64:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/vp8_decoder.hh:4:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/lepton_codec.hh:4:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/model.hh:10:
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/numeric.hh:304:32: error: call to '_mm_mullo_epi32' is ambiguous
    __m128i t = _mm_srli_epi32(_mm_mullo_epi32(m, num), log_max_numerator);
                               ^~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/include/smmintrin.h:130:1: note: candidate function
_mm_mullo_epi32 (__m128i __V1, __m128i __V2)
^
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/../util/mm_mullo_epi32.hh:38:1: note: candidate function
_mm_mullo_epi32(const __m128i &a, const __m128i &b)
^
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/jpgcoder.cc:64:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/vp8_decoder.hh:4:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/lepton_codec.hh:4:
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/model.hh:903:27: error: call to '_mm_mullo_epi32' is ambiguous
        __m128i deq_low = _mm_mullo_epi32(coeffs_x_low, icos_low);
                          ^~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/include/smmintrin.h:130:1: note: candidate function
_mm_mullo_epi32 (__m128i __V1, __m128i __V2)
^
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/../util/mm_mullo_epi32.hh:38:1: note: candidate function
_mm_mullo_epi32(const __m128i &a, const __m128i &b)
^
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/jpgcoder.cc:64:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/vp8_decoder.hh:4:
In file included from /tmp/lepton-20160718-65941-texea9/lepton-1.2/src/lepton/lepton_codec.hh:4:
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/model.hh:904:28: error: call to '_mm_mullo_epi32' is ambiguous
        __m128i deq_high = _mm_mullo_epi32(coeffs_x_high, icos_high);
                           ^~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.3.0/include/smmintrin.h:130:1: note: candidate function
_mm_mullo_epi32 (__m128i __V1, __m128i __V2)
^
/tmp/lepton-20160718-65941-texea9/lepton-1.2/src/vp8/model/../util/mm_mullo_epi32.hh:38:1: note: candidate function
_mm_mullo_epi32(const __m128i &a, const __m128i &b)
^

Build environment:

CPU: quad-core 64-bit ivybridge
OS X: 10.11.5-x86_64
Xcode: 7.3.1
CLT: 7.3.1.0.1.1461711523
Clang: 7.3 build 703

EDIT: The log above is for v1.2 (08c52d9).

Existing zlib should be used whenever possible

All modern Unix-like systems (BSDs, MacOS, Linux, Solaris) already come with zlib. Same is usually true about OpenSSL.

When building lepton on such systems, the dependencies/ folder should be completely ignored. In fact, it should be possible to not have it at all. The fix will be three-fold:

• teach configure.ac and Makefile.am to not freak out, when dependencies/ does not exist
• teach the same to set the -I, the -L and the -l flags properly
• change the sources to stop #includeing by full path (../../dependencies/...) and refer simply to header-file names: #include <md5.h>. The compiler will find the headers based on the -I flags set by configure

(error) Using 'memset' on struct that contains a 'std::vector'

Greetings. I found three issues, where memset is used on a struct that contains a std::vector. This can be dangerous and can lead to inconsistent memory (see discussion at http://stackoverflow.com/questions/6877281/memset-structure-with-stdstring-contained).

[src/vp8/util/block_context.hh:92]: (error) Using 'memset' on struct that contains a 'std::vector'.
[src/vp8/util/block_context.hh:100]: (error) Using 'memset' on struct that contains a 'std::vector'.
[src/vp8/util/block_context.hh:123]: (error) Using 'memset' on struct that contains a 'std::vector'.

Found by https://github.com/bryongloden/cppcheck

Some memory corruptions in lepton

Hi, I at this url you can download some samples that will cause memory corruption problems in lepton:

https://github.com/marcograss/marcograss.github.io/blob/master/assets/lepton_testcases1.zip?raw=true

you can reproduce with ./lepton/lepton -singlethread -unjailed -preload testcase.jpeg /tmp/out.lep

1. unknown.jpeg

lepton v1.0-91619e2
START ACHIEVED 1468716163 822728
decode error in scan0 / mcu2TS_MAIN (0) 0.000000
TS_MODEL_INIT_BEGIN (0) 0.004525
TS_MODEL_INIT (0) 0.004543
TS_READ_STARTED (0) 0.004637
TS_READ_FINISHED (0) 0.004768
TS_JPEG_DECODE_STARTED (0) 0.004768
TS_JPEG_DECODE_FINISHED (0) 0.004975
TS_DONE (0) 0.004976
6573388 bytes needed to decompress this file

::::BILL::::

==76344==ERROR: AddressSanitizer: unknown-crash on address 0x0000008cb038 at pc 0x00000052eb79 bp 0x7ffd0cfd5720 sp 0x7ffd0cfd5710
READ of size 208 at 0x0000008cb038 thread T0
#0 0x52eb78 in std::__atomic_base::load(std::memory_order) const /usr/include/c++/6/bits/atomic_base.h:396
#1 0x52eb78 in std::atomic_base::operator unsigned int() const /usr/include/c++/6/bits/atomic_base.h:259
#2 0x52eb78 in print_bill(int) src/vp8/util/billing.cc:145
#3 0x46b7f3 in process_file(IOUtil::FileReader, IOUtil::FileWriter, int, bool) src/lepton/jpgcoder.cc:1616
#4 0x406c73 in main src/lepton/jpgcoder.cc:773
#5 0x7effabc9182f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#6 0x40afc8 in _start (/home/marco/VulnResearch/misc/lepton/lepton+0x40afc8)

0x0000008cb090 is located 0 bytes to the right of global variable 'billing_map' defined in 'src/vp8/util/billing.cc:10:23' (0x8cafc0) of size 208
SUMMARY: AddressSanitizer: unknown-crash /usr/include/c++/6/bits/atomic_base.h:396 in std::__atomic_base::load(std::memory_order) const
Shadow bytes around the buggy address:
0x0000801115b0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115c0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115d0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115e0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x000080111600: 00 00 00 00 00 00 00[00]00 00 00 00 00 00 00 00
0x000080111610: 00 00 f9 f9 f9 f9 f9 f9 00 00 00 00 04 f9 f9 f9
0x000080111620: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111640: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111650: 00 00 00 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==76344==ABORTING
SHORT_READ

1. invalid_access.jpeg

lepton v1.0-91619e2
START ACHIEVED 1468716389 229829

ASAN:DEADLYSIGNAL

==20540==ERROR: AddressSanitizer: SEGV on unknown address 0x0000008cec00 (pc 0x000000455164 bp 0x000000119d80 sp 0x7ffc54cfc1c0 T0)
#0 0x455163 in setup_imginfo_jpg(bool) src/lepton/jpgcoder.cc:4023
#1 0x49649f in bool read_jpeg(std::vector<std::pair<unsigned int, unsigned int>, std::allocator<std::pair<unsigned int, unsigned int> > >, ibytestream) src/lepton/jpgcoder.cc:2096
#2 0x446089 in std::function<bool ()>::operator()() const /usr/include/c++/6/functional:2136
#3 0x446089 in execute(std::function<bool ()> const&) src/lepton/jpgcoder.cc:1728
#4 0x46d8af in process_file(IOUtil::FileReader_, IOUtil::FileWriter_, int, bool) src/lepton/jpgcoder.cc:1510
#5 0x406c73 in main src/lepton/jpgcoder.cc:773
#6 0x7f816da5d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#7 0x40afc8 in _start (/home/marco/VulnResearch/misc/lepton/lepton+0x40afc8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV src/lepton/jpgcoder.cc:4023 in setup_imginfo_jpg(bool)
==20540==ABORTING
SHORT_READ

1. global_bof.jpeg

lepton v1.0-91619e2

START ACHIEVED 1468716464 964521

==45164==ERROR: AddressSanitizer: global-buffer-overflow on address 0x0000008c9f00 at pc 0x0000004571f1 bp 0x7ffeecdd5e30 sp 0x7ffeecdd5e20
READ of size 2 at 0x0000008c9f00 thread T0
#0 0x4571f0 in setup_imginfo_jpg(bool) src/lepton/jpgcoder.cc:4023
#1 0x49649f in bool read_jpeg(std::vector<std::pair<unsigned int, unsigned int>, std::allocator<std::pair<unsigned int, unsigned int> > >, ibytestream) src/lepton/jpgcoder.cc:2096
#2 0x446089 in std::function<bool ()>::operator()() const /usr/include/c++/6/functional:2136
#3 0x446089 in execute(std::function<bool ()> const&) src/lepton/jpgcoder.cc:1728
#4 0x46d8af in process_file(IOUtil::FileReader_, IOUtil::FileWriter_, int, bool) src/lepton/jpgcoder.cc:1510
#5 0x406c73 in main src/lepton/jpgcoder.cc:773
#6 0x7f16d425282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#7 0x40afc8 in _start (/home/marco/VulnResearch/misc/lepton/lepton+0x40afc8)

0x0000008c9f00 is located 32 bytes to the left of global variable 'read_done' defined in 'src/lepton/jpgcoder.cc:302:9' (0x8c9f20) of size 8
0x0000008c9f00 is located 24 bytes to the right of global variable 'overall_start' defined in 'src/lepton/jpgcoder.cc:303:9' (0x8c9ee0) of size 8
SUMMARY: AddressSanitizer: global-buffer-overflow src/lepton/jpgcoder.cc:4023 in setup_imginfo_jpg(bool)
Shadow bytes around the buggy address:
0x000080111390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0000801113a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0000801113b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0000801113c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0000801113d0: f9 f9 f9 f9 00 f9 f9 f9 f9 f9 f9 f9 00 f9 f9 f9
=>0x0000801113e0:[f9]f9 f9 f9 00 f9 f9 f9 f9 f9 f9 f9 00 f9 f9 f9
0x0000801113f0: f9 f9 f9 f9 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
0x000080111400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==45164==ABORTING
SHORT_READ

1. global_bof2.jpeg

lepton v1.0-91619e2

START ACHIEVED 1468716550 602819

==73428==ERROR: AddressSanitizer: global-buffer-overflow on address 0x0000008c7c28 at pc 0x00000045392d bp 0x7fff93915920 sp 0x7fff93915910
WRITE of size 2 at 0x0000008c7c28 thread T0
#0 0x45392c in build_huffcodes(unsigned char_, unsigned char_, huffCodes_, huffTree_) src/lepton/jpgcoder.cc:5099
#1 0x45392c in parse_jfif_jpg(unsigned char, unsigned int, unsigned char_) src/lepton/jpgcoder.cc:4109
#2 0x46ffd0 in decode_jpeg(std::vector<std::pair<unsigned int, unsigned int>, std::allocator<std::pair<unsigned int, unsigned int> > > const&, std::vector<ThreadHandoff, std::allocator >) src/lepton/jpgcoder.cc:2465
#3 0x446089 in std::function<bool ()>::operator()() const /usr/include/c++/6/functional:2136
#4 0x446089 in execute(std::function<bool ()> const&) src/lepton/jpgcoder.cc:1728
#5 0x46ccc8 in process_file(IOUtil::FileReader, IOUtil::FileWriter_, int, bool) src/lepton/jpgcoder.cc:1525
#6 0x406c73 in main src/lepton/jpgcoder.cc:773
#7 0x7f8525d6782f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#8 0x40afc8 in _start (/home/marco/VulnResearch/misc/lepton/lepton+0x40afc8)

0x0000008c7c28 is located 24 bytes to the left of global variable 'hcodes' defined in 'src/lepton/jpgcoder.cc:311:16' (0x8c7c40) of size 8208
0x0000008c7c28 is located 8 bytes to the right of global variable 'htrees' defined in 'src/lepton/jpgcoder.cc:312:16' (0x8c5c20) of size 8192
SUMMARY: AddressSanitizer: global-buffer-overflow src/lepton/jpgcoder.cc:5099 in build_huffcodes(unsigned char_, unsigned char_, huffCodes_, huffTree_)
Shadow bytes around the buggy address:
0x000080110f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x000080110f80: 00 00 00 00 f9[f9]f9 f9 00 00 00 00 00 00 00 00
0x000080110f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x000080110fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==73428==ABORTING
SHORT_READ

1. global_bof3.jpeg

lepton v1.0-91619e2

START ACHIEVED 1468716625 47000

==97386==ERROR: AddressSanitizer: global-buffer-overflow on address 0x0000008cad0a at pc 0x0000004fe249 bp 0x7fff934d51b0 sp 0x7fff934d51a0
READ of size 2 at 0x0000008cad0a thread T0
#0 0x4fe248 in ProbabilityTablesBase::set_quantization_table(BlockType, unsigned short const_) src/vp8/model/model.hh:233
#1 0x4fe248 in VP8ComponentEncoder::vp8_full_encoder(UncompressedComponents const_, IOUtil::FileWriter_, ThreadHandoff const_, unsigned int) src/lepton/vp8_encoder.cc:465
#2 0x47b3a8 in write_ujpg(std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >) src/lepton/jpgcoder.cc:3660
#3 0x48b4ee in bool std::Bind<bool ((std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >))(std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >_)>::__call<bool, , 0ul, 1ul>(std::tuple<>&&, std::Index_tuple<0ul, 1ul>) /usr/include/c++/6/functional:943
#4 0x48b4ee in bool std::Bind<bool (*(std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >))(std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >)>::operator()<, bool>() /usr/include/c++/6/functional:1002
#5 0x48b4ee in std::Function_handler<bool (), std::Bind<bool (*(std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >))(std::vector<ThreadHandoff, std::allocator >, std::vector<unsigned char, Sirikata::JpegAllocator >)> >::M_invoke(std::Any_data const&) /usr/include/c++/6/functional:1726
#6 0x446089 in std::function<bool ()>::operator()() const /usr/include/c++/6/functional:2136
#7 0x446089 in execute(std::function<bool ()> const&) src/lepton/jpgcoder.cc:1728
#8 0x46ceb5 in process_file(IOUtil::FileReader, IOUtil::FileWriter, int, bool) src/lepton/jpgcoder.cc:1531
#9 0x406c73 in main src/lepton/jpgcoder.cc:773
#10 0x7fe3fb5cd82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#11 0x40afc8 in _start (/home/marco/VulnResearch/misc/lepton/lepton+0x40afc8)

0x0000008cad0a is located 54 bytes to the left of global variable 'chroma_debug_height' defined in 'src/vp8/util/debug.cc:23:5' (0x8cad40) of size 4
0x0000008cad0a is located 6 bytes to the right of global variable 'raw_decoded_fp_Y' defined in 'src/vp8/util/debug.cc:109:5' (0x8cad00) of size 4
SUMMARY: AddressSanitizer: global-buffer-overflow src/vp8/model/model.hh:233 in ProbabilityTablesBase::set_quantization_table(BlockType, unsigned short const*)
Shadow bytes around the buggy address:
0x000080111550: f9 f9 f9 f9 00 f9 f9 f9 f9 f9 f9 f9 00 f9 f9 f9
0x000080111560: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 f9 f9 f9
0x000080111570: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
0x000080111580: 00 00 00 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9
0x000080111590: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
=>0x0000801115a0: 04[f9]f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115b0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115c0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115d0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115e0: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
0x0000801115f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==97386==ABORTING
SHORT_READ

Thank you

Marco

Question with big images

Hi,
when i want to compress a big JPG (around 54MB) with no tuning settings i get the following result but no 'a.lep' file:

./lepton a.jpg a.lep
lepton v1.0-
START ACHIEVED 1470607045 175695
OOM
SHORT_READ

I have to set the -memory up to 3GB to get it working, is that correct?
(using OSX, installed with brew). Would be nice if lepton decides for themselves how much memory is required to compress the image.

./lepton -memory=3072M a.jpg a.lep
lepton v1.0-
START ACHIEVED 1470607052 506527
TS_MAIN (0) 0.000000
TS_MODEL_INIT_BEGIN (0) 10.512067
TS_MODEL_INIT_BEGIN (1) 10.412967
TS_MODEL_INIT_BEGIN (2) 10.413010
TS_MODEL_INIT_BEGIN (3) 10.413037
TS_MODEL_INIT_BEGIN (4) 10.413102
TS_MODEL_INIT_BEGIN (5) 10.432212
TS_MODEL_INIT_BEGIN (6) 10.479795
TS_MODEL_INIT_BEGIN (7) 10.487485
TS_MODEL_INIT (0) 10.516036
TS_MODEL_INIT (1) 10.418249
TS_MODEL_INIT (2) 10.421953
TS_MODEL_INIT (3) 10.416062
TS_MODEL_INIT (4) 10.417307
TS_MODEL_INIT (5) 10.433794
TS_MODEL_INIT (6) 10.481520
TS_MODEL_INIT (7) 10.490361
TS_THREAD_STARTED (0) 0.001805
TS_READ_STARTED (0) 0.001858
TS_READ_FINISHED (0) 0.987120
TS_JPEG_DECODE_STARTED (0) 0.987120
TS_JPEG_DECODE_FINISHED (0) 10.397602
TS_STREAM_MULTIPLEX_STARTED (0) 27.475147
TS_STREAM_MULTIPLEX_FINISHED (0) 27.787765
TS_THREAD_WAIT_STARTED (1) 26.034472
TS_THREAD_WAIT_STARTED (2) 26.034480
TS_THREAD_WAIT_STARTED (3) 26.034482
TS_THREAD_WAIT_STARTED (4) 26.034483
TS_THREAD_WAIT_STARTED (5) 26.034486
TS_THREAD_WAIT_STARTED (6) 26.034487
TS_THREAD_WAIT_STARTED (7) 26.034489
TS_THREAD_WAIT_FINISHED (1) 26.034480
TS_THREAD_WAIT_FINISHED (2) 26.034482
TS_THREAD_WAIT_FINISHED (3) 26.034483
TS_THREAD_WAIT_FINISHED (4) 26.034486
TS_THREAD_WAIT_FINISHED (5) 26.034487
TS_THREAD_WAIT_FINISHED (6) 26.034489
TS_THREAD_WAIT_FINISHED (7) 27.475146
TS_ARITH_STARTED (0) 10.512066
TS_ARITH_STARTED (1) 10.412967
TS_ARITH_STARTED (2) 10.413009
TS_ARITH_STARTED (3) 10.413037
TS_ARITH_STARTED (4) 10.413102
TS_ARITH_STARTED (5) 10.432212
TS_ARITH_STARTED (6) 10.479794
TS_ARITH_STARTED (7) 10.487484
TS_ARITH_FINISHED (0) 26.034472
TS_ARITH_FINISHED (1) 23.823887
TS_ARITH_FINISHED (2) 22.709831
TS_ARITH_FINISHED (3) 23.167249
TS_ARITH_FINISHED (4) 24.222650
TS_ARITH_FINISHED (5) 24.642485
TS_ARITH_FINISHED (6) 25.132216
TS_ARITH_FINISHED (7) 27.475085
TS_STREAM_FLUSH_STARTED (0) 27.787765
TS_STREAM_FLUSH_FINISHED (0) 27.787767
TS_DONE (0) 27.787819
77233505 bytes needed to decompress this file
37811888 54085715
69.91%
START ACHIEVED 1470607080 541507
TIMING (new method): 18.372288 to first byte 18.372288 total
Read took: 0.000197
TS_MAIN (0) 0.000000
TS_MODEL_INIT_BEGIN (0) 28.143415
TS_MODEL_INIT_BEGIN (1) 28.143373
TS_MODEL_INIT_BEGIN (2) 28.143380
TS_MODEL_INIT_BEGIN (3) 28.143389
TS_MODEL_INIT_BEGIN (4) 28.146351
TS_MODEL_INIT_BEGIN (5) 28.150565
TS_MODEL_INIT_BEGIN (6) 28.150580
TS_MODEL_INIT_BEGIN (7) 28.158941
TS_MODEL_INIT (0) 28.149988
TS_MODEL_INIT (1) 28.145129
TS_MODEL_INIT (2) 28.145243
TS_MODEL_INIT (3) 28.145077
TS_MODEL_INIT (4) 28.148581
TS_MODEL_INIT (5) 28.151989
TS_MODEL_INIT (6) 28.165408
TS_MODEL_INIT (7) 28.160659
TS_THREAD_STARTED (0) 28.036782
TS_READ_STARTED (0) 28.036851
TS_READ_FINISHED (0) 28.037023
TS_STREAM_MULTIPLEX_STARTED (0) 28.143414
TS_STREAM_MULTIPLEX_STARTED (1) 28.143372
TS_STREAM_MULTIPLEX_STARTED (2) 28.143380
TS_STREAM_MULTIPLEX_STARTED (3) 28.143389
TS_STREAM_MULTIPLEX_STARTED (4) 28.146351
TS_STREAM_MULTIPLEX_STARTED (5) 28.150565
TS_STREAM_MULTIPLEX_STARTED (6) 28.150580
TS_STREAM_MULTIPLEX_STARTED (7) 28.158940
TS_STREAM_MULTIPLEX_FINISHED (0) 28.150405
TS_STREAM_MULTIPLEX_FINISHED (1) 28.145878
TS_STREAM_MULTIPLEX_FINISHED (2) 28.145659
TS_STREAM_MULTIPLEX_FINISHED (3) 28.145555
TS_STREAM_MULTIPLEX_FINISHED (4) 28.148985
TS_STREAM_MULTIPLEX_FINISHED (5) 28.152365
TS_STREAM_MULTIPLEX_FINISHED (6) 28.165804
TS_STREAM_MULTIPLEX_FINISHED (7) 28.161039
TS_THREAD_WAIT_STARTED (1) 44.732890
TS_THREAD_WAIT_STARTED (2) 44.758106
TS_THREAD_WAIT_STARTED (3) 44.783672
TS_THREAD_WAIT_STARTED (4) 44.809829
TS_THREAD_WAIT_STARTED (5) 44.834752
TS_THREAD_WAIT_STARTED (6) 44.859716
TS_THREAD_WAIT_STARTED (7) 44.883532
TS_THREAD_WAIT_FINISHED (1) 44.732898
TS_THREAD_WAIT_FINISHED (2) 44.758110
TS_THREAD_WAIT_FINISHED (3) 44.783676
TS_THREAD_WAIT_FINISHED (4) 44.809833
TS_THREAD_WAIT_FINISHED (5) 44.834755
TS_THREAD_WAIT_FINISHED (6) 44.859720
TS_THREAD_WAIT_FINISHED (7) 46.382081
TS_ARITH_STARTED (0) 28.143414
TS_ARITH_STARTED (1) 28.143371
TS_ARITH_STARTED (2) 28.143380
TS_ARITH_STARTED (3) 28.143389
TS_ARITH_STARTED (4) 28.146350
TS_ARITH_STARTED (5) 28.150564
TS_ARITH_STARTED (6) 28.150579
TS_ARITH_STARTED (7) 28.158939
TS_ARITH_FINISHED (0) 44.732889
TS_ARITH_FINISHED (1) 42.568798
TS_ARITH_FINISHED (2) 41.561109
TS_ARITH_FINISHED (3) 42.013757
TS_ARITH_FINISHED (4) 43.175662
TS_ARITH_FINISHED (5) 43.604729
TS_ARITH_FINISHED (6) 44.189564
TS_ARITH_FINISHED (7) 46.382012
TS_JPEG_RECODE_STARTED (0) 44.732889
TS_JPEG_RECODE_STARTED (1) 44.732898
TS_JPEG_RECODE_STARTED (2) 44.758110
TS_JPEG_RECODE_STARTED (3) 44.783676
TS_JPEG_RECODE_STARTED (4) 44.809833
TS_JPEG_RECODE_STARTED (5) 44.834755
TS_JPEG_RECODE_STARTED (6) 44.859720
TS_JPEG_RECODE_STARTED (7) 46.382081
TS_JPEG_RECODE_FINISHED (0) 46.409170
TS_JPEG_RECODE_FINISHED (1) 44.758106
TS_JPEG_RECODE_FINISHED (2) 44.783672
TS_JPEG_RECODE_FINISHED (3) 44.809829
TS_JPEG_RECODE_FINISHED (4) 44.834752
TS_JPEG_RECODE_FINISHED (5) 44.859716
TS_JPEG_RECODE_FINISHED (6) 44.883531
TS_JPEG_RECODE_FINISHED (7) 46.409054
TS_DONE (0) 46.409178
149788324 bytes needed to decompress this file
37811888 54085715
69.91%

md5.c.obj not found

Hi,

I built it on Windows with generator CodeBlocks MinGW
cmake .. -G "CodeBlocks - MinGW Makefiles"

It was successful but then when I open .cbp file in CodeBlock and build the project, I got this

C:\PROGRA~2\CODEBL~1\MinGW\bin\gcc.exe   @CMakeFiles/localmd5.dir/includes_C.rsp -DHAVE_CONFIG_H   /arch:SSE2   -DDEFAULT_ALLOW_PROGRESSIVE -DHIGH_MEMORY -o CMakeFiles\localmd5.dir\dependencies\md5\md5.c.obj   -c "D:\Open-source code\lepton-master\dependencies\md5\md5.c"
gcc.exe: error: /arch:SSE2: No such file or directory
mingw32-make.exe[2]: *** [CMakeFiles/localmd5.dir/dependencies/md5/md5.c.obj] Error 1
mingw32-make.exe[1]: *** [CMakeFiles/localmd5.dir/all] Error 2
mingw32-make.exe: *** [all] Error 2
CMakeFiles\localmd5.dir\build.make:62: recipe for target 'CMakeFiles/localmd5.dir/dependencies/md5/md5.c.obj' failed
mingw32-make.exe[2]: Leaving directory 'D:/Open-source code/lepton-master/build'
CMakeFiles\Makefile2:66: recipe for target 'CMakeFiles/localmd5.dir/all' failed
mingw32-make.exe[1]: Leaving directory 'D:/Open-source code/lepton-master/build'
D:/Open-source code/lepton-master/build/Makefile:126: recipe for target 'all' failed

There's no "CMakeFiles\localmd5.dir\dependencies\md5\md5.c.obj", the file does not exist, the folder "md5" is empty
My system: CodeBlocks 16.01 with GCC 4.9.2 running on Windows 10 64 bit
Please can you advise?

UB with USE_STANDARD_MEMORY_ALLOCATORS

I'm compiling project with -DUSE_STANDARD_MEMORY_ALLOCATORS
So I have next one error when decompress .lep to .jpg
malloc: *** error for object 0x106f431d0: pointer being freed was not allocated

Here is ASAN output

==5914==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x00010499e9d0 in thread T0
    #0 0x10166db89 in wrap_free (libclang_rt.asan_osx_dynamic.dylib+0x48b89)
    #1 0x1013b3ef4 in custom_free memory.cc:111
    #2 0x1012e83b8 in Sirikata::JpegAllocator<unsigned char>::free_wrapper(void*, void*) Allocator.hh:53
    #3 0x10125fffa in Sirikata::JpegAllocator<unsigned char>::deallocate(unsigned char*, unsigned long) Allocator.hh:145
    #4 0x1012e4b38 in std::__1::__vector_base<unsigned char, Sirikata::JpegAllocator<unsigned char> >::~__vector_base() memory:1487
    #5 0x1012e4734 in std::__1::vector<unsigned char, Sirikata::JpegAllocator<unsigned char> >::~vector() vector:457
    #6 0x1012d87c4 in std::__1::vector<unsigned char, Sirikata::JpegAllocator<unsigned char> >::~vector() vector:457
    #7 0x1012e80d6 in Sirikata::MemReadWriter::~MemReadWriter() MemReadWriter.hh:3
    #8 0x1012dc9a4 in Sirikata::MemReadWriter::~MemReadWriter() MemReadWriter.hh:3
    #9 0x1012b617c in read_ujpg() jpgcoder.cc:3959
    #10 0x101313727 in bool std::__1::__invoke_void_return_wrapper<bool>::__call<bool (*&)()>(bool (*&&&)()) __functional_base:416
    #11 0x1013136b8 in std::__1::__function::__func<bool (*)(), std::__1::allocator<bool (*)()>, bool ()>::operator()() functional:1437
    #12 0x1012d88f4 in std::__1::function<bool ()>::operator()() const functional:1817
    #13 0x10129c141 in execute(std::__1::function<bool ()> const&) jpgcoder.cc:1754
    #14 0x10129257b in process_file(IOUtil::FileReader*, IOUtil::FileWriter*, int, bool) jpgcoder.cc:1584
    #15 0x10128753b in main jpgcoder.cc:803
    #16 0x7fff935ac5ac in start (libdyld.dylib+0x35ac)
    #17 0x2  (<unknown module>)

0x00010499e9d0 is located 4560 bytes inside of 2080376256-byte region [0x00010499d800,0x00018099ddc0)
allocated by thread T0 here:
    #0 0x10166df27 in wrap_calloc (libclang_rt.asan_osx_dynamic.dylib+0x48f27)
    #1 0x1014cf868 in Sirikata::memmgr_init(unsigned long, unsigned long, unsigned long, unsigned long, bool) MemMgrAllocator.cc:184
    #2 0x101287160 in main jpgcoder.cc:747
    #3 0x7fff935ac5ac in start (libdyld.dylib+0x35ac)
    #4 0x2  (<unknown module>)

git really needed?

I get warnings when git is not installed, but it compiles fine.

Is it really needed?

Assertion failure on truncated file

The following file: crash.zip (or rather, the file inside said archive, as Github won't allow me to upload it directly), causes an assertion failure on default settings on my machine:

lepton v1.0-a34ee2f
lepton: src/lepton/bitops.cc:459: unsigned int ibytestream::read(unsigned char*, unsigned int): Assertion `size' failed.

(Alternatively, a hexdump of said file is d8ff 30ff 0200)

Inconsistency between autoreconf and CMake

I discover CMake with this project and it seems to be a great tool.
However, providing both autoreconf and CMake way makes it a little hard to understand the build logic, especially because there's inconsistency between the two.

On the same linux system, CMake and autoreconf will not set the same compiler flags (With CMake: static, msse4.2 ... and with autoreconf: mssse3 ... ).
CMake also adds the avx target.

Shouldn't it be better to maintain only the CMake version?

Fails compressing a JPG file

I tryed the build provided for windows, but when i use a jpg it returns an error:
C:\Users\VA\Downloads>lepton.exe 111.jpg 111.lep lepton v1.0-
Input Size 90750 != Roundtrip Size 0 33448df365f5979a2e4d77ba40b6d4ff != 0600000005000000ffffffff00000000 ROUNDTRIP_FAILURE

The image file is here (decompress it): 111.zip

Hope it helps improve the code.

reserved identifier violation

I would like to point out that identifiers like "_JPGCODER_HH_" and "_LEPTON_CODEC_HH_" do eventually not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

• fprintf ⇒ fork_serve
• signal ⇒ socket_serve

Build failures on OS X

I am trying to submit lepton to Homebrew (version 1.0 with the header patch in d122450). It compiles fine locally on my Mid-2015 15'' rMBP with OS X 10.11.5, but fails on Homebrew's Jenkins servers on all three versions of OS X tested: 10.11, 10.10 and 10.9.

Specifically,

10.11 (build log):

/usr/local/Library/ENV/4.3/clang++    -I/tmp/lepton-20160715-76959-d9zmzc/lepton-1.0 -I/tmp/lepton-20160715-76959-d9zmzc/lepton-1.0/src/vp8/util -I/tmp/lepton-20160715-76959-d9zmzc/lepton-1.0/src/vp8/model -I/tmp/lepton-20160715-76959-d9zmzc/lepton-1.0/src/vp8/encoder -I/tmp/lepton-20160715-76959-d9zmzc/lepton-1.0/src/vp8/decoder  -std=c++11 -fno-exceptions -fno-rtti -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.11   -march=core-avx2   -o CMakeFiles/lepton-avx.dir/src/lepton/idct.cc.o -c /tmp/lepton-20160715-76959-d9zmzc/lepton-1.0/src/lepton/idct.cc
/tmp/lepton-20160715-76959-d9zmzc/lepton-1.0/src/lepton/idct.cc:228:15: error: always_inline function '_mm_mullo_epi32' requires target feature 'sse4.1', but would be inlined into function 'idct_sse' that is compiled without support for 'sse4.1'
        xv8 = _mm_mullo_epi32(_mm_set1_epi32(w7), _mm_add_epi32(xv4, xv5));
              ^
        ...

10.10 (build log) and 10.9 (build log):

In file included from /tmp/lepton-20160715-88309-10duq3n/lepton-1.0/src/lepton/jpgcoder.cc:64:
In file included from /tmp/lepton-20160715-88309-10duq3n/lepton-1.0/src/lepton/vp8_decoder.hh:4:
In file included from /tmp/lepton-20160715-88309-10duq3n/lepton-1.0/src/lepton/lepton_codec.hh:4:
In file included from /tmp/lepton-20160715-88309-10duq3n/lepton-1.0/src/vp8/model/model.hh:10:
In file included from /tmp/lepton-20160715-88309-10duq3n/lepton-1.0/src/vp8/model/numeric.hh:11:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/include/smmintrin.h:28:2: error: "SSE4.1 instruction set not enabled"
#error "SSE4.1 instruction set not enabled"

Also

/tmp/lepton-20160715-88309-10duq3n/lepton-1.0/src/vp8/model/numeric.hh:295:32: error: use of undeclared identifier '_mm_mullo_epi32'; did you mean '_mm_mullo_epi16'?
    __m128i t = _mm_srli_epi32(_mm_mullo_epi32(m, abs_num), log_max_numerator);
                               ^~~~~~~~~~~~~~~
                               _mm_mullo_epi16
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/include/emmintrin.h:727:1: note: '_mm_mullo_epi16' declared here
_mm_mullo_epi16(__m128i __a, __m128i __b)
^

etc.

Any idea what's going on here? Thanks.

P.S. The linked build logs contain info (OS, CPU, compiler, etc.) about build environments at the end.

---

Note: I removed some of my comments on this thread because they belong to a kind of pointless argument that skews the thread.

create 7-Zip plugin

please create a plugin for 7-Zip so that this open and free archiver can also compress JPEG.
This would also help making Lepton more famous in the world.

there are already plugins that work with 7-Zip (including descriptions and sources):

ZSTD

https://github.com/mcmilk/7-Zip-Zstd

LZHAM
http://richg42.blogspot.de/search?updated-max=2015-12-03T16:22:00-08:00&max-results=7&start=21&by-date=false
https://github.com/richgel999/lzham_codec_devel

Lepton as a shared library

I'd like to suggest supporting a shared library build with an simple ABI-neutral API. It would be very useful for creating wrappers (for example, I'd like to create a C# P/Invoke-based wrapper for .NET on Windows and a Go one to aid creating a simple http+jpeg<->GridFS+lepton proxy).
Implementing ABI-neutral stream-based API might be challenging, since different systems/CRTs use different stream implementations (hence trying to provide external implementations of IOUtil::FileReader/IOUtil::FileWriter might quickly become to complex), so maybe the first version should only support byte* input and byte* output (along with a function to free the result).

So, the simplest prototype seems to be this one:

int32_t process_data(const uint8_t* input_data, uint32_t input_length, const uint8_t** output_data, uint32_t* output_length);

The return code can used as a success (0) or failure (non-zero) indicator for now.

I might take a look at it myself and maybe even submit a PR, but would like to discuss this with authors -- you probably had some thoughts going in this direction and might already have a few brighter ideas.

Dependencies and build instructions

Can you provide dependencies and build instructions in the Readme?

Test_Invariants Build Failure

I've tested on a Windows 10 VM running Ubuntu 16.04 LTS.
During the build process I fail to pass the test_invariants, I then looked into the log file and saw this:

mmap: Cannot allocate memory
Insufficient memory: unable to mmap or calloc 1006633216 bytes
FAIL test_suite/test_invariants (exit status: 37)

I don't understand why a calloc would have failed since I really doubt I don't have enough contiguous memory available.

Thanks,
Alessandro

Error with avx : ROUNDTRIP_FAILURE

When trying lepton-avx on my Core i7 2760QM which is said to support AVX, I have the following error :

lepton v1.0-
Input Size 6632440 != Roundtrip Size 0
ebde2fe7982b79ad101495a1bdbe4e90 != 0600000005000000ffffffff00000000
ROUNDTRIP_FAILURE

Running lepton.exe is completely fine.

jpegoptim

Hi,

Do you guys have any benchmarks or comments on this tool versus jpegoptim?

Unable to build on Win7

When I try to build the solution in visual studio it throws me numerous instances of the below error:

error C4430: missing type specifier - int assumed. Note: C++ does not support default-int ..\lepton\src\vp8\util\memory.cc

as well as:

Error 116 error C3867: 'UncompressedComponents::header_': function call missing argument list; use '&UncompressedComponents::header_' to create a pointer to member ..\lepton\src\lepton\uncompressed_components.hh line 175

I'm building v1.0 from commit 911d79a

How to get 5MB/s encode rate?

I am testing on Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, using the sockettester.py. But only get 1.2MB/s for many jpgs. Is there any trick or argument I need to set to make it faster?

aligned_memchr16ff is not

The function assumes, its argument is properly aligned for casting it to __m128i const *. The assumption appears valid on FreeBSD/amd64, but on a 32-bit FreeBSD/i386 it causes core-dumps.

The below patch makes the function forego the SSE-trickery and always use memchr on i386:

+++ src/lepton/recoder.cc
@@ -99,5 +99,5 @@

 static bool aligned_memchr16ff(const unsigned char *local_huff_data) {
-#if 1
+#if !defined(__i386__)
     __m128i buf = _mm_load_si128((__m128i const*)local_huff_data);
     __m128i ff = _mm_set1_epi8(-1);

Are you normally building/testing on 32-bit systems? With this patch I no longer have crashes running the test suit, but 5 of the 40 tests still fail...

Using ANS coding for lepton

Right now we encode and decode symbols forward.
It would be interesting if we buffered symbols in boolwriter.hh and then used one of the public domain ANS systems to write them backwards.
Then we could reread them using boolreader.hh

Would be interesting to see if the decoder becomes faster.

Some JPEG files become even larger after lepton compression

Example: https://cloud.githubusercontent.com/assets/1442206/17919118/005ee04c-69dc-11e6-83e0-4d027edfb399.jpg

It gives 102% (bigger file)

Unable to build on Debian

Hi,

On a virtualized box we have for testing thing out:

$ make && make check && make install
make  all-am
make[1]: entrant dans le répertoire « /usr/local/src/lepton-1.2.1 »
depbase=`echo dependencies/zlib/inflate.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
        gcc -DHAVE_CONFIG_H -I.    -std=c99 -DDEFAULT_ALLOW_PROGRESSIVE -DHIGH_MEMORY  -pedantic -Wall -Wextra -Wno-write-strings -Wno-unused-parameter -march=native  -DNDEBUG -g -O2 -MT dependencies/zlib/inflate.o -MD -MP -MF $depbase.Tpo -c -o dependencies/zlib/inflate.o dependencies/zlib/inflate.c &&\
        mv -f $depbase.Tpo $depbase.Po
dependencies/zlib/inflate.c:1:0: error: CPU you selected does not support x86-64 instruction set
make[1]: *** [dependencies/zlib/inflate.o] Erreur 1
make[1]: quittant le répertoire « /usr/local/src/lepton-1.2.1 »
make: *** [all] Erreur 2

On a dedicated box:

$ make && make check && make install
[...]
In file included from src/io/Reader.hh:32:0,
                 from src/io/MemReadWriter.hh:1,
                 from src/io/MemReadWriter.cc:1:
src/io/Allocator.hh: In instantiation of ‘T* Sirikata::JpegAllocator<T>::allocate(Sirikata::JpegAllocator<T>::size_type, const void*) [with T = unsigned char; Sirikata::JpegAllocator<T>::pointer = unsigned char*; Sirikata::JpegAllocator<T>::size_type = long unsigned int]’:
/usr/include/c++/4.7/bits/stl_vector.h:169:51:   required from ‘std::_Vector_base<_Tp, _Alloc>::pointer std::_Vector_base<_Tp, _Alloc>::_M_allocate(std::size_t) [with _Tp = unsigned char; _Alloc = Sirikata::JpegAllocator<unsigned char>; std::_Vector_base<_Tp, _Alloc>::pointer = unsigned char*; std::size_t = long unsigned int]’
/usr/include/c++/4.7/bits/stl_vector.h:1109:44:   required from ‘std::vector<_Tp, _Alloc>::pointer std::vector<_Tp, _Alloc>::_M_allocate_and_copy(std::vector<_Tp, _Alloc>::size_type, _ForwardIterator, _ForwardIterator) [with _ForwardIterator = __gnu_cxx::__normal_iterator<const unsigned char*, std::vector<unsigned char, Sirikata::JpegAllocator<unsigned char> > >; _Tp = unsigned char; _Alloc = Sirikata::JpegAllocator<unsigned char>; std::vector<_Tp, _Alloc>::pointer = unsigned char*; std::vector<_Tp, _Alloc>::size_type = long unsigned int]’
/usr/include/c++/4.7/bits/vector.tcc:186:19:   required from ‘std::vector<_Tp, _Alloc>& std::vector<_Tp, _Alloc>::operator=(const std::vector<_Tp, _Alloc>&) [with _Tp = unsigned char; _Alloc = Sirikata::JpegAllocator<unsigned char>]’
src/io/MemReadWriter.hh:24:19:   required from here
src/io/Allocator.hh:138:6: error: ‘custom_exit’ was not declared in this scope
make[1]: *** [src/io/MemReadWriter.o] Error 1
make[1]: Leaving directory `/usr/local/src/lepton-1.2.1'
make: *** [all] Error 2

This is using the releases tar.gz files. The ./autogen.sh script failed to run its git commands since that tool is not installed on the boxes.

Please tell me what information you need to debug this.

gcc version 4.7.2 (Debian 4.7.2-5)

serve images live?

Any examples on how to serve images live? The README describes this as a possibility.

Listen server hangs on OS X

From the current master in github.com, I clone master and cd into that directory. In one terminal I start lepton with:

./lepton -listen=2020

In a second terminal, I run:

cat images/iphone.jpg | netcat localhost 2020

On my linux virtual machine, this works exactly as expected - binary compressed data is written to stdout. On mac it hangs at jpgcoder.cc:1532 while consuming input:

$ ./lepton -listen=2020
/tmp/e4a49584aa-2616-b410-c273f5cf602a-2c.uport
START ACHIEVED 1469565561 607237

It will sit there indefinitely until you ctrl-c the window running netcat. Then you'll see the output window print a bunch of output, seemingly indicating that it is processing the file.

<many lines...>
TS_DONE (0) 35.022779
12133028 bytes needed to decompress this file
DONE
Child 60631 exited with signal 13

I've attached the full output log from the server. Any help you can provide would be great.

log.txt

missing VP8 copyright ?

Should you add http://www.webmproject.org/license/software/ to the repo ?

Javascript

Are there any plans for a javascript decoder?