google / open-vcdiff

An encoder/decoder for the VCDIFF (RFC3284) format

open-vcdiff README

open-vcdiff is an encoder and decoder for the VCDIFF format, as described in RFC 3284: The VCDIFF Generic Differencing and Compression Data Format.

You will need to first synchronize gflags and gtest by running git submodule update --init --recursive. Or if you have system installed gflags and/or gtest libraries you can provide -Dvcdiff_use_system_gflags=ON and -Dvcdiff_use_system_gtest=ON for cmake invokation in the build step.

A library with a simple API is included, as well as a command-line executable that can apply the encoder and decoder to source, target, and delta files. For further details, please refer to this link.

open-vcdiff comes with a CMake build script ( CMakeLists.txt) that can be used on a wide range of platforms ("C" stands for cross-platform.). If you don't have CMake installed already, you can download it for free from http://www.cmake.org/.

CMake works by generating native makefiles or build projects that can be used in the compiler environment of your choice. The typical workflow starts with:

mkdir mybuild       # Create a directory to hold the build output.
cd mybuild
cmake ${OPEN_VCDIFF_DIR}  # Generate native build scripts.

If you want to disable build of build tests and/or executable and build libraries only replace last command with

cmake -Dvcdiff_build_test=OFF -Dvcdiff_build_exec=OFF ${OPEN_VCDIFF_DIR}

If you are on a *nix system, you should now see a Makefile in the current directory. Just type 'make' to build gtest.

If you use Windows and have Visual Studio installed, a gtest.sln file and several .vcproj files will be created. You can then build them using Visual Studio.

On Mac OS X with Xcode installed, a .xcodeproj file will be generated.

After compilation you should have the unit tests as well as vcdiff, a simple command-line utility to run the encoder and decoder. Typical usage of vcdiff is as follows (the < and > are file redirect operations, not optional arguments):

vcdiff encode -dictionary file.dict < target_file > delta_file
vcdiff decode -dictionary file.dict < delta_file  > target_file

To see the command-line syntax of vcdiff, use vcdiff --help or just vcdiff.

To run tests just use make test inside build directory.

To call the encoder from C++ code, assuming that dictionary, target, and delta are all std::string objects:

#include <google/vcencoder.h>  // Read this file for interface details

// [...]

open_vcdiff::VCDiffEncoder encoder(dictionary.data(), dictionary.size());
encoder.SetFormatFlags(open_vcdiff::VCD_FORMAT_INTERLEAVED);
encoder.Encode(target.data(), target.size(), &delta);

Calling the decoder is just as simple:

#include <google/vcdecoder.h>  // Read this file for interface details

// [...]

open_vcdiff::VCDiffDecoder decoder;
decoder.Decode(dictionary.data(), dictionary.size(), delta, &target);

When using the encoder, the C++ application must be linked with the library options -lvcdcom and -lvcdenc; when using the decoder, it must be linked with -lvcdcom and -lvcddec.

To verify that the package works on your system, especially after making modifications to the source code, please run the unit tests using make check.

For further details, please refer to this link.

open-vcdiff's Issues

Redundant comma in json output

Original issue 37 created by sunzhuoshi on 2012-10-31T15:24:04.000Z:

What steps will reproduce the problem?

1. create file "test1.js", input "test1", and save it.
2. create file "test2.js", input "test2", and save it, too.
3. run command line:
   vcdiff encode -json -dictionary test1.js -target test2.js -delta delta.js

What is the expected output? What do you see instead?
the excpeted output:
["test2"]
what I see:
["test2",]

What version of the product are you using? On what operating system?
version: 0.8.3
OS: Mac OS X 10.7.4

Please provide any additional information below.

autotools issue

Original issue 43 created by mcrapet on 2014-04-01T12:58:23.000Z:

Hi,

Using 0.8.3 on Linux. Got some situation using openembedded framework and autoreconf:

aclocal: error: non-option arguments are not accepted: ''.

This comes from ACLOCAL_AMFLAGS in Makefile.am which is multiline.
Here is a fix patch.

Regards,
Matthieu

"Value too large for defined data type" when dictionary file is > 4GB

Original issue 16 created by jeff.gustafson on 2008-10-09T22:13:01.000Z:

What steps will reproduce the problem?

1. File larger than 4GB
   2.
   3.

What is the expected output? What do you see instead?
normal operation

What version of the product are you using? On what operating system?
0.2

Please provide any additional information below.

Unit test blockhash_test fails on FreeBSD platform

Original issue 2 created by openvcdiff on 2008-08-29T22:43:05.000Z:

What steps will reproduce the problem?

1. Download and extract open-vcdiff v0.1 source tarball
   on a FreeBSD 6.0 machine.
2. In top extracted directory, execute: ./configure && make check

What is the expected output?

The "make check" should finish by reporting:

All 13 tests passed

What do you see instead?

The output ends with the following messages:

1 of 13 tests failed

Please report to [email protected]

*** Error code 1

Stop in open-vcdiff-0.1.

Looking at the earlier output, the test that fails is found to be
blockhash_test:
[ FAILED ] 1 test, listed below:
[ FAILED ] BlockHashTest.BlockContentsMatchIsAsFastAsBlockCompareWords

Decoder should not crash if it runs out of memory

Original issue 8 created by openvcdiff on 2008-09-11T23:02:35.000Z:

What steps will reproduce the problem?
(On a Linux system, or anywhere that ulimit is available:)

1. Build and/or install the vcdiff executable for open-vcdiff v0.2.
2. Download the attached delta file "allocates_4gb.vcdiff".
3. Limit memory usage to 32MB per process, and create a dummy dictionary
   whose contents will be ignored:
   ulimit -d 32768
   ulimit -m 32768
   echo foo > dummy.dict
4. Try to decode the delta file:
   vcdiff decode -dictionary dummy.dict -delta allocates_4gb.vcdiff

What is the expected output? What do you see instead?
The delta file tries to allocate 4GB of memory but the ulimit only allows
32MB memory usage. The expected behavior is for the vcdiff executable to
output an error message and return a non-zero (unsuccessful) result. What
happens instead is that the decoder throws an unhandled C++ exception:

terminate called after throwing an instance of 'std::length_error'
what(): basic_string::_S_create
Aborted

A relatively small malicious delta file can be tailored to produce an
arbitrarily large target file and so cause an out-of-memory crash, or at
least a huge slowdown as virtual memory starts swapping to disk. The
decoder does have the restriction that no target window can be larger than
64MB in size, but a malicious encoding can still use multiple target
windows to reach an arbitrary size. The attached file contains 64 windows
of 64MB each, for a total of 4GB.

If open-vcdiff is embedded within a larger application, then blocking that
application or throwing it an unhandled exception may have disastrous
consequences.

Adding try/catch statements around each of the external interfaces will
keep the unhandled exception from crashing the application that calls the
open-vcdiff decoder library, but it will not prevent the slowdown
associated with swapping virtual memory pages before running out of memory
entirely.

To solve this problem, the decoder should have configurable limits on some
of the following values:

1. The maximum size of a single COPY or RUN instruction.
2. The maximum size of a delta window (this is currently 64MB.)
3. The maximum size of a delta file.

Restricting (2.) will also implicitly restrict (1.), and restricting (3.)
will also implicitly restrict (1.) and (2.)

Fails to compile on OS X 10.9 (missing <tr1/tuple>)

Original issue 41 created by Jens.Alfke on 2013-10-27T04:00:35.000Z:

What steps will reproduce the problem?

1. Install OS X 10.9 and Xcode 5.0.1
2. Download and unpack open-vcdiff 0.8.3 tar ball
3. ./configure
4. make

What is the expected output? What do you see instead?

...
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/zlib -I./gflags/src -I./gtest -I./gtest/include -DNDEBUG -DNO_THREADS -Wall -Wwrite-strings -Woverloaded-virtual -W -g -O2 -MT gtest.lo -MD -MP -MF .deps/gtest.Tpo -c gtest/src/gtest.cc -fno-common -DPIC -o .libs/gtest.o
In file included from gtest/src/gtest.cc:34:
In file included from ./gtest/include/gtest/gtest.h:55:
In file included from ./gtest/include/gtest/internal/gtest-internal.h:40:
./gtest/include/gtest/internal/gtest-port.h:433:10: fatal error: 'tr1/tuple' file not found

include <tr1/tuple>

     ^

1 error generated.
make: *** [gtest.lo] Error 1

What version of the product are you using? On what operating system?

open-vcdiff 0.8.3
Mac OS X 10.9
Xcode 5.0.1

This is the first time I've tried to build this, so I don't know whether 0.8.3 would build on an earlier OS X. (But the home page says it's compatible, so I imagine 0.8.3 must've been tested on 10.8?)

Can't find zlib.h

Original issue 35 created by jbeder on 2012-04-30T20:43:23.000Z:

What steps will reproduce the problem?

1. Download open-vcdiff-0.8.3.tar.gz
2. Open vsprojects/open-vcdiff.sln in Visual Studio 2010
3. Allow it to upgrade the solution file
4. Build all

What is the expected output? What do you see instead?

It can't find the header file zlib.h.

If you add, in each project, the include directory "../../src/zlib", then it builds correctly.

What version of the product are you using? On what operating system?

Open vcdiff 0.8.3
Visual Studio 2010
Windows 7
Please provide any additional information below.

"make check" fails on AIX 5.3.8

Original issue 25 created by openvcdiff on 2010-02-16T19:04:14.000Z:

What steps will reproduce the problem?

1. Download the source tarball for open-vcdiff to a machine running AIX 5.3.8.
2. Run ./configure && make && make check.

What is the expected output? What do you see instead?

The "make check" output should show zero failures, but ends with the

following (see attached file):

3 of 18 tests failed

Please report to [email protected]

The specific unit test failures are as follows:

1. BlockContentsMatchIsAsFastAsBlockCompareWords in blockhash_test.cc
   fails. This means that the label VCDIFF_USE_BLOCK_COMPARE_WORDS should be
   defined for this compiler and architecture in order to improve performance.
   Currently the label is defined only when the compiler is gcc and the
   architecture is x86.
2. vcdecoder5_test throws an unhandled exception "std::bad_alloc" during
   the test VCDiffLargeTargetTest.Decode. That test allocates, frees, and
   reallocates chunks of 64MB of memory, and should have a maximum of two such
   chunks allocated at any given moment. In this case, it's possible that the
   ulimit settings on the build machine do not allow a user process to
   allocate 128MB of virtual memory.
3. Test 15 in vcdiff_test.sh fails: "vcdiff with /dev/kmem as dictionary
   should fail, but succeeded."

Recursive directory diff support

Original issue 30 created by schmidlin on 2011-09-04T19:32:39.000Z:

Please add a feature for recursive diff between two directories, creating a single patch that describes the differences.

API should use void*, not char*, for binary data

Original issue 4 created by Jens.Alfke on 2008-09-05T20:28:44.000Z:

Several API methods take parameters that are pointers to arbitrary binary data. These parameters
are typed as "const char*". Because of this, clients of the API that have data stored in other types,
have to static_cast their pointers to pass them to vcdiff.

Instead, vcdiff's API should use "const void_" instead of "const char_". This permits any const
pointer to be passed in without a type-cast, and also makes it clear to the human reader that the
method takes arbitrary data, not just C strings. This is the usual convention in C/C++ APIs
nowadays.

As a specific example, change vcencoder.h:70 from:
HashedDictionary(const char* dictionary_contents,
size_t dictionary_size);
to:
HashedDictionary(const void* dictionary_contents,
size_t dictionary_size);
(There are other instances of this at other points in the API.)

use unique_ptr instead of auto_ptr in c++11 builds

Original issue 42 created by [email protected] on 2014-03-19T22:45:47.000Z:

std::auto_ptr is deprecated in c++11, let's use std::unique_ptr instead in that case.

Upgrade dependencies and switch to cmake.

This is one single issue because one is directly tied with the other. Both gflags and gtest now use cmake and they no longer work with our autotools script. We should update them at the same time as we change the build mechanism.

Main points:

• Move the gflags and gtest folders to a third_party directory.
• Update gflags and gtest to the latest released version.
• Move gflags to a git submodule referencing the gflags github repository.
• Create the cmake files and remove the old build scripts.

VCD_COMPILE_ASSERT warns with -Wunused-local-typedef

Both gcc and clang warn on unused typedefs in recent versions. VCD_COMPILE_ASSERT triggers this. Please apply this patch to fix this for projects using c++11 (like e.g. chromium):

diff --git a/src/compile_assert.h b/src/compile_assert.h
index 3f4344f..86a1269 100644
--- a/src/compile_assert.h
+++ b/src/compile_assert.h
@@ -46,9 +46,13 @@ struct CompileAssert {

 }  // namespace open_vcdiff

+#if __cplusplus >= 201103L
+#define VCD_COMPILE_ASSERT(expr, msg) static_assert(expr, #msg)
+#else
 #define VCD_COMPILE_ASSERT(expr, msg) \
   typedef open_vcdiff::CompileAssert<static_cast<bool>(expr)> \
       msg[static_cast<bool>(expr) ? 1 : -1]
+#endif

 // Implementation details of VCD_COMPILE_ASSERT:
 //

Compile error on Oracle Linux 7.2

[root@3fe56e8adf47 open-vcdiff]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @file support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
./configure: line 6848: /usr/bin/file: No such file or directory
checking for mt... no
checking if : is a manifest tool... no
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for ANSI C header files... (cached) yes
checking ext/rope usability... no
checking ext/rope presence... no
checking for ext/rope... no
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for unistd.h... (cached) yes
checking windows.h usability... no
checking windows.h presence... no
checking for windows.h... no
checking for gettimeofday... yes
checking for QueryPerformanceCounter... no
checking for memalign... yes
checking for posix_memalign... yes
checking for mprotect... yes
checking for stdint.h... (cached) yes
checking for sys/types.h... (cached) yes
checking for inttypes.h... (cached) yes
checking fnmatch.h usability... yes
checking fnmatch.h presence... yes
checking for fnmatch.h... yes
checking for uint16_t... yes
checking for u_int16_t... yes
checking for int16... no
checking for strtoll... yes
checking for strtoq... yes
checking for setenv... yes
checking for putenv... yes
checking for __attribute... yes
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking whether to check for GCC pthread/shared inconsistencies... yes
checking whether -pthread is sufficient with -shared... yes
checking whether the compiler implements namespaces... yes
checking what namespace STL code is in... std
configure: creating ./config.status
config.status: creating Makefile
config.status: creating gflags/src/gflags/gflags.h
config.status: creating gflags/src/gflags/gflags_completions.h
config.status: creating src/config.h
config.status: src/config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands

[root@3fe56e8adf47 open-vcdiff]# make
/bin/sh ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/zlib -I./gflags/src -I./gtest -I./gtest/include -DNDEBUG -DNO_THREADS -DGTEST_HAS_TR1_TUPLE=0 -Wall -Wwrite-strings -Woverloaded-virtual -W -g -O2 -MT headerparser.lo -MD -MP -MF .deps/headerparser.Tpo -c -o headerparser.lo test -f 'src/headerparser.cc' || echo './'src/headerparser.cc
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I./src -I./src -I./src/zlib -I./gflags/src -I./gtest -I./gtest/include -DNDEBUG -DNO_THREADS -DGTEST_HAS_TR1_TUPLE=0 -Wall -Wwrite-strings -Woverloaded-virtual -W -g -O2 -MT headerparser.lo -MD -MP -MF .deps/headerparser.Tpo -c src/headerparser.cc -fPIC -DPIC -o .libs/headerparser.o
src/headerparser.cc: In function 'bool {anonymous}::SumWouldOverflow2(size_t, size_t)':
src/headerparser.cc:27:14: error: 'SIZE_MAX' was not declared in this scope
return a > SIZE_MAX - b;
^
src/headerparser.cc:28:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
make: *** [headerparser.lo] Error 1

Add unit test for vcdiff command-line executable

Original issue 13 created by openvcdiff on 2008-09-18T19:38:39.000Z:

The code in vcdiff_main.cc is currently not covered by unit tests.
Add a new unit test which will invoke the "vcdiff" command-line executable
and exercise most of the code paths in vcdiff_main.cc.

vcdiff --version should print version number

Original issue 3 created by openvcdiff on 2008-09-03T23:28:33.000Z:

What steps will reproduce the problem?

• Build and install open-vcdiff v0.2.
• Execute "vcdiff --version".

What is the expected output? What do you see instead?
"vcdiff --version" should show the version number (0.2).
Instead, it simply prints the command name "vcdiff".

.deb packages install libtool wrapper for vcdiff, not the actual binary

Original issue 27 created by ttjoseph on 2010-07-12T13:57:18.000Z:

What steps will reproduce the problem?

1. Download and untar http://open-vcdiff.googlecode.com/files/open-vcdiff_0.7-1_amd64.deb.tar.gz
2. Install all .deb files
3. Run /usr/bin/vcdiff

What is the expected output? What do you see instead?

Expected: vcdiff should run. Instead:

$ /usr/bin/vcdiff
/usr/bin/vcdiff: error: `/usr/bin/.libs/vcdiff' does not exist
This script is just a wrapper for vcdiff.
See the libtool documentation for more information.

What version of the product are you using? On what operating system?

0.7, as available at URL above, on Debian lenny amd64.

Please provide any additional information below.

$ dpkg -L open-vcdiff0
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/open-vcdiff0
/usr/share/doc/open-vcdiff0/changelog.gz
/usr/share/doc/open-vcdiff0/copyright
/usr/share/doc/open-vcdiff0/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/vcdiff.1.gz
/usr/bin
/usr/bin/vcdiff

Diff file merge capability.

Original issue 20 created by bwang221 on 2009-02-25T23:04:03.000Z:

What steps will reproduce the problem?

1. This is no merger funtionality for the delta files. Would be nice to
   have the function to merge multiple delta files. (like xdelas)
   2.
   3.

What is the expected output? What do you see instead?

What version of the product are you using? On what operating system?

Please provide any additional information below.

Add option to optimize VCDIFF decoder when VCD_TARGET will not be used as source segment

Original issue 9 created by openvcdiff on 2008-09-11T23:24:54.000Z:

The open-vcdiff decoder currently supports the full VCDIFF format as
specified in RFC 3284. The open-vcdiff encoder uses only a subset of the
features available in that format.

One example is using target data as the source segment. RFC 3284 allows
each delta window to specify one source segment to be referenced by COPY
instructions. This source segment can come either from the dictionary
(also known as source file) or from anywhere in the previously decoded
target file. The latter possibility means that the decoder must preserve
the entire contents of the previously decoded target file in memory, or at
least be prepared to load any given portion of the target file into memory
on demand.

The open-vcdiff encoder will never use the previously decoded target file
as the source window. It always uses a source window that starts at offset
0 of the dictionary and includes the entire dictionary contents.

If the decoder could be guaranteed that the encoder would never send a
delta window that used previously decoded target data as the source window,
then it would no longer have to save the previously decoded target file in
memory, resulting in significant memory savings and eliminating one memcpy
operation.

Proposal: add a decoder option that prohibits the use of VCD_TARGET as the
source window type. If this option were enabled and a delta window
containing the VCD_TARGET flag were passed to the decoder, it would fail
with a decoding error.

The Xdelta encoder (another open-source VCDIFF package) does not use
VCD_TARGET either, according to its author.

A restriction has been added to the Appendix of the SDCH protocol
(http://sdch.googlegroups.com/web/Shared_Dictionary_Compression_over_HTTP.pdf)
to prohibit the use of VCD_TARGET for the VCDIFF source segment in the
context of SDCH.

Incorrect test case in vcdiffengine_test.cc

Original issue 32 created by ehrmann on 2011-11-19T06:33:13.000Z:

EngineEncodeSampleTextInterleavedWithTargetMatching is identical to EngineEncodeSampleTextInterleaved. Notably, this is the first line:

Encode(/* interleaved = / true, / target matching = */ false);

It should test target matching.

encoder crashes with NULL source and 0 source length

Original issue 40 created by rashapiro83 on 2013-09-17T15:25:56.000Z:

What steps will reproduce the problem?

1. Attempt to encode with source NULL
   open_vcdiff::VCDiffEncoder enc(NULL, 0)

You get a segfault.

The problem is in BlockHash::AddAllBlocksThroughIndex
Specifically, these lines of code cause a problem:

int last_legal_hash_index = static_cast<int>(source_size() - kBlockSize);
if (end_limit > last_legal_hash_index) {
end_limit = last_legal_hash_index + 1;
}
const char* block_ptr = source_data() + NextIndexToAdd();
const char* const end_ptr = source_data() + end_limit;

At this point, block_ptr has the value 0, end_ptr has the bit pattern for -15.

But since these are pointers, and on linux these are unsigned objects, the next line

while (block_ptr < end_ptr) {

executes which results in a null pointer reference.

Note that if the source length == 0 but the source pointer is something > 16, everything is OK. The simplest fix is probably to just bail out early if the source data size is 0.

This is in version 0.8.3, on linux.

vcdiff command-line executable crashes on startup on FreeBSD platform

Original issue 1 created by openvcdiff on 2008-08-29T22:15:43.000Z:

What steps will reproduce the problem?

1. Download and extract open-vcdiff v0.1 source tarball
   on a FreeBSD 6.0 machine.
2. In top extracted directory, execute: ./configure && make
3. Execute: ./vcdiff

What is the expected output?
vcdiff should produce an error ("Must specify exactly one command option"),
then should output its usage information as with vcdiff --help.

What do you see instead?
A segmentation fault occurs.

format_extension_flags.h missing in debian packages

Original issue 29 created by marstj on 2011-07-12T22:54:15.000Z:

After installing the deb packages in open-vcdiff_0.8.1-1.deb.tar.gz, compilation fails with:

/usr/include/google/vcencoder.h:20:43: fatal error: google/format_extension_flags.h: No such file or directory

And indeed there is no format_extension_flags.h installed anywhere, yet the package dependencies are satisfied. These packages are installed:

libvcdcom0 (0.8.1-1)
libvcdcom-dev (0.8.1-1)
libvcddec0 (0.8.1-1)
libvcddec-dev (0.8.1-1)
libvcdenc0 (0.8.1-1)
libvcdenc-dev (0.8.1-1)

vcdiff_test.sh specifies potentially incorrect shell location, test fails

Original issue 26 created by greenreaper on 2010-03-18T14:54:43.000Z:

What steps will reproduce the problem?

1. Download source package and extract with tar xvf
2. Run ./configure && make && make check

What is the expected output? What do you see instead?

I expect all the tests to pass. Instead, the last test involving
src/vcdiff_test.sh did not pass.

The error code given was:
./src/vcdiff_test.sh: not found
FAIL: src/vcdiff_test.sh

What version of the product are you using? On what operating system?

Version 0.7 on FreeBSD 7.1-RELEASE-p8 in an OS-based jail.

Please provide any additional information below.

On this system, bash is located in /usr/local/bin/bash. The first line of
the .sh file src/vcdiff_test.sh specifies /bin/bash

When unpacked, the source file was not writable. If set writable, and the
shebang line line was removed, or changed to specify /usr/local/bin/bash,
the test worked.

Encoder should buffer writes until flushed

Original issue 10 created by openvcdiff on 2008-09-12T16:46:51.000Z:

The current encoder interface VCDiffStreamingEncoder::EncodeChunk() accepts a contiguous block of memory and creates a single VCDIFF delta window (including a window header) each time it is called. The intended use is for the caller to gather all available input data into a buffer before calling EncodeChunk(), so that the encoded delta window can be sent to streaming output such as a TCP connection or output file.

However, there may be cases in which the currently available input data is split up into more than one contiguous memory block. Imagine that, instead of a single block "ABCDEF", two blocks "ABC" and "DEF" are available for encoding. EncodeChunk("ABC", ...) will be called, then EncodeChunk("DEF", ...), and each of the two blocks will get its own delta window header (about a dozen bytes.) To avoid wasting that space, the caller has to copy the multiple blocks into a single contiguous memory buffer before calling EncodeChunk().

Proposed enhancement: add a new interface (probably a new encoder class in addition to VCDiffStreamingEncoder) that will queue up calls to EncodeChunk() and not output a delta window until a function FinishWindow() is called. The caller will use FinishWindow() to indicate that no more
input data is available at the moment, and that the encoder should generate as much output data as it can. Calling FinishEncoding() will implicitly finish the current window.

The dictionary file cannot be an FIFO

What steps will reproduce the problem?

1. Create a FIFO 'f' using mkfifo t
2. Run vcdiff encode -dictionary t
3. Run echo "source data" > t in a different terminal

What is the expected output? What do you see instead?

Vcdiff should work with a FIFO dictionary, but it says Error finding size of dictionary file 't': Illegal seek. Apparently the dictionary file is handled as a random access file. Xdelta3, on the other hand, allows FIFO sources.

What version of the product are you using? On what operating system?

open-vcdiff 0.8.3, Mac OS X/Homebrew.

Please provide any additional information below.

If the dictionary file is a FIFO (or in general a non-seekable source), the vcdiff command-line tool has the option of loading the entire dictionary file into its memory.

fetch_prereq requires svn

The fetch_prereq.sh and .bat scripts to fetch prerequisite items use svn. This means you cannot build open-vcdiff if you don't have svn installed. Is it possible to make these prerequisites available through git instead (or also)?

open-vcdiff fails to compile on Fedora 9

Original issue 15 created by openvcdiff on 2008-10-04T01:48:38.000Z:

Originally reported as Chromium issues 2662 and 2988:
http://code.google.com/p/chromium/issues/detail?id=2662
http://code.google.com/p/chromium/issues/detail?id=2988
Reporters were craig.schlenter and phajdan.jr.

What steps will reproduce the problem?
Download the source tarball for open-vcdiff v0.2. Build the package using
"./compile" and "make check".

What is the expected output? What do you see instead?
Expected: the package should build successfully and "make check" should
report "All 13 tests passed".

Actual: Compilation fails because the standard header files were not
included before using memcpy() and exit().

See the attached log for the exact error messages.

Failed to handle utf8 characters in .json file

Original issue 36 created by sunzhuoshi on 2012-10-31T15:02:34.000Z:

What steps will reproduce the problem?

1. create file "test1.js", input "你好", and save it without BOM.
2. create file "test2.js", input "你好么", and save it without BOM, too.
3. use command line: vcdiff encode -json -dictionary test1.js -target test2.js -delta delta.js

What is the expected output? What do you see instead?
the excpeted output:
["\u00e4\u00bd\u00a0\u00e5\u00a5\u00bd\u00e5\u0090\u0097",]
what I see:
["\ufffff\ufffff\ufffff\ufffff\ufffff\ufffff\ufffff\ufffff\ufffff",]

What version of the product are you using? On what operating system?
version: 0.8.3
OS: Mac OS X 10.7.4

Please provide any additional information below.
The issue is caused by bad "JSONCodeTableWritter::JSONEscape(...)". and the root cause is using "signed char" as "unsigned char" in line 55. here is the fragment and fix:
void JSONCodeTableWriter::JSONEscape(const char* data,
size_t size,
string* out) {
for (size_t i = 0; i < size; ++i) {
//const char c = data[i]; // wrong type
const unsigned char c = (const unsigned char)data[i]; // use unsigned char instead

Fail to test on MinGW 5.1.4

Original issue 21 created by toliuch on 2009-03-26T01:16:37.000Z:

What steps will reproduce the problem?

1. ./configure; make
2. make check > test.log 2>&1
   3.

MinGW is installed in Windows XP. It's OK to build the open-vcdiff and the
vcdiff.exe can run.

This is the log file content:

make addrcache_test.exe blockhash_test.exe codetable_test.exe
decodetable_test.exe encodetable_test.exe headerparser_test.exe
instruction_map_test.exe output_string_test.exe rolling_hash_test.exe
varint_bigendian_test.exe vcdecoder_test.exe vcdiffengine_test.exe
vcencoder_test.exe src/vcdiff_test.sh
make[1]: Entering directory /c/open-vcdiff-0.5' make[1]:blockhash_test.exe' is up to date.
make[1]: codetable_test.exe' is up to date. make[1]:decodetable_test.exe' is up to date.
make[1]: encodetable_test.exe' is up to date. make[1]:headerparser_test.exe' is up to date.
make[1]: instruction_map_test.exe' is up to date. make[1]:output_string_test.exe' is up to date.
make[1]: rolling_hash_test.exe' is up to date. make[1]:varint_bigendian_test.exe' is up to date.
make[1]: vcdecoder_test.exe' is up to date. make[1]:vcdiffengine_test.exe' is up to date.
make[1]: vcencoder_test.exe' is up to date. make[1]: Nothing to be done forsrc/vcdiff_test.sh'.
make[1]: Leaving directory /c/open-vcdiff-0.5' make check-TESTS make[1]: Entering directory/c/open-vcdiff-0.5'
FAIL: addrcache_test.exe
FAIL: blockhash_test.exe
FAIL: codetable_test.exe
FAIL: decodetable_test.exe
FAIL: encodetable_test.exe
FAIL: headerparser_test.exe
FAIL: instruction_map_test.exe
Running main() from gtest_main.cc

[==========] Running 5 tests from 1 test case.

[----------] Global test environment set-up.

[----------] 5 tests from OutputStringTest

[ RUN ] OutputStringTest.Append

[ OK ] OutputStringTest.Append

[ RUN ] OutputStringTest.Clear

[ OK ] OutputStringTest.Clear

[ RUN ] OutputStringTest.PushBack

[ OK ] OutputStringTest.PushBack

[ RUN ] OutputStringTest.Reserve

[ OK ] OutputStringTest.Reserve

[ RUN ] OutputStringTest.Size

[ OK ] OutputStringTest.Size

[----------] Global test environment tear-down

[==========] 5 tests from 1 test case ran.

[ PASSED ] 5 tests.

PASS: output_string_test.exe
FAIL: rolling_hash_test.exe
FAIL: varint_bigendian_test.exe
FAIL: vcdecoder_test.exe
FAIL: vcdiffengine_test.exe
FAIL: vcencoder_test.exe
/bin/sh: ./src/vcdiff_test.sh: No such file or directory

FAIL: src/vcdiff_test.sh

13 of 14 tests failed

Please report to [email protected]

make[1]: *** [check-TESTS] Error 1
make[1]: Leaving directory `/c/open-vcdiff-0.5'
make: *** [check-am] Error 2

Add "pull-model" interface to decoder with a limited output buffer size

Original issue 12 created by openvcdiff on 2008-09-12T18:40:51.000Z:

The current version of the open-vcdiff decoder API can execute an arbitrary-sized append operation on its OutputString -- at least up to 64MB, which is the limit on target window size. See issue #8, which suggests placing stricter limits on this allocation.

The author of the application that calls the decoder may wish to place a limit on the amount of decoded data that can be returned to it after any one call to DecodeChunk(), and so prevent the decoder from dynamically allocating large amounts of memory.

Proposal: Add a new API to the decoder which places its output into a supplied output buffer, and never exceeds the capacity of that buffer.

The new API will return the number of bytes that were processed from the input chunk (which will be <= the input size), and also the number of bytes that were placed into the output buffer (which will be <= the size of the output buffer.)

The API might look something like:

DecodeChunkToBuffer(const char** source_buffer, size_t*
remaining_source_length, char* destination_buffer, size_t* destination_length)

DecodeChunkToBuffer will read some (not necessarily all) data from the source buffer, writing expansion to the destination_buffer, but not exceeding the supplied destination_length.

When the function returns, the parameters may have been modified. The source_buffer values should be updated to reflect what data has been processed (advancing the pointer, and diminishing the length). If all data was not processed, then the function can be called again with the
remaining, or relocated source buffer (when more destination_buffer space is made available) to process additional data.

The supplied value of *destination_length is the size of the supplied destination_buffer, and the output value is the number of bytes actually written to the supplied destination_buffer.

The interface details may vary from this example, but the critical point is that the caller can specify how much output data to emit. As a result, no more than a single "block" of the decoded target file will ever have to reside in the application's memory space.

It will be possible to fill the output buffer before processing all the input data. If the output buffer becomes full, the number of input bytes processed will be smaller than the input size, and the caller will be in charge of conserving the unprocessed input bytes and passing them along with the next call to DecodeChunk.

The decoder may have to alter the input string to change the last instruction's size and address. Any instruction can be only partially processed; for example, an COPY instruction for 5K bytes with an output buffer of only 4K bytes. In this case, the remaining COPY size will be decremented to 1K and its address will be moved forward by 4K. This may change the size of the instruction in the input stream -- even increasing its size in the (admittedly contrived) example that the original size had an opcode with an implicit size value and the decremented size did not.

Add JSON output support

Original issue 23 created by jamesdebo on 2009-11-19T20:22:37.000Z:

Add an output mode to open-vcdiff which outputs the diff in a JSON format.

Allow linking zlib library

Original issue 34 created by [email protected] on 2012-03-01T22:15:32.000Z:

open-vcdiff contains old copies of zlib's adler32.c, zlib.h and zconf.h. For a project that uses open-vcdiff and links to zlib library (e.g. Chromium), the duplicated code will cause unnecessary binary size increase, or conflicts during linking.

blockhash_test is failing with -DCMAKE_BUILD_TYPE=Release

Configuring build with -DCMAKE_BUILD_TYPE=Release causes blockhash_test to fail.

Comparing 131072 identical values:
BlockHash::BlockCompareWords: 0.0055232 us per operation
BlockHash::BlockContentsMatch: 0.00925851 us per operation
BlockContentsMatch is 67.6293% SLOWER than BlockCompareWords
Comparing 131071 identical values and one mismatch:
BlockHash::BlockCompareWords: 0.00432491 us per operation
BlockHash::BlockContentsMatch: 0.00776339 us per operation
BlockContentsMatch is 79.5039% SLOWER than BlockCompareWords
Comparing blocks that differ at byte 0
BlockHash::BlockCompareWords: 0.00239897 us per operation
BlockHash::BlockContentsMatch: 0.0024426 us per operation
BlockContentsMatch is 1.81872% SLOWER than BlockCompareWords
Comparing blocks that differ at byte 1
BlockHash::BlockCompareWords: 0.00225163 us per operation
BlockHash::BlockContentsMatch: 0.00825119 us per operation
BlockContentsMatch is 266.455% SLOWER than BlockCompareWords
/home/bacek/src/open-vcdiff/src/blockhash_test.cc:386: Failure
Expected: (time_for_block_compare_words * 2.0) > (time_for_block_contents_match), actual: 0.00450325 vs 0.00825119
Comparing blocks that differ at byte 8
BlockHash::BlockCompareWords: 0.00258827 us per operation
BlockHash::BlockContentsMatch: 0.00763488 us per operation
BlockContentsMatch is 194.98% SLOWER than BlockCompareWords
/home/bacek/src/open-vcdiff/src/blockhash_test.cc:386: Failure
Expected: (time_for_block_compare_words * 2.0) > (time_for_block_contents_match), actual: 0.00517654 vs 0.00763488
Comparing blocks that differ at byte 15
BlockHash::BlockCompareWords: 0.00231504 us per operation
BlockHash::BlockContentsMatch: 0.00583315 us per operation
BlockContentsMatch is 151.967% SLOWER than BlockCompareWords
/home/bacek/src/open-vcdiff/src/blockhash_test.cc:386: Failure
Expected: (time_for_block_compare_words * 2.0) > (time_for_block_contents_match), actual: 0.00463009 vs 0.00583315
[  FAILED  ] BlockHashTest.BlockContentsMatchIsAsFastAsBlockCompareWords (259 ms)

Please pass Debian's licensecheck

Original issue 31 created by [email protected] on 2011-09-26T22:22:18.000Z:

$ licensecheck -r sdch/open-vcdiff/ | grep UNKNOWN
sdch/open-vcdiff/autogen.sh: No copyright UNKNOWN
sdch/open-vcdiff/src/codetablewriter_interface.h: UNKNOWN
sdch/open-vcdiff/packages/deb.sh: No copyright UNKNOWN
sdch/open-vcdiff/packages/rpm.sh: No copyright UNKNOWN

vcdiff incorrect binary I/O on Windows

Original issue 7 created by [email protected] on 2008-09-11T11:02:32.000Z:

On Windows, stdin and stdout are open in text mode, not binary mode, by
default. This means, for example, that even data that were conceived as
and seemingly being output as binary to stdout, are in fact processed by
the runtime to perform CR LF translation. If it encounters unpaired LF, it
outputs CR LF instead etc.

What steps will reproduce the problem?
vcdiff encode --dictionary test.dict < in.txt > diff.diff
vcdiff decode --dictionary test.dict < diff.diff > out.txt

Please find in.txt attached. (in.txt can be any binary file that contains
a 0x0A without 0x0D. A UNIX text file, transferred to Windows in binary
mode will be OK too.)

What is the expected output? What do you see instead?
out.txt has different content from in.txt (even different size), I expect
them to be identical. (fc /b in.txt out.txt)

BTW when compared in text mode (fc in.txt out.txt), they are "identical".

What version of the product are you using? On what operating system?
Microsoft Visual Studio 2008 Version 9.0.21022.8 RTM
OS Name Microsoft Windows XP Professional
Version 5.1.2600 Service Pack 2 Build 2600

Please provide any additional information below.

Note, that this
vcdiff encode --dictionary test.dict --target in.txt --delta diff.diff
vcdiff decode --dictionary test.dict --delta diff.diff --target out.txt
works OK.

I suggest something of the sort to be inserted in vcdiff berore any I/O
takes place:

ifdef WIN32

setmode(fileno(stdout), O_BINARY);
setmode(fileno(stdin), O_BINARY);

endif

vcdiff crashes with zero-size dictionary

Original issue 6 created by [email protected] on 2008-09-11T05:57:48.000Z:

What steps will reproduce the problem?

1. Create empty file empty.dict
2. Create a non-empty file sample.txt
3. vcdiff encode --dictionary empty.dict < sample.txt > out.delta

The crash may be platform/STL/configuration-dependent, see the comment
below.

What is the expected output? What do you see instead?
I expect it to work as if no dictionary was specified, as a compressor,
looking for string re-occurences within a single input file; or, if that
is not desired behaivior, printing an error message.

Instead it crashes.

Platform:
Microsoft Visual Studio 2008 Version 9.0.21022.8 RTM
OS Name Microsoft Windows XP Professional
Version 5.1.2600 Service Pack 2 Build 2600

The crash takes place within STL, when empty dictionary_ is indexed by [0]
here:
dictionary_.resize(dictionary_size);
if (fread(&dictionary_[0], 1, dictionary_size, dictionary_file)
!= dictionary_size) {

In Debug configuration, STL debug assertion _DEBUG_ERROR("vector subscript
out of range"); is triggered.

Certain STL implementation in some configurations may allow taking an
address of a first element of an empty vector, even if this address is
technically NULL. If this address is never dereferened (which is a natural
assumption since the size is zero), this never leads to a crash; but it is
not a safe practice.

STLPort has a similar check is _STLP_DEBUG mode.

Fetching prerequisites no longer works

Running ./fetch_prereq.sh emits the following errors:

svn: E160013: Unable to connect to a repository at URL 'http://google-gflags.googlecode.com/svn/trunk'
svn: E160013: 'http://google-gflags.googlecode.com/svn/trunk' path not found
svn: E160013: Unable to connect to a repository at URL 'http://googletest.googlecode.com/svn/trunk'
svn: E160013: 'http://googletest.googlecode.com/svn/trunk' path not found

The above repositories have been relocated to:

https://github.com/gflags/gflags.git
https://github.com/google/googletest.git

The script fetch_prereq.sh and .cmd should be changed to:

git clone https://github.com/gflags/gflags.git gflags
git clone https://github.com/google/googletest.git gtest

Furthermore, autoconf is also broken, because it references invalid paths for the gflags and googletest dependencies (which now both use CMake).

Related issue: #52.

No default support for Visual Studio 2010 or 2012.

Original issue 39 created by jakerstrom on 2013-01-15T17:29:30.000Z:

What steps will reproduce the problem?

1. Launch the existing .sln file in VS2010 or VS2012
2. Rebuild solution

What is the expected output? What do you see instead?
Rebuild All Succeded

What version of the product are you using? On what operating system?
Latest Trunk, Windows 8!

Please provide any additional information below.
There was 1225 errors on the build. Did not look too much into it since I'm a C# developer, not so much a C developer and wanted to use the resulting library in a C# project.

Building RPM package fails on Fedora 9: Installed (but unpackaged) file vcdiff.1.gz

Original issue 18 created by openvcdiff on 2009-01-12T20:02:06.000Z:

What steps will reproduce the problem?

1. Unpack the source tarball into a temporary directory on a machine
   running Fedora release 9 (Sulphur).
2. In that directory, execute "./configure && make dist && make rpm".

What is the expected output? What do you see instead?

The expected output should end with:

• exit 0
  The rpm package file(s) are located in [directory name]

Instead, the following errors appear during "make rpm":
/usr/bin/install -c -m 644 './man/vcdiff.1'
'/var/tmp/open-vcdiff-root/usr/man/man1/vcdiff.1'
[...]
Processing files: open-vcdiff-0.4-1
error: File not found: /var/tmp/open-vcdiff-root/usr/man/man1/vcdiff.1
[...]
Checking for unpackaged file(s): /usr/lib/rpm/check-files
/var/tmp/open-vcdiff-root
error: Installed (but unpackaged) file(s) found:
/usr/man/man1/vcdiff.1.gz
RPM build errors:
File not found: /var/tmp/open-vcdiff-root/usr/man/man1/vcdiff.1
Installed (but unpackaged) file(s) found:
/usr/man/man1/vcdiff.1.gz

Provide a log handler to log errors.

Original issue 33 created by [email protected] on 2011-12-09T03:58:01.000Z:

#include <iostream> introduces static initializers, see http://crbug.com/94794

open-vcdiff logs through cerr. If open-vcdiff must log, it should use fprintf(stderr, ...) – or, probably better, provide a log handler that embedders could override to do what they please.

Debian package files do not install vcdiff executable properly

Original issue 24 created by openvcdiff on 2009-12-18T21:49:12.000Z:

What steps will reproduce the problem?

1. Download the Debian package files for open-vcdiff version 0.7:
   curl -O http://open-vcdiff.googlecode.com/files/open-vcdiff_0.7-
   1_amd64.deb.tar.gz
2. Unpack the .deb files:
   tar zxvf open-vcdiff_0.7-1_amd64.deb.tar.gz
3. Install the .deb files:
   sudo dpkg -i libvcd_.deb open-vcdiff_.deb
4. Run the vcdiff executable:
   vcdiff --help

What is the expected output? What do you see instead?

The expected output is:
$ vcdiff --help
vcdiff: {encode | delta | decode | patch }[ <options> ]
encode or delta: create delta file from dictionary and target file
decode or patch: reconstruct target file from dictionary and delta file
[...]

Instead, the following error appears:
$ vcdiff --help
/usr/bin/vcdiff: error: `/usr/bin/.libs/vcdiff' does not exist
This script is just a wrapper for vcdiff.
See the libtool documentation for more information.

To clean up after trying this test case, remove the installed packages:
sudo dpkg -P libvcdcom-dev libvcdcom0 libvcddec-dev libvcddec0 libvcdenc-
dev libvcdenc0 open-vcdiff0

This problem did not exist in open-vcdiff version 0.5, but is present in
versions 0.6 and 0.7.

add ability to stream source and not requre the entire source in memory

Original issue 17 created by lordjebus on 2008-10-21T22:48:44.000Z:

Currently the entire source is required to be in memory meaning it can not
handle large files or stream from a network.

Makefile.am doesn't list all library dependencies causing linking errors with gold

Original issue 22 created by jamesdebo on 2009-11-19T20:20:44.000Z:

What steps will reproduce the problem?

1. Use the GNU gold linker.

$ ld -v
GNU gold (GNU Binutils 2.19.1) 1.7

1. Build and run distcheck

$ ./configure ; make distcheck
...
/usr/bin/ld: decodetable_test.o: in function open_vcdiff::(anonymous
namespace)::DecodeTableTest_UnGetAddCopy_Test::TestBody():../src/logging.h:
54: undefined reference to 'open_vcdiff::g_fatal_error_occurred'

What is the expected output? What do you see instead?
distcheck should pass.

What version of the product are you using? On what operating system?
SVN head.

ERROR: Length of target window (100001916) exceeds limit of 67108864 bytes

Original issue 19 created by ferragina on 2009-01-29T09:02:14.000Z:

I'm using VCDIFF without a dictionary and thus to copy only within the
target file. So a sort of GZIP with a large window. So I'm trying to
extend the size of the window within the target file from where a "copy"
can be found.

My files can be larger than 67Mb, and I wish to copy from large distances.

I played with the options max_target_window_size, max_target_file_size,
and buffersize but I get errors, like the one in the subject, when I act
on files larger than 50Mb.

Can you please provide me an help?

Thanks.

Sub-optimal usage of string.reserve in decoder

Original issue 28 created by martyroxen on 2010-07-29T16:10:13.000Z:

I have identified a bottleneck in the streaming decoder implementation, more precisely where decoded_target->reserve() is called unconditionally in VCDiffDeltaFileWindow::ReadHeader.

When string.reserve is called, the standard C++ library typically over-allocates space in its internal buffer - when a string is expanded incrementally (through string.append for example) one would like to avoid reallocating a larger C-string and memcpy:ing the contents for each operation. A typical approach is to double the internal buffer size each time it overflows. This algorithm also applies to string.reserve, at least in recent GNU libstdc++.

Starting with the GNU Standard C++ Library v3, string.reserve honors shrink requests, which means that a smaller internal string will be reallocated if string.reserve is called with an argument smaller than the string's current capacity. This means that the cleverly over-allocated string will be shrunk again at the next call to VCDiffDeltaFileWindow::ReadHeader. Then, at the next call, it will be expanded and over-allocated, and then shrunk again and so on. This proves to be expensive when the number of chunks is relatively large.

In my benchmark program (attached), performance is increased by more than a factor 10 simply by avoiding the call to string.reserve if it already had the sufficient space:

---

marty@java:~$ time ./test

real 0m38.713s
user 0m17.581s
sys 0m21.133s
marty@java:~$ time LD_PRELOAD=/home/marty/open-vcdiff-read-only/.libs/libvcddec.so ./test

real 0m2.531s
user 0m1.872s

sys 0m0.656s

I have also attached my proposed patch.

Return error codes from calls to decoder/encoder

Original issue 11 created by openvcdiff on 2008-09-12T16:55:16.000Z:

Currently the encoder and decoder methods return only a Boolean value to indicate success or failure. Error log messages, which indicate the reason for each failure, are sent to standard error output. Diagnosis of failures depends on having access to this error output.

Proposal: Expand the interface to return one of a number of error codes, rather than just false in the error case.

Additional proposal: Most of the error log messages include parameters for interesting values that help to explain the error. If each error code is encapsulated with its parameters, the error messages can be placed into a message catalog, which will allow internationalization of log messages and will reduce the sizes of the encoder and decoder libraries.

Issue during cross compilation of open-vcdiff for powerpc-linux

Original issue 38 created by suyash.kools on 2013-01-03T05:33:47.000Z:

What steps will reproduce the problem?

1. Configure open-vcdiff to be compiled for powerpc by using --host=powerpc-linux option with ./configure in the spec file.
2. start the compilation by going into the root directory of your linux platform powerpc project and use the commands:

make -C build open-vcdiff.addpkg
make -C build open-vcdiff

1. The compilation starts but throws a compilation error.

What is the expected output? What do you see instead?
The compilation is supposed to complete and produce a cross compiled RPM for powerpc. Instead we see error like this:

libvcdcom.so.0, needed by ./.libs/libvcddec.so, not found (try using -rpath or -rpath-link)
./.libs/libvcddec.so: undefined reference to `open_vcdiff::g_fatal_error_occurred'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VarintBE<long long>::Parse(char const*, char const**)'
./.libs/libvcdenc.so: undefined reference to `open_vcdiff::VarintBE<int>::Length(int)'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffCodeTableData::Validate(unsigned char) const'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffAddressCache::Init()'
./.libs/libvcdenc.so: undefined reference to `open_vcdiff::VarintBE<int>::AppendToOutputString(int, open_vcdiff::OutputStringInterface*)'
./.libs/libvcdenc.so: undefined reference to `open_vcdiff::VarintBE<long long>::Length(long long)'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VarintBE<int>::Parse(char const*, char const**)'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffAddressCache::DecodeAddress(int, unsigned char, char const**, char const*)'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffAddressCache::VCDiffAddressCache()'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffCodeTableData::kDefaultCodeTableData'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffInstructionName(open_vcdiff::VCDiffInstructionType)'
./.libs/libvcddec.so: undefined reference to `open_vcdiff::VCDiffAddressCache::VCDiffAddressCache(int, int)'
./.libs/libvcdenc.so: undefined reference to `open_vcdiff::VCDiffAddressCache::EncodeAddress(int, int, int*)'
./.libs/libvcddec.so: undefined reference to `adler32'
./.libs/libvcdenc.so: undefined reference to `open_vcdiff::VarintBE<long long>::AppendToOutputString(long long, open_vcdiff::OutputStringInterface*)'
./.libs/libvcdenc.so: undefined reference to `open_vcdiff::VarintBE<int>::AppendToString(int, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
collect2: ld returned 1 exit status
Makefile:836: *** [vcdiff] Error 1

What version of the product are you using? On what operating system?

I am using open-vcdiff-0.8.3 compiling for WindRiver linux for powerpc architecture.

Please provide any additional information below.
I am guessing that it can be result of improperly defined macros in the programs where the above function calls cannot be made because maybe they are not defined for powerpc. I will have to look up the code for this. If anyone has faced a similar issue Please help me with this. I am stuck without a way forward.

HashedDictionary may free memory twice if implicitly copied

Original issue 14 created by openvcdiff on 2008-09-18T19:52:11.000Z:

What steps will reproduce the problem?

1. Build and install open-vcdiff v0.2 as described in the README file.
2. Create a file double_free_hd.cc containing the following code:

include <google/vcencoder.h>

int main() {
open_vcdiff::HashedDictionary hd1("", 0);
hd1.Init();
open_vcdiff::HashedDictionary hd2 = hd1;
return 0;
}

1. Compile the source file (for example, "g++ double_free_hd.cc -lvcdenc -o
   double_free_hd")
2. Run the resulting executable (for example,
   "LD_LIBRARY_PATH=/usr/local/lib ./double_free_hd")

What is the expected output? What do you see instead?

Expected behavior:
the executable should finish without producing any output.

Actual behavior:
Memory is freed twice and an error message appears, such as:
*** glibc detected *** double free or corruption (fasttop): 0x0804a018 ***
Aborted

Proposed solution: Create a private assignment operator and a private
copy constructor for HashedDictionary so that client code cannot copy
HashedDictionary objects using the versions of these functions that are
implicitly generated by the compiler.

Unit tests pass with errors

Original issue 5 created by [email protected] on 2008-09-08T11:01:04.000Z:

The following unit tests report errors to stderr:
vcencoder_test
addrcache_test
vcdecoder_test
codetable_test

However, all these tests output "[ PASSED ]" to stdout

The project was built from source files with VS2008, under WinXP SP2.

To reproduce the problem:

1. Open open-vcdiff.sln with VS2008
2. Have the project converted by VS
3. Re-build the solution
4. Run the unit tests (found in Release sub-dir) one by one
   (they also are run automatically as a part od the build process)

I see "ERROR" lines of output. See the attached stderr files.

Platform:
Microsoft Visual Studio 2008 Version 9.0.21022.8 RTM
OS Name Microsoft Windows XP Professional
Version 5.1.2600 Service Pack 2 Build 2600

vcdecoder_test error output mentions "VCDIFF delta file"; there's no such
file provided nor generated by another test.