bisqwit / tinydeflate Goto Github PK
View Code? Open in Web Editor NEWA deflate/gzip decompressor that requires minimal amount of memory to work
A deflate/gzip decompressor that requires minimal amount of memory to work
This is a response to some false claims made which use the respected word "Engineering" in it.
Hey
@bisqwit just wanted to say that Engineering has nothing to do with sitting behind a keyboard and writing code. There is no such thing as "reverse" "engineering". Engineering is a real subject. Engineering means designing and building real mechanical systems, like vehicular engines and turbojets. Software devs are claiming the most ridiculous of false claims these days.
Software dev is easy and for kids, and has absolutely nothing to do with a real and serious complex subject like engineering.
Mechanical Engineering is the only Engineering.
bye bye
As title suggests building for C++20 where std::result_of has been removed after being deprecated in C++17, fails. I've implemented a generic solution, and what's actually being used simply copied from headers , thus not using a pull request. If you would consider adding proper support that would be ideal.
Putting this snipped at the top of the gunzip.hh then simply removing std:: from instances of result_of_t.
#if __cplusplus > 202000UL
// Normal solution for result_of , not what's being used
template <typename F, typename... Args>
struct result_of<F(Args...)> : std::invoke_result<F, Args...> {};
template <class _Fty>
struct result_of { // explain usage
static_assert(_Always_false<_Fty>, "result_of<CallableType> is invalid; use "
"result_of<CallableType(zero or more argument types)> instead.");
};
template <class _Ty>
using result_of_t = typename result_of<_Ty>::type;
#endif
When compiling on Windows with the Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29914 for x64 compiler and enabling warnings I get the warnings:
...\TinyDeflate\gunzip.hh(983): warning C4800: Implicit conversion from 'uint_least64_t' to bool. Possible information loss
...\TinyDeflate\gunzip.hh(983): note: consider using explicit cast or comparison to 0 to avoid this warning
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29910\include\stdint.h(34): note: see declaration of 'uint_least64_t'
Changing the lines 983 and 984 from:
if(header&0x0800) for(;;) { GetBits(8, bool q); if(!q) break; } // NAME: Skip filename if FNAME was present
if(header&0x1000) for(;;) { GetBits(8, bool q); if(!q) break; } // COMMENT: Skip comment if FCOMMENT was present
to
if(header&0x0800) for(;;) { GetBits(8, std::uint_least64_t q); if(!q) break; } // NAME: Skip filename if FNAME was present
if(header&0x1000) for(;;) { GetBits(8, std::uint_least64_t q); if(!q) break; } // COMMENT: Skip comment if FCOMMENT was present
solves the problem, though it might be even better to change them in:
if(header&0x0800) for(;;) { GetBits(8, std::uint_least64_t q); if(q == 0) break; } // NAME: Skip filename if FNAME was present
if(header&0x1000) for(;;) { GetBits(8, std::uint_least64_t q); if(q == 0) break; } // COMMENT: Skip comment if FCOMMENT was present
I'm doing some tests with TinyDeflate, but I noticed a divergence in the number of bytes read.
In both cases I have the same gzip file loaded into a std::vector<uint8_t>
(this approach is just for a proof of concept).
The code below returns 80275 bytes consumed:
std::vector<uint8_t> gzip_content{ /*...*/ }; // 80283 bytes
std::vector<uint8_t> bin_content;
auto result = Deflate(
[&]() {
static size_t i = 0;
if (i < gzip_content.size())
return (int)gzip_content[i++];
return EOF;
},
[&](uint8_t data) { bin_content.push_back(data); },
DeflateTrackBothSize{});
// result.first = 0
// result.second.first = 80275 (8 bytes less: chesksum + trailer ?)
// result.second.second = 221863
// bin_content is OK
However, the code below returns 81850 bytes consumed:
std::vector<uint8_t> gzip_content{ /*...*/ }; // 80283 bytes
size_t n = /* ... */;
uint8_t *bin_content = new uint8_t[n];
auto result = Deflate(
(uint8_t *)gzip_content.data(),
(uint8_t *)gzip_content.data() + gzip_content.size(),
(uint8_t *)bin_content,
(uint8_t *)bin_content + n,
DeflateTrackBothSize{});
// result.first = 0
// result.second.first = 81850 (1567 bytes more)
// result.second.second = 221863
// bin_content is OK
Is this because of sentence (15)?
"This method is backtrackable, meaning that some bytes in the input may be read twice."
extern const char compressed_data[];
extern unsigned compressed_data_length;
extern unsigned char outbuffer[131072];
int result = Deflate(compressed_data+0, compressed_data_length, outbuffer, outbuffer + sizeof(outbuffer));
if(result != 0) std::fprintf(stderr, "Error\n");
This is the code in question.
The result:
In file included from ../../project/test/test.cpp:13:
../../project/test/TinyDeflate/gunzip.hh: In instantiation of ‘auto gunzip_ns::DeflateOutputDispatch(BtFun&&, InFun&&, T1&&, T2&&) [with unsigned char code = 1; BtFun = DeflateInputDispatch<0, dummy, const char*, unsigned int&, unsigned char (&)[131072], unsigned char*>(dummy&&, const char*&&, unsigned int&, unsigned char (&)[131072], unsigned char*&&)::<lambda(bool)>&; InFun = DeflateInputDispatch<0, dummy, const char*, unsigned int&, unsigned char (&)[131072], unsigned char*>(dummy&&, const char*&&, unsigned int&, unsigned char (&)[131072], unsigned char*&&)::<lambda()>&; T1 = unsigned char (&)[131072]; T2 = unsigned char*]’:
../../project/test/TinyDeflate/gunzip.hh:1386:67: required from ‘auto gunzip_ns::DeflateInputDispatch(BtFun&&, T1&&, T2&&, T&& ...) [with unsigned char code = 0; BtFun = dummy; T1 = const char*; T2 = unsigned int&; T = {unsigned char (&)[131072], unsigned char*}]’
../../project/test/TinyDeflate/gunzip.hh:1427:46: required from ‘auto Deflate(T&& ...) [with T = {const char*, unsigned int&, unsigned char (&)[131072], unsigned char*}]’
../../project/test/test.cpp:22:23: required from here
../../project/test/TinyDeflate/gunzip.hh:1311:31: error: static assertion failed: Deflate: Unknown output parameter type
1311 | static_assert(code==0xFF, "Deflate: Unknown output parameter type");
| ~~~~^~~~~~
../../project/test/TinyDeflate/gunzip.hh:1311:31: note: the comparison reduces to ‘(1 == 255)’
I'm using GCC 12 with -std=c++20
if that makes a difference.
As a not-so-experienced C programmer, I am not sure what to do with the .hh file. Could you kindly give a complete working solution?
Could you provide a demo how to use your code?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.