mattsta / crcspeed

This make CRC be fast. Included implementations: CRC-64-Jones and CRC-16-CCITT

mattsta

hey wow this is a special box cool huh

i'm matt and mattsta and https://matt.sh and other fun facts

i'm at [email protected] or [email protected] or [email protected] or [email protected] or even [email protected]

crcspeed's Issues

CRC16 test reports ERROR, likely because 32 bits are printed/used as CRC16 checksum

The accelerated CRC16 implementation seems to be misbehaving. I get CRC errors regardless of which files are used, for the below examples I've used the 60k test file http://ftp.acc.umu.se/mirror/media/StarWreck-InThePirkinning/star_wreck_in_the_pirkinning_subtitles_eng.srt (Creative Commons licensed).

However, I think that the errors are benign as CRC16 is 2 bytes, and all mismatches I've seen seems to be in the additional 2 bytes that are emitted for some reason. Coding cleanup gone wrong?

I don't think it's a compiler/CPU-specific issue, as I'm seeing this on various machines. The examples below are a git pull of this repo and make clean; make to have the crc*speed-test binaries compiled on each platform.

RHEL 6.7, Intel E5-2643:

crc16 (no table)
CRC = 0000000022dda97d
0.002447 seconds at 23.98 MB/s (131.23 CPU cycles per byte)

crc16 (lookup table)
CRC = 000000002ddaa97d
0.000250 seconds at 234.70 MB/s (13.38 CPU cycles per byte)
ERROR: CRC results don't match! (0000000022dda97d vs. 000000002ddaa97d)

crc16speed
CRC = 000000002ddaa97d
0.000052 seconds at 1128.34 MB/s (2.80 CPU cycles per byte)
ERROR: CRC results don't match! (0000000022dda97d vs. 000000002ddaa97d)

Ubuntu 14.04.5 LTS, Intel E5-2630:

crc16 (no table)
CRC = 0000000022dda97d
0.004516 seconds at 12.99 MB/s (168.45 CPU cycles per byte)

crc16 (lookup table)
CRC = 000000002ddaa97d
0.000514 seconds at 114.15 MB/s (19.17 CPU cycles per byte)
ERROR: CRC results don't match! (0000000022dda97d vs. 000000002ddaa97d)

crc16speed
CRC = 000000002ddaa97d
0.000103 seconds at 569.65 MB/s (3.85 CPU cycles per byte)
ERROR: CRC results don't match! (0000000022dda97d vs. 000000002ddaa97d)

Ubuntu 16.04.2 LTS, Intel E5-2620 v4:

crc16 (no table)
CRC = 0000000022dda97d
0.004750 seconds at 12.35 MB/s (162.13 CPU cycles per byte)

crc16 (lookup table)
CRC = 000000002ddaa97d
0.000513 seconds at 114.37 MB/s (17.50 CPU cycles per byte)
ERROR: CRC results don't match! (0000000022dda97d vs. 000000002ddaa97d)

crc16speed
CRC = 000000002ddaa97d
0.000071 seconds at 826.39 MB/s (2.42 CPU cycles per byte)
ERROR: CRC results don't match! (0000000022dda97d vs. 000000002ddaa97d)

crc16speed.c

static uint16_t crc16_table_little[8][256] = {{0}};
static uint16_t crc16_table_big[8][256] = {{0}};

Я бы хранил уже посчитанные

static constexpr uint16_t crc16_table_little[][256]{
    {...},
};
static constexpr uint16_t crc16_table_big[][256]{
    {...},
};

License for the code

Hi @mattsta!

I was having a look at your code, as I would be interested in using it to calculate the CRC64 to generate RDB files but I have noticed that the license on the files, i just 2 of them, refers to Redis but this repo is not under the Redis Labs organization or under the Redis project.

Specifically both crc16speed.c and crc64speed.c refer Redis in the license meanwhile crcspeed.c has a different license and the header files have no license.

Was it because of a copy & paste or is the code owner by Redis Labs?

Disclaimer:
I am implementing in my Key-Value store, it's called cachegrand ( https://github.com/danielealbano/cachegrand ) and as I am to make it Redis compatible I am adding the necessary bits and pieces to load and save RDB dumps.
Of course to have full compatibility I would like to have the CRC64 generation and check but after quite a while attempting to validate the RDB generated by my code with redis-check-rdb I discovered I was using a different polynomial (I am not an expert with CRC64) and after multiple attempts, even with pycrc, I wasn't able to make it work.

If it's a copy & paste I can make a PR to fix it, if not, thanks for the time spent reading my message :)

The CRC-16 of string name of Li is not the same, need to be verified.

I have check more implements of CRC-CCITT
POLYNOMIAL:0x1021,
INITIAL:0x0000,
FINAL_XOR_VALUE:0x0000,
Little endian.

For char test="123456789", the crc result is # 0x31c3
For char li[] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed "
"do eiusmod tempor incididunt ut labore et dolore magna "
"aliqua. Ut enim ad minim veniam, quis nostrud exercitation "
"ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis "
"aute irure dolor in reprehenderit in voluptate velit esse "
"cillum dolore eu fugiat nulla pariatur. Excepteur sint "
"occaecat cupidatat non proident, sunt in culpa qui officia "
"deserunt mollit anim id est laborum.", the crc result is # 0x224f

you can check them:
http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
http://www.barrgroup.com/Embedded-Systems/How-To/CRC-Calculation-C-Code

So i think your result for string Li of 4b20 is not correct.