v2020 version is not 30-40% better than v2016 version? about fastcdc-rs

[nlfiedler]

Which CPU architecture were the benchmarks running on? That probably makes a difference. The paper has some discussion about the number of instructions and such, so they may have been testing with systems that yielded the results they claimed. I'll also run some tests again. As I recall, when I tested with a 7gb file it was around 20% faster (on an ARM64 system).

[nlfiedler]

Did more testing and yep, the results are about 20% faster. The results may depend on the size of the input file and the size of the chunks. I was using larger files and larger chunks. I've updated the comment in the code to give some warning.

[bduffany]

Ah. I was not using mmap in my benchmark - the fact that my benchmark is calling read() likely dilutes things a bit, since I'm now seeing a 10-12% speedup for a 2.5 GB file if I switch to mmap. I see from your comment you tested with an M1 processor, I guess maybe that might be accounting for the remaining difference between our results (or maybe I need a 7GB file to test with). I'm on Linux, x86_64 (Intel i7-9700K)