A high-performance duplicate file finder. Meant for use with btrfs/xfs + duperemove utility.
- gradle assemble
- java -jar -XX:+UseShenandoahGC jensame.jar $fdupesOutput $pathsToRecurse
- cat $fdupesOutput | sudo duperemove -d -b32768 --fdupes -v
- Example output:
- Found 10049077 files, hashed 802450 files, totalling 388119MB, and identified 691490 duplicates in 656920ms at 590MBps
- User defined thread count
- Default thread count determined by storage medium
- Alternate hash algorithms
- Zero-Allocation Hashing (Apache-2.0), https://github.com/OpenHFT/Zero-Allocation-Hashing