ah906632 / finddups Goto Github PK

This is a command line Java app to find duplicate files on your computer.

You can either build the application from source, or download a pre-compiled version.
To build from source, you need Git, a Java JDK, and Maven:

git clone https://github.com/caluml/finddups
mvn assembly:single install
java -jar ~/.m2/repository/finddups/finddups/0.0.1-SNAPSHOT/finddups-0.0.1-SNAPSHOT-jar-with-dependencies.jar 1000000 /home /tmp

To just run the supplied version (might be a little out of date):

Download https://github.com/caluml/finddups/raw/master/finddups.jar
java -jar finddups.jar 1000000 /home /tmp


First, it recurses the given directory, building a list of files.
Then, it looks for files which have identically sized files.
It then reads those identically sized files, comparing them byte for byte until it finds a byte that doesn't match, or it reaches the end of the file.

I chose not to use any kind of hashing or CRC system, as that would require reading the entire file, which would be less efficient if a byte near the beginning was different.

The app outputs its errors to stderr, and its results to stdout as it goes.

Any suggestions/comments to my github username at gmail.com

TODO: Time generating two MD5s of two identical files and compare them, with comparing those two files byte by byte.
CAVEAT: I have only tested this on Linux, with Java 6.