kassoulet / ultrafastduplicatefilesfinder Goto Github PK

Ultra Fast Duplicate Files Finder
=================================
  by Gautier Portet <kassoulet gmail com>


Takes a list of file from stdin.
And print the duplicate ones.


example use:

  find ~/ -size +10M | ./UltraFastDuplicateFilesFinder.py

to find duplicates in your home folder, all files more than 10MB.

UltraFastDuplicateFilesFinder compares only the very beginning of the files.
Its sufficient for most uses, but use with caution.

But this way is quite useful to detect duplicates within corrupted media files...


this is public domain.



------------------------------------------------------------------------------
example run, took less than a second to answer...


gautier@quad:~/code/tmp$ find /home -size +10M | ./duplicate.py
      size   filename
  12467906   /home/gautier/Photos/pict4614.mov
  12467906   /home/gautier/Photos/Videos/PICT4614.MOV

  13068570   /home/gautier/Photos/pict4588.mov
  13068570   /home/gautier/Photos/Videos/PICT4588.MOV

[...]

  20865498   /home/gautier/Photos/pict4695.mov
  20865498   /home/gautier/Photos/Videos/PICT4695.MOV

  28270824   /home/gautier/tmp/tsunami 1 œ ǒǑ.flac
  28270824   /home/gautier/tmp/tsunami-1.flac

136 files checked (22.75 GiB), 8 duplicates (153.45 MiB).