johanneswiesner / demetrius

A repository for finding and copying files while preserving their folder structure

Demetrius

A python library for finding and copying files within their parent directories.

NOTE

Feel free to send pull requests by having a look at the FIXMEs and TODOs inside demetrius.py

Motivation

Your Grandpa gives you his PC and asks you if you could make a backup of his photos and videos. There's just one problem: All his mediafiles are stored in different folders and in different locations on his drive! In case you don't want to go manually through each folder, this is a case for demetrius. Just pass in the source and destination directory and demetrius will crawl through all the folders, find files whose extensions match the extensions you are looking for and copies the files in the destination directory.

Demetrius will copy files within their parent directory to ensure that you don't end up with one destination folder where all files are stored in. So for example, all files within a folder holiday in the source directory will also be in a holiday folder in the destination directory. Demetrius also takes care of duplicate folder names. For example, if there are several holiday folders on Grandpa's PC (which is not unlikely, because Grandpa forgot that it might have been a good idea to specify a unique name for each holiday he spent with the family...) demetrius will conserve the original structure and creates holiday, holiday_2, etc..

Finally, Demetrius also respect case insensitive OS like Windows where you can't create a folder holiday and Holiday in the same directory (e.g. by adding indices ('holiday (1)','Holiday (2)')).

Usage

Demetrius can be used via the console. For example: python demetrius.py -src ./foo -dst ./bar will copy all found files in ./foo to ./bar withing their parent directories. For that, the script uses all file suffixes found in suffixes.json for the search. You can also filter for specifc file suffixes via the -sfx flag (e.g. -sfx png jpg) or even for broad file categories using the -cat flag (e.g. -cat video for only searching for video files). Use the -e flag if you want to ignore certain directories and their children directories (e.g. -e Windows "Program Files"). Use -v if you want demetrius to show progress information.

What Demetrius can't do

Demetrius is dumb. If you have a folder in the source directory that is called foobar which contains a file named 123.png (which is a photo of dickbutt) it will copy that with that folder to the destination directory. Accordingly, you have to weight demetrius's dumbness against how much time you want to spend with manually clicking through Grandpas PC. U

Links

I can strongly recommend to run AntiDupl after Demetrius was run to identify file duplicates.

Related projects

Total Commander had a plugin called Copy Tree that did exactly what demetrius does. The project site is down, so here's a link from the Wayback Machine

Preview

Before Demetrius	After Demetrius
src/ ┣ grandpa.doe/ ┃ ┗ Holiday/ ┃ ┗ lena.png ┣ holiday/ ┃ ┗ lena.png ┣ jane.doe/ ┃ ┗ holiday/ ┃ ┗ lena.png ┣ john.doe/ ┃ ┗ Holiday/ ┃ ┗ lena.png ┗ .lena.png	dst/ ┣ Holiday_1 (1)/ ┃ ┗ lena.png ┣ Holiday_2 (1)/ ┃ ┗ lena.png ┣ holiday_1 (2)/ ┃ ┗ lena.png ┣ holiday_2 (2)/ ┃ ┗ lena.png ┗ src/ ┗ .lena.png

demetrius's Issues

Allow user to decide to save information for all found files.

This list should either be placed in the same directory as the destination directories or separately in each of destination directory (showing only the source files for this directory

Separate verbosity from function logic in _copy_files

Would be nice if _copy_files wouldn't have anything to do with verbosity just like it's currently done with _find_files. Asked question here:

https://stackoverflow.com/questions/77730834/separate-inner-logic-of-function-from-displaying-user-information-using-tqdm

Enable auto-complete for source and destination directory

Would be nice if paths could be auto-completed when doing

python demetrius.py -src ... -dst ...

Maybe this helps?

https://stackoverflow.com/questions/14597466/custom-tab-completion-in-python-argparse

Two column preview in README

Show before and after in two-column layout:

https://gist.github.com/panoply/176101828af8393adc821e49578ac588#file-equal-width-md

Create separate function for duplicate check

Code would be more readable if the duplicate checks would get their own function with their own doc-strings:

demetrius/demetrius.py

Lines 166 to 178 in 1d4c6d3

	# find literal duplicates and modify the respective destination directories
	for _,dir_name in dst_dirs_df.groupby('src_dir_name'):
	if not dir_name['src_dir_path'].nunique() == 1:
	for idx,(_,src_dir_path) in enumerate(dir_name.groupby('src_dir_path'),start=1):
	dst_dirs_df.loc[src_dir_path.index,'dst_dir_path'] = dst_dirs_df.loc[src_dir_path.index,'dst_dir_path'] + '_' + str(idx)
	# find pseudo duplicates and modify the respective destination directories
	dst_dirs_df['dst_dir_path_lower_case'] = dst_dirs_df['dst_dir_path'].map(str.lower)
	for _,dst_dir_path in dst_dirs_df.groupby('dst_dir_path_lower_case'):
	if dst_dir_path['src_dir_path'].nunique() != 1:
	for idx,(_,dir_name) in enumerate(dst_dir_path.groupby('src_dir_name'),start=1):
	dst_dirs_df.loc[dir_name.index,'dst_dir_path'] = dst_dirs_df.loc[dir_name.index,'dst_dir_path'] + ' (' + str(idx) + ')'

Put file corruption check in own function

Currently, there's a very rough check for file corruption using os.path.exists(). Would probably be better to put that in a separate function (e.g. after finding all files, run over the list and check if files are corrupt).

Allow to choose directory depth

Would be nice to control the directory depth:

See here:

• 'choose number of levels to copy' dialog that allows observing relative paths for all cases

But probably not that easy to do, because then duplicate and pseudo duplicate folders would have to be checked for each level?

Add fancy progress bar

https://github.com/tqdm/tqdm

Do not copy hidden files

Is demetrius ignoring hidden files or not? I.e. files that start with a .

verbose should be on by default

Makes much more sense if this is enabled by default

Allow to pass file as input for --exclude

The list of directories that should be ignored can be quite long. Should be possible to pass an input file (.json?) to --exclude to exclude all those files.

Switch over to halo

Maybe it would be better so switch over to halo, because here we could use function decorators and then the code would be way less verbose:

https://github.com/manrajgrover/halo

One could then do something like:

if verbose:
  with Halo(text='Loading', spinner='dots'):
      function()
else:
     function()

Make pip-installable

Let users decide on own fashion of how indices for literal and pseudo duplicates should be created

Let the user decide over the fashion of added indices (e.g. '_1' or ' (1)')