iemejia / catho

A file catalog utility inspired by the awesome Robert Vasicek's Cathy project. Or my excuse to hack something that I really need.

Catho

Catho is a catalog utility inspired by the awesome Robert Vasicek's Cathy project. The idea is to have an util to save the catalog of the different files that you have in different media, volumes, network places, etc, that you can search, and update locally without having to put or connect to such media.

Or put in other words, it's my excuse to hack some python. Yes, yes I promised to do it in haskell, but i don't have time now :P.

Installation

TODO

Requirements

For python < 2.7 pip install argparse

Use

# Prepare catalog folder
catho init

# Add a catalog with alias
catho add name path  

# Remove a catalog
catho rm name

# Search for filenames matching with a pattern (ex. *.zip, c*.*) in some catalogs or in all if none is provided

catho find pattern [catalog1] [catalog2] [catalogn]

# List all catalogs
catho ls

# Find apparently existing files from dir in the catalog
catho scan name dir

# Import existing cathy catalogs to the catho format
catho import file.cat

Developing

The catalog correspond to a simple sqlite3 database, for more info about the catalog structure, see the docs/catalog.sql file. Catalogs are saved automatically in the ~/.catho folder.

We would like to create a sort of simple minimalist catalog system so people can create nicer tools based on the catho system (e.g. GUI, web, etc).

Continuous integration is running automatically via travis-ci . The current project status is:

Collaboration

Contributions are welcome, but please add new unit tests to test your changes and/or features. Also, please try to make changes platform independent and backward compatible.

catho's Issues

Add progress when adding new catalog

Refactor internal representation

Remove the global catalogs variable and put it in an object
Create class for the DAO operations (maybe the same)

Return the name of the catalog in the find

Following discussion #7 (comment)

refactor to pep-8 standard

Implement hashing

incompatible import with python 3

The import:
from utils import get_file_info
throws an error with python 3

Configure Continous Integration via Travis

Incremental scan (useful also for cathod)

Save only relative paths

When saving catalog, save the full path in the metadata.
And only relative paths in the catalog table.
Currently it works for ., but it fails with the system auto completed paths (e.g. ~)

create a GUI for catho

It could be a textual or graphical gui

implement args parsing with argparse

Create PIP Installer

Support native binaries, (.exe file) via py2exe

Write docs, tutorial

Tutorials:

• How to use catho
• Encrypting your catalog
• Synchronizing with dropbox

Implement scan option

Error parsing hardlinks

Not such file or directory while parsing a hardlink

$ ls -al ~/Dropbox/iPad/
lrwxr-xr-x 1 rgamez staff 38 18 feb 2012 test.pdf -> /Users/rgamez/Documents/test.pdf

$ ./catho.py add iPad ~/Dropbox/iPad
Creating catalog: iPad
An error occurred: [Errno 2] No such file or directory: '/Users/rgamez/Dropbox/iPad/test.pdf'

create a configuration mechanism

something similar to .cathoconfig
that allows the minimum possible configurations, e.g. string output format

add option to continue indexing

add -c to enable the completion of an unfinished indexing, or the update of a catalog in case of changes.

Error indexing names with quotes in continuous indexing

if a file name includes the character ' in it it throws an error when it builds the query.

Index metadata (e.g. id3tags, etc)

create path traversal method

similar to os.path, but for catho paths.

Implement help

--help -h

Add path in metadata

Implement as REST web services

Write results to stdout

Writing with the logger to stderr by default prevent of using unix tools like grep for filtering, for example $ catho/catho.py find \* media | grep Animal just print 133652 records from the test catalog.

There is a workaround, redirecting stderr to stdout, but it is not really straight forward.
$ catho/catho.py find \* media 2>&1 >/dev/null | grep Animal

Fix test so they don't create again the database when it's not needed

Error indexing when a file has the same name of a directory

try to index e.g. /file/file and boom, the problem is in the path_block_iterator, in the way it builds the path names.

Error with unicode characters in paths

Add decent logging

including debug and verbose options.

check if database exists and abort

Optional: create option -f to force creation.

an option to optimize the databases

like 'git gc', it executes ANALYZE; and then VACUUM; in sqlite

Implement cathy import option

Implement a Cathy (.cat) importer

add before init is not fatal error

if ~/.catho doesn't exist because catho init hasn't been executed, errors are displayed but the parsing of the directory continues.

$ ~/catho/catho/catho.py add Home ~
Creating catalog: Home
An error occurred: unable to open database file
An error occurred: unable to open database file
An error occurred: unable to open database file
$ ls ~/.catho
ls: cannot access ~/.catho: No such file or directory

Two options/suggestions @iemejia

• It's a fatal error that ends with the suggestion of executing catho init.
• ~/.catho it's "touched" before adding any catalog and perhaps any operation.

Create a mechanism to ignore paths or filetypes

Probably a file similar to .gitignore, because folders like .git or .svn can return many false positives.

Operate over the metadata

To add, or change values such as basepath
something like -set-meta key, value

Encrypt database

for security reasons in case of use in cloud servers
-e --encrypt
we have to decide a schema

Implement search of find option

Implement rm option

Support detection of repeated folders

A function to detect repeated folders (if n hashes are equal per folder there is a possiblitiy).
How to optimize ?

fix space in catho ls output

output of catho ls should be more consitent in columns, something like ls -l

Create tests

List catalog information

with the ls command plus the name of the catalog, it should display metadata and eventually the contents.

cathod service

a runtime service that automatically indexes modified files and upgrades the catalog.

Something to detect junk-files

something to remove useless files (.DS_Store, *.url, Torrent downloaded from, etc)

Add fdupes command

Based on the idea of:
https://github.com/seabre/finddupes/blob/master/finddupes.sh
SELECT hash, location FROM files WHERE hash NOT IN (SELECT hash FROM files GROUP BY hash HAVING ( COUNT(hash) = 1 )

TypeError on listing the catalogs

$ catho/catho.py ls
Traceback (most recent call last):
File "catho/catho.py", line 334, in
logger.info(catalogs_str())
File "catho/catho.py", line 226, in catalogs_str
date = str(datetime.fromtimestamp(timestamp))
TypeError: a float is required

Memory Error

When indexing really long volumes

Add tag support

We have to discuss what to support:

• filesystem tags ?
• user generated tags ?
• inferred tags (e.g. .mp4 = video, or .mp3 = audio ?

Error when calculating hash of BIG files

I found it trying to index a VM that's bigger than the memory in my local machine:

Python(15742) malloc: *** mmap(size=140645843243008) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "catho/catho.py", line 303, in
for files in filesubsets:
File "catho/catho.py", line 77, in file_get_filelist
hash = file_hash(fullpath, hash_type)
File "catho/catho.py", line 50, in file_hash
sha1.update(f.read())
MemoryError

Implement ls option