fld / shatag-scrub Goto Github PK
View Code? Open in Web Editor NEWshatag in the fight against silent data corruption / data rot
License: GNU General Public License v3.0
shatag in the fight against silent data corruption / data rot
License: GNU General Public License v3.0
OVERVIEW Shatag is a tool for computing and caching file checksums, and do remote duplicate detection. Files are compared on their SHA-256 hash to find duplicates, and will use filesystem extended attributes to cache the checksum values. REQUIREMENTS - Python 3 - argparse - pyxattr (until other backends become available) - If you want to use a database store: sqlite3 or psycopg2 - If you want to use an http remote store: requests - If you want to use the http store server: the Bottle framework QUICKSTART Before you attempt to use shatag, make sure the filesystem you will be working on has support for user extended attributes. This is typically enabled by adding the user_xattr mount option to the appropriate filesystem in /etc/fstab. You can also enable support without unmounting with: sudo mount -o remount,user_xattr /mountpoint If you want to use a remote database, setup a configuration file ~/.shatagrc with contents like: database: "pg: dbname=shatag host=localhost user=shatag password=xxxsecretpasswordxxx" Next, start tagging some files with: shatag -qrt /home/data Upload the proper tags to the database with: shatag -qrp /home/data (you can combine the -t and -p flags in one command) You can start a daemon to automatically recompute tags when needed: shatagd -dr /home/data Check for duplicates in a directory with shatag -rl . SUMMARY OF TOOLS shatag(1) is the main tool, and used for computing and displaying file checksums, as well as reporting duplicate. shatagd(1) will monitor directories for file changes and update the checksums automatically. it requires pyinotify. Refer to the man pages of these tools for more information. TECHNICAL DETAILS shatag stores a single sha256 checksum in hexadecimal (64 ascii characters) in the "shatag.sha256" attribute, and the modification time of the file at the moment the checksum was computed, as decimal unix time, in "shatag.ts". The checksum is considered valid if the modification time of the file is still equal to the timestamp stored in the attribute. With respect to race conditions, shatag will always read the stored timestamp before the checksum, and the checksum before the actual mtime; it will always write the checksum before the timestamp. In case of accidental parallel operation, shatag will rather compute unneccesary checksums than consider an old checksum valid. The local sqlite database contains a single table: create table contents(hash text, name text, path text, primary key (hash,name,path)); When querying the database, shatag will lookup the hash and count the matching paths at each location. The file will be considered a dupe if any location has the file more than once. TODO: * Fix probable encoding quirks for filenames
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.