chriscummins / emu

Fast, incremental, rotating backups

emu - Fast, incremental rotating backups

Emu is a command line backup tool, inspired by git and Time Machine. It is:

• Fast: The battle tested, optimized rsync perform the heavy lifting of transferring files.
• Incremental: Hardlinks minimize the overhead of storing multiple backups when few files change.
• Rotating: All backups are kept within the past day, daily backups are kept for the past month, weekly backups are kept for previous months. The oldest backups are removed to make space for the newest.

Additionally, emu is:

• Transparent: There are no proprietary formats or data blobs. Backups are files; metadata is plaintext.
• Accessible: Use your favorite file manager to browse old backups, or use the Python API for more control.
• Simple: The core operations are simple: It creates new backups, it deletes old backups as required. One line in your crontab is all it takes for peace of mind.

Installation

Requires Python >= 3.6.

$ pip install emu

After installation, we recommend running the test suite to ensure everything is in tip-top working order:

$ emu test

License

Copyright © 2012-2020 Chris Cummins.

Released under the terms of the GPLv3 license. See COPYING for details.

emu's Issues

C-c during emu verify causes an IndexError

$ emu verify -S /mnt/primary
^Cemu: received SIGINT
Traceback (most recent call last):
  File "/usr/local/bin/emu-verify", line 63, in <module>
    ret = main(argv, len(argv))
  File "/usr/local/bin/emu-verify", line 52, in main
    use_cached=options.use_cached):
  File "/usr/local/libexec/emu/libemu.py", line 602, in verify
    clean = worker_thread.get() == self.id.checksum
  File "/usr/local/libexec/emu/libemu.py", line 2485, in get
    return stdout.split()[0][:32]
IndexError: list index out of range

$ emu version
emu version 0.1.50

Merge `emu status' into `emu stack'

Add emu-monitor

usage: emu monitor 4000. launches a web server on port 4000 which provides a visual log of backups

Split out Util class

lockfile: Claim lock from dead processes

Mac OS X compatability issues

 ./scripts/run-tests.sh                                              19:30:49
    clean/deadlink PASS
    clean/in-progress PASS
    clean/lock PASS
    clean/orphans FAIL
    init/init PASS
    init/permissions PASS
    init/quiet PASS
    init/reinitialize PASS
    init/template PASS
    init/update PASS
    log/limit FAIL
    log/log PASS
    log/no-sink PASS
    log/push FAIL
    log/sinks FAIL
    log/summary FAIL
    push/dry-run PASS
    push/excludes FAIL
    push/max FAIL
    push/message FAIL
    push/no-sink FAIL
    push/push FAIL
    sink/add PASS
    sink/rm PASS
    sink/sink FAIL
    verify/hash FAIL
    verify/invalid-hash FAIL
    verify/sink-hash FAIL

14 passed, 14 failed
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1
head: illegal line count -- -1

A log file of failed tests can be found at 'test.log'

Replace black box tests with python unit tests

run-tests: Doesn't report skipped tests on interrupt

If you cancel running the tests early, you don't get any notification that some tests were missed. Example:

$ ./scripts/run-tests.sh 
    clean/deadlink PASS
    clean/debug PASS
    clean/in-progress PASS
    clean/lock PASS
    clean/orphans ^C
All 4 tests passed

Split up monolithic __init__ file

Add unit tests and coverage, CI server, and coverage reports

Add explicit support for branching

Prune the feature bloat

Features to remove:

• the squash command
• the HEAD pointer, and anything to do with branching
• checkout - replace it with a 'restore' command
• most of the command line flags. they should either go in .emu/config, or be hardcoded
• the explicit clean command - this should be done automatically on push

configure: Check for required packages

• Python (v2)
• rsync

Implement an io module

Interface:

enable_verbose_messages()
disable_verbose_messages()
verbose(*args, **kwargs)

enable_debug_messages()
disable_debug_messages()
debug(*args, **kwargs)

enable_warning_messages()
disable_warning_messages()
warning(*args, **kwargs)

error(*args, **kwargs)

fatal(*args, **kwargs)

Emu squash merges snapshots in chronological order

This means that the older files overwrite the newer files. This order should be reversed.

Emu squash always sets generated snapshot as new HEAD

Replace .emu/excludes with .emuignore files

A la .gitignore

Revised directory naming

emu stack causes uncaught TypeError

$ emu sink -v
Traceback (most recent call last):
  File "/usr/local/bin/emu-sink", line 128, in <module>
    ret = main(argv, len(argv))
  File "/usr/local/bin/emu-sink", line 49, in main
    Util.readable(options.source_dir, error=True)
  File "/usr/local/libexec/emu/libemu.py", line 1502, in readable
    Util.exists(path, error=error)
  File "/usr/local/libexec/emu/libemu.py", line 1481, in exists
    exists = os.path.exists(path)
  File "/usr/lib/python2.7/genericpath.py", line 18, in exists
    os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found

$ emu --version
emu version 0.1.51

s/sink/remote/

Let's at least by consistent with git

Add `emu status` command

$ emu status
New snapshot in progress, started 12 hours ago

$ emu status
Last snapshot 2017-06-14-113212 created 3 days ago

$ emu status --seconds
0  # if snapshot in progress

$ emu status --seconds
189  # if last snapshot 3 minutes old

Add -f --force argument to all emu commands

Operations for which --force is not supported:

• emu prune

Reimplement separate scripts as single python entry point

See how clgen does it.

https://github.com/ChrisCummins/clgen/blob/master/clgen/cli.py