Coder Social home page Coder Social logo

josync's Introduction

Josync is backup using rsync on Windows

Josync is a light-weight Python application for backup using rsync/cygwin, using shadow copies for read access to open files.

Documentation at http://josync.readthedocs.org

Please post any bugs you experience in the issue tracker!

Josync is released under a MIT License (see LICENSE.md)

Why?

We wanted a simple, scriptable backup on Windows for personal use.

The goal

Timeline backups using rsync on Windows. The aim is a Timeline job type which makes chronological backups (getting sparser as time passes) with hard links for unchanged files.

Right now

We are at version 1.0 where the core functionality and utilities are in place. A regular syncronization job runs, and logging works.

Next up

* Easy installation by bundling everything in an installer
* Timeline job type
* (and see Issue tracker for more, or for posting requests!)

Please note

Josync is a new project under development, and can not in any way be regarded as ready for production. That said, we run it on our own files without any hesitation.

josync's People

Contributors

joelgoop avatar

Watchers

Jonas Einarsson avatar avatar

josync's Issues

Add marker file to in target directory

I just had josync fill a USB stick that was temporarily connected to my computer, since it happened to get the same drive letter that my external backup drive usually has. To avoid this, we could put a marker file in the target directory telling josync that it is a valid target. If josync can't find the marker, the user can be prompted to create it.

Go public?

Make repo public (maybe on github) and use readthedocs.org for hosting docs. I see no point in repo being private. Yes/No/Dead cat

Capital drive letters with mapped network folders cause error

I am syncing to a network location like \\domain.com\myfolder\sync. For convenience I have mapped my home folder on the network, \\domain.com\myfolder, to a drive letter H:. If i set up the sync job with "target" : "h:/sync", everything works as intended. However, if I change to a capital drive letter, "target" : "H:/sync", josync crashes with the following output:

__main__ [INFO] ************************************************************
__main__ [INFO] Session started. Josync version 0.0.
utils [INFO] net use reported 2 mapped drives
utils [WARNING] Config file user.josync-config not found (ignoring).
utils [INFO] Creating FailureNotifier from test.josync-job.
utils [INFO] No notification settings were found. Disabling notifications.
jobs [INFO] Creating Job from test.josync-job.
josync_run [ERROR] Josync job test.josync-job failed with an exception: u'H:'
__main__ [ERROR] u'H:'
Traceback (most recent call last):
  File "josync.py", line 55, in main
    job = jobs.create_job_from_file(jobfile)
  File "c:\temp\josync\jobs.py", line 189, in create_job_from_file
    return job_types[params['type']](params)
  File "c:\temp\josync\jobs.py", line 145, in __init__
    super(SyncJob, self).__init__(params)
  File "c:\temp\josync\jobs.py", line 117, in __init__
    super(BaseSyncJob, self).__init__(params)
  File "c:\temp\josync\jobs.py", line 35, in __init__
    unc = utils.net_drives[target_drive]
KeyError: u'H:'
__main__ [INFO] Session ended.

I am running josync 1.0 on Windows 7 x64.

test vshadow error

i made a new error type for vshadow errors, and caught it in the main file. Please test running it without admin privileges to see if it works ;)

Excludes are relative

Excludes cannot be given as absolute paths and we should therefore have source-specific excludes and global excludes. Right now "excludes" options in job-file means global.

Generate notifications on errors and warnings

If the backup does not succeed, we should notify the user somehow. It could be as simple as an extra logging handler for the "error" and "warning" levels? e-mail, syslog, facebook status update, run a program, display a message, i don't know :) probably an e-mail is a good start.

Wrong file recommended for SMTP settings

The documentation on failure notification says "The SMTP settings goes in default.josync-config". This is not good practice since it will be overwritten by an update. user.josync-config should be used.

Make compatible with pythonw

To be able to schedule to run in background, the easiest solution appears to be to run with pythonw. However, it currently fails without any exceptions being logged, probably due to writing to stdout. In josync, stdout communications come from logging and from the subprocess call to rsync

Stats report wrong total size

Example:

Josync job work.josync-job successfully run. 4.6 of 841567.4 kB updated (0.0 %).

However, target folder is 87 GB, some orders of magnitude off.

Updates unnecessary files

rsync updates files that have not changed, probably due to some windows issue with modified dates.

Missing cygwin install does not give sufficient feedback in log file

If a cygwin install is missing, josync crashes because it cannot find cygpath. However the logs from a normal run does not display any information about the problem, only the lines

2015-08-13 12:43:58,059 [INFO] ************************************************************
2015-08-13 12:43:58,059 [INFO] Session started. Josync version 1.0.
2015-08-13 12:43:58,085 [INFO] net use reported 1 mapped drives

Run Rsync

More specifically:

  • decide on structure at target (full paths, starting with drives?)
  • generate rsync commands
  • run rsync
  • parse or at least display output
  • log success/failure in some reasonable fashion
  • does rsync give a list of files updated?

Hints for documentation

After a little bit of tinkering, I have successfully set up a josync job from my work computer to the company network. A couple of things noted on the way which may help other users:

Escapes

I am backing up to my home folder on the network, which is located at something like \\domain.com\myfolder. Writing a josync-job file with

"target": "\\domain.com\myfolder\sync"

fails with an error message like josync_run [ERROR] Josync job test.josync-job failed with an exception: Invalid \escape: [line/column/char].

Everything works well after escaping the path like

"target": "\\\\domain.com\\myfolder\\sync"

File permissions

The other problem is that I don't have "Full control" permissions on my home folder on the network. I can read/write/modify/execute, but not change NTFS permissions on files. The result was that each file in the backup generated a warning message like

utils [WARNING] rsync: failed to set permissions on "//domain.com/myfolder/sync/myfile.ext": Permission denied (13)

and at the end also

utils [WARNING] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /usr/src/ports/rsync/rsync-3.0.9-1/src/rsync-3.0.9/main.c(1052) [sender=3.0.9]

and finally, after successfully deleting the shadow copy, josync threw an error

josync_run [ERROR] Josync job test.josync-job failed with an exception: rsync returned with exit code 23.

The backup seemed to work though, all files being copied with reasonable file permissions etc. But the error was annoying me and I eventually found a solution. You can tell cygwin to not modify the file system Access Control Lists (ACLs) as explained in the cygwin documentation. Long story short, problem was solved by adding the noacl option in the the file c:/cygwin64/etc/fstab. My fstab file now says

none /cygdrive cygdrive binary,noacl,posix=0,user 0 0

and everything now works like a charm.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.