gchamon / sysrsync

Simple and safe system's rsync wrapper for Python

sysrsync

Simple and safe native rsync wrapper for Python 3

Requirements

• rsync
• python 3.6+

development:

• poetry (be sure to have both poetry and pip upgraded to the latest version)

Installation

pip install sysrsync

Basic rules

• Syncs source contents by default, so it adds a trailing slash to the end of source, unless sync_source_contents=False is specified
• Removes trailing slash from destination
• Extra arguments are put right after rsync
• Breaks if source_ssh and destination_ssh are both set

Usage

• Basic file sync

import sysrsync

sysrsync.run(source='/home/user/foo.txt',
             destination='/home/server/bar')
# runs 'rsync /home/users/foo.txt /home/server/files'

• sync whole folder

import sysrsync

sysrsync.run(source='/home/user/files',
             destination='/home/server/',
             sync_source_contents=False)
# runs 'rsync /home/user/files /home/server'

• sync folder contents

import sysrsync

sysrsync.run(source='/home/user/files',
             destination='/home/server/',
             sync_source_contents=True)
# runs 'rsync /home/user/files/ /home/server'

• ssh with options

import sysrsync

sysrsync.run(source='/home/user/files',
             destination='/home/server/files',
             destination_ssh='myserver',
             options=['-a'])
# runs 'rsync -a /home/users/files/ myserver:/home/server/files'

• exclusions

import sysrsync

sysrsync.run(source='/home/user/files',
             destination='/home/server/files',
             destination_ssh='myserver',
             options=['-a'],
             exclusions=['file_to_exclude', 'unwanted_file'])
# runs 'rsync -a /home/user/files/ myserver:/home/server/files --exclude file_to_exclude --exclude unwanted_file'

• Private key

import sysrsync

sysrsync.run(source='/home/user/files',
             destination='/home/server/files',
             destination_ssh='myserver',
             private_key="totally_secure_key")
# runs 'rsync --rsh='ssh -i totally_secure_key' /home/user/files/ myserver:/home/server/files'

API

sysrsync.run

argument	type	default	description
cwd	str	os.getcwd()	working directory in which subprocess will run the rsync command
strict	bool	True	raises RsyncError when rsync return code is different than 0
verbose	bool	False	verbose mode: currently prints rsync command before executing
**kwargs	dict	Not Applicable	arguments that will be forwarded to call to sysrsync.get_rsync_command

returns: subprocess.CompletedProcess

raises:

• RsyncError when strict = True and rsync return code is different than 0 (Success)

sysrsync.get_rsync_command

argument	type	default	description
source	str	-	Source folder or file
destination	str	-	Destination folder
source_ssh	Optional[str]	None	Remote ssh client where source is located
destination_ssh	Optional[str]	None	Remote ssh client where destination is located
exclusions	Optional[Iterable[str]]	None	List of excluded patterns as in rsync's --exclude
sync_source_contents	bool	True	Abstracts the elusive trailing slash behaviour that source normally has when using rsync directly, i.e. when a trailing slash is present in source, the folder's content is synchronized with destination. When no trailing slash is present, the folder itself is synchronized with destination.
options	Optional[Iterable[str]]	None	List of options to be used right after rsync call, e.g. ['-a', '-v'] translates to rsync -a -v
private_key	Optional[str]	None	Configures an explicit key to be used with rsync --rsh command
rsh_port	Optional[int]	None	Specify port to be used for --rsh command
strict_host_key_checking	Optional[bool]	None	set StrictHostKeyChecking property for rsh #cf. https://superuser.com/questions/125324/how-can-i-avoid-sshs-host-verification-for-known-hosts

returns: List[str] -> the compiled list of commands to be used directly in subprocess.run

raises:

• RemotesError when both source_ssh and destination_ssh are set. Normally linux rsync distribution disallows source and destination to be both remotes.
• PrivateKeyError when private_key doesn't exist

Contributing

• Fork project
• Install dependencies with poetry install
• Make changes
• Lint with poetry run pylint ./sysrsync
• Test with poetry run python -m unittest
• Run end-to-end tests with bash end-to-end-tests/run-tests.sh
• Submit changes with a pull request

Contributors ✨

Thanks goes to these wonderful people (emoji key):

plazmakeks 💻

Steve Henderson 💻

This project follows the all-contributors specification. Contributions of any kind welcome!

sysrsync's Issues

collections type error in iterators.py

Running Python 3.10.0 in a venv, on macOS 11.5.

When using the option exclusions=, I'm hitting a problem in helpers/iterators.py:

def flatten(input_iter: Iterable[Any]) -> List[Any]:
    list_of_lists = (element if isinstance(element, collections.Iterable)

Specifically, it's throwing this error:

  File "/Users/rhornsby/projects/python/rs_media_sync/.venv/lib/python3.10/site-packages/sysrsync/helpers/iterators.py", line 8, in <genexpr>
    list_of_lists = (element if isinstance(element, collections.Iterable)
AttributeError: module 'collections' has no attribute 'Iterable'

As far as I can tell, this is because the collections type at some point changed to be collections.abc.Iterable (abc here is literal).

I ran into a second issue that might be my ignorance while trying to fix this. After cloning sysrsync, while trying to install the dependencies with pip3 install -r requirements.txt, it fails here:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typing-extensions>=3.7.4 from https://files.pythonhosted.org/packages/05/e4/baf0031e39cf545f0c9edd5b1a2ea12609b7fcba2d58e118b11753d68cf0/typing_extensions-4.0.1-py3-none-any.whl#sha256=7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b (from mypy==0.812->-r requirements.txt (line 48))

I think this this is because mypy doesn't specify hashes in its requirements, where it depends on typing_extensions? I'm not sure how to disable the require-hashes or somehow otherwise fix it to get past this to be able to make, test, and submit a PR for sysrsync.

Missing Releases on GitHub

Looks like a great project.

Noticed that Releases on GitHub only have 0.1.2, but PyPi has 0.2.1.

Investigate if there is a simple workflow to automate push to pypi

As described in title

sysrsync breaks when a path contains a whitespace symbol

If I invoke sysrsync like sysrsync.run(source="/mnt/foo/bar bar/buzz", destination="/mnt/dest"), it doesn't pass the source path /mnt/foo/bar bar/buzz as a single argument to rsync. Instead rsync receives it as two arguments /mnt/foo/bar and /bar/buzz and thus it exits with an error.

The version of sysrsync used is 1.0.1.

ssh password issue

How would you give ssh password without using console
is there a way to use sshpass - p in sysrsync?

Bug: strict_host_key_checking False is ignored

Issue

Setting strict_host_key_checking to false is ignored if you do not either specify private_key or rsh_port
This can occur in the following usecase: sysrsync is called by an application ran as a user who already has a preconfigured private_key specified in their ssh environment.

Replication

Prep environment:

Assumption: the user has a private_key in their ~/.ssh folder (e.g. id_rsa)

mkdir /tmp/in
mkdir /tmp/out
echo "hello" >> /tmp/in/test.txt
ssh-keygen -R 127.0.0.1

Run the following:

import sysrsync

sysrsync.run(source="/tmp/in",source_ssh="127.0.0.1",options=['-a'],strict_host_key_checking=False, destination="/tmp/out")
                  

Will yield something like this:

python rsync_check.py 
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
ED25519 key fingerprint is SHA256:6a26OyQqLw5USUetuC/3aKO36/JxE4jK1NHNbDmXGIQ.
This host key is known by the following other names/addresses:
    .
    .
    .
Are you sure you want to continue connecting (yes/no/[fingerprint])?  ssh-keygen -R localhost

Cause

The following check in sysrsync/command_maker.py will not return true (and thus return an empty options list) if the strict_host_key_checking value equals False and private_key and rsh_port are not specified:

if any((private_key, rsh_port, strict_host_key_checking))

Solution

PR 32 Submitted

Nose is no longer maintained, port tests to unittest

As title says

Broken --rsh option

In pr single quotes got removed from --rsh option, doesn't work anymore.

Add docstrings to libraries and helpers

As pointed by pylint

Save output from rsync

I'd like to be able to parse the output from the rsync command in order to log what was updated. I believe this can be implemented in the subprocess.run command by passing capture_output=True. I'm wondering if this capability was omitted on purpose?

Allow multiple sources

Rsync supports multiple source paths. For example,

When source is local:

rsync /path/to/file1 /path/to/file2 /path/to/file3 user@ip:/dest/path/

When source is remote:

rsync user@ip:/path/to/file1 user@ip:/path/to/file2 user@ip:/path/to/file3 /dest/path/

The code here supports only a single source, although I think the adjustment would be straightforward.

Add option to specify path to rsync

On some systems rsync isn't in the PATH. Could we add support for specifying the path to the rsync executable for these situations? It could be an optional argument to sysrsync.run for example

Add private key option

Add option to use a private key

Missing Rsync options

Am I not seeing it, or does this wrapper not have the rsync option "inclusion" available? "exclusion" seems to be working, but attempting inclusion in the same approach results in "TypeError: get_rsync_command() got an unexpected keyword argument 'inclusion'"?

An how do I add rsync options like "--numeric-ids"?

Can I define "--log-file" and "--link-dest"?

Inconsistent use of Type Hints

I noticed that there is an inconsistent use of Type Hits within this program.

Some modules and functions have type hints in place, others do not.

Specifically, the run command doesn't have type hints, while many of the other commands do.

sysrsync/sysrsync/runner.py

Line 9 in c978be1

def run(cwd=os.getcwd(), strict=True, verbose=False, **kwargs):

Have you considered adding type hints?

get_rsync_command tries to connect via SSH

Probably a public API titled get_rsync_command should not remotely connect to another machine to generate the command.

This is unnecessarily called even when sync_source_contents if False, because there is no guard on this statement. Easy fix on that though.

other SSH options: port

How can I specify a custom SSH port ?
How can I add other SSH options like StrictHostKeyChecking=accept-new