fabric / patchwork

Common deployment/sysadmin operations, built on Fabric

Welcome to Fabric!

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.

To find out what's new in this version of Fabric, please see the changelog.

The project maintainer keeps a roadmap on his website.

patchwork's Issues

py3 compatibility - iteritems

The usage of iteritems() is not py3 compatible.

Can't currently use the package() directive in fabric2 tasks on python3.

Docker module

I.e. Docker management commands that wrap the docker CLI on a remote system.

This might want to live in Invocations instead, depending; at the very least it may be a good example of a sometimes-local, sometimes-remote use case.

Triggered by fabric/fabric#1874

Patchwork 0.2.0 not install [Python 3]

$ pip install patchwork==0.2.0
Downloading/unpacking patchwork==0.2.0
  Downloading patchwork-0.2.0.tar.gz
  Running setup.py (path:/home/patrick/Envs/cursodeextensao/build/patchwork/setup.py) egg_info for package patchwork
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/patrick/Envs/cursodeextensao/build/patchwork/setup.py", line 9, in <module>
        version_module = execfile('patchwork/_version.py', _locals)
    NameError: name 'execfile' is not defined
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/patrick/Envs/cursodeextensao/build/patchwork/setup.py", line 9, in <module>

    version_module = execfile('patchwork/_version.py', _locals)

NameError: name 'execfile' is not defined

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /home/patrick/Envs/cursodeextensao/build/patchwork
Storing debug log for failure in /home/patrick/.pip/pip.log

What does patchwork.util.set_runner do?

The documentation is a bit unclear to me. What would be a use case? A more elaborate example might help.

transfers.rsync() error when excluding tuple

Using transfers.rsync() and excluding tuple:

@task
def rsync(ctx):
    transfers.rsync(
        ctx,
        source=SOURCE,
        target=TARGET,
        exclude=('.git', '*.pyc')
    )

Generates the following error:

Traceback (most recent call last):
  File "/Users/kegan/.pyenv/versions/indoorcms-pyramid3/bin/fab", line 11, in <module>
    sys.exit(program.run())
  File "/Users/kegan/.pyenv/versions/3.6.5/envs/indoorcms-pyramid3/lib/python3.6/site-packages/invoke/program.py", line 332, in run
    self.execute()
  File "/Users/kegan/.pyenv/versions/3.6.5/envs/indoorcms-pyramid3/lib/python3.6/site-packages/invoke/program.py", line 480, in execute
    executor.execute(*self.tasks)
  File "/Users/kegan/.pyenv/versions/3.6.5/envs/indoorcms-pyramid3/lib/python3.6/site-packages/invoke/executor.py", line 133, in execute
    result = call.task(*args, **call.kwargs)
  File "/Users/kegan/.pyenv/versions/3.6.5/envs/indoorcms-pyramid3/lib/python3.6/site-packages/invoke/tasks.py", line 127, in __call__
    result = self.body(*args, **kwargs)
  File "/Users/kegan/projects/indoorcms-pyramid3/fabfile.py", line 50, in rsync
    delete=True
  File "/Users/kegan/.pyenv/versions/3.6.5/envs/indoorcms-pyramid3/lib/python3.6/site-packages/patchwork/transfers.py", line 116, in rsync
    'exclude': exclude_opts.format(exclusions),
IndexError: tuple index out of range

rsync() generates invalid rsync command line if IPv6 address used

bug in rsync() function fom patchwork/transfers.py:

if env.host.count(':') > 1:
        # Square brackets are mandatory for IPv6 rsync address,
        # even if port number is not specified
        cmd = "rsync %s %s [%s@%s]:%s" % (options, source, user, host, target)
    else:
        cmd = "rsync %s %s %s@%s:%s" % (options, source, user, host, target)

for IPv6 rsync address be "%s@[%s]", not "[%s@%s]" as it is now.

patchwork source code: https://github.com/fabric/patchwork/blob/master/patchwork/transfers.py#L111

details about IPv6 adresses with rsync: https://serverfault.com/questions/420158

Better remote system info handling

E.g. determine which family and/or distro and/or release, a remote target is.

Currently patchwork.info does this in a super limited/basic fashion, which needs extending.

To do:

• Look for prior art in Fab 461's linked libs
• Look at how Facter/Ohai do it
• See this IRC discussion for other notes on the implementation

broken link to http://patchwork.fabfile.org

on https://github.com/fabric/patchwork the link to http://patchwork.fabfile.org is broken.

`transfers.rsync` ignores ssh agent and always asks for key password

Thanks for making patchwork compatible with Python 3! Trying to port one small project to Fabric v2:

from invoke import task
from patchwork.transfers import rsync

def get_proj_path():
    from fabric.main import program
    return program.collection.loaded_from

@task
def upload(c):
    rsync(c, '%s/configs' % get_proj_path(), '/tmp')

> /Users/user/.virtualenvs/test1/lib/python3.6/site-packages/patchwork/transfers.py(131)rsync()
    129     cmd = cmd.format(options, source, user, host, target)
    130     import ipdb; ipdb.set_trace()
--> 131     return c.local(cmd)

Remote command works fine with ssh agent running:

ipdb> print(c.run('pwd'))
/root
Command exited with status 0.
=== stdout ===
/root

(no stderr)

c.local("rsync -pthrvz --rsh='ssh -p 22 ' /Users/user/work/project/configs [email protected]:/tmp") asks for ssh key:

ipdb> c
Enter passphrase for key '/Users/user/.ssh/id_rsa':

Running the same command from the shell works fine:

% rsync  -pthrvz  --rsh='ssh  -p 22 ' /Users/user/work/project/configs [email protected]:/tmp
sending an incremental file list

sent 237 bytes  received 18 bytes  72.86 bytes/sec
total size is 10.45K  speedup is 40.99

Calls to Result.succeeded fail

See fabric/fabric#1764

'transfers.rsync' hangs if host not accessible

In a simple Fabric task that uses transfers.rsync to update a file on a host, everything works well if the host is accessible. If the host is down for some reason, though, rsync seems to hang indefinitely (well, I've waited 10 minutes or so) rather than time out and raise an exception. This task runs on a schedule, so it's important that it fail cleanly.

As a workaround, I just do a

c.run("pwd")

before the rsync step. If the host is inaccessible, run fails quickly as expected and gives me my exception.

Thank you for your time and consideration!

http://patchwork.fabfile.org/ doesn't seem to exist

Just found this project by accident, tried the link in the readme, not found.

S

files.exists relies on SSH commands

Hello,
I'm using patchwork 1.0.1 with python 3.10.12 and Fabric 2.7.1.

I want to check if a file exists on a pure SFTP server (no SSH). This fails because files.exists uses test-e to check wether the file exists. This command is not available on pure sftp via openssh.

Append some lines in README to explain the difference between patchwork and Fabtools

Append some lines in README to explain the difference between patchwork and Fabtools http://fabtools.readthedocs.org/en/0.17.0/

files.append with sudo failed

Trying to use file.append with sudo option:

# ctx is fabric context
files.append(ctx, filename='/home/foo/.profile', text='export foo=bar', sudo=True)

The underneath cmd looks like:

sudo -S -p '[sudo] password: ' echo  'export foo=bar' >> /home/foo/.profile

and giving err msg:
bash: /home/foo/.profile: Permission denied
Also tried with runner_method='sudo' or runner=ctx.sudo and facing same issue

Found something might related
https://unix.stackexchange.com/questions/4335/how-to-insert-text-into-a-root-owned-file-using-sudo

Improved copy of fabric.contrib.rsync_project

• Don't fucking name it rsync_project
• Add in a kwarg for the StrictHostKeyChecking=no as this is a commonly used ssh sub-option
• Don't print both your own "I am running command X!" and allow local() to print. Hide one or the other.

rsync not found on windows system

Trying to run rsync on Windows system, got:

'rsync' is not recognized as an internal or external command, operable program or batch file

System PATH includes a right path to the rsync executable (I can run it from command line from any directory). Have tried to manually add PATH to context variable:

from fabric import Connection
from patchwork.transfers import rsync

with Connection(HOST) as c:
  c.config['env'] = {'PATH': r'C:\Program Files (x86)\cwRsync\bin'}
  rsync(c, src, dst)

with no results.

Add installation instructions

Add installation instructions (even if they're simple) to either the README or to https://fabric-patchwork.readthedocs.io.

Porting the files.sed operation?

Is there any plan on porting the sed operation from Fabric v1?

`transfers.rsync` don't support rsync from remote to local

docs: declarative or imperative configuration management?

Configuration management can be declarative or imperative.

I guess you know the difference, but I explain it here for other people:

declarative : There should be a symlink from file "foo" to "bar".

imperative/procedural: Create a symlink from "foo" to "bar".

Reading the current docs, it looks like Patchwork will go the imperative way. Please tell in the docs which way you want to go. Thank you.

single quotes not properly escaped

This line doesn't seem to properly escapes single quotes using bash as a shell: https://github.com/fabric/patchwork/blob/master/patchwork/files.py#L130, possibly because bash needs a $'...\'...' (https://wiki.bash-hackers.org/scripting/bashchanges#quoting_expansions_substitutions_and_related).

patchwork.files.append(c, filename="/etc/postgresql/10/main/postgresql.conf", text="listen_addresses = '*'", escape=True)
# line that is added:
# listen_addresses = \*\\

This script shows the problem that i am experiencing:

import subprocess

def run(script):
    process = subprocess.run(
        ["bash", "-s"], capture_output=True, input=script.encode("ascii"))
    stderr = process.stderr.decode("ascii")
    print(stderr, end="")
    stdout = process.stdout.decode("ascii")
    print(stdout, end="")

line = "listen_addresses = '*'"

escaped = line.replace("'", r"'\\''")
script = f"echo '{escaped}'"
run(script)
# listen_addresses = \*\\

line = "listen_addresses = '*'"
escaped = "$'" + line.replace("'", r"\'") + "'"
script = f"echo {escaped}"
run(script)
# listen_addresses = '*'

Thank you for Fabric! :)

Exists function is crashing.

Here is the error I have when I try to use the exists function:

Traceback (most recent call last):
  File "env/bin/fab", line 9, in <module>
    load_entry_point('fabric==2.0.0', 'console_scripts', 'fab')()
  File "/home/tim/Workspace/wintest/env/lib/python3.5/site-packages/invoke/program.py", line 332, in run
    self.execute()
  File "/home/tim/Workspace/wintest/env/lib/python3.5/site-packages/invoke/program.py", line 480, in execute
    executor.execute(*self.tasks)
  File "/home/tim/Workspace/wintest/env/lib/python3.5/site-packages/invoke/executor.py", line 133, in execute
    result = call.task(*args, **call.kwargs)
  File "/home/tim/Workspace/wintest/env/lib/python3.5/site-packages/invoke/tasks.py", line 127, in __call__
    result = self.body(*args, **kwargs)
  File "/home/tim/Workspace/wintest/fabfile.py", line 58, in website_dependencies
    if not exists(context, remote_env_path):
  File "/home/tim/Workspace/wintest/env/lib/python3.5/site-packages/patchwork/util.py", line 113, in inner
    return f(*args, **kwargs)
  File "/home/tim/Workspace/wintest/env/lib/python3.5/site-packages/patchwork/files.py", line 31, in exists
    return runner(cmd, hide=True, warn=True).succeeded
AttributeError: 'Result' object has no attribute 'succeeded'

It looks like the attribute succeeded have been replaced with ok in recent versions.

cannot import name 'six' from 'invoke.vendor'

>>> import patchwork._version
>>> patchwork._version.__version__
'1.0.1'
>>> import invoke
>>> invoke.__version__
'1.3.0'
>>> import patchwork.files
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/patchwork/files.py", line 7, in <module>
    from invoke.vendor import six
ImportError: cannot import name 'six' from 'invoke.vendor' (/usr/lib/python3/dist-packages/invoke/vendor/__init__.py)

Abandonware?

I love the idea of using a Fabric based deployment setup but am a bit concerned that there's been no work done in the last 4 months. Is this project still being maintained?

support fabric 3.0