ronin13 / pyvolume

Docker Volume plugin in Python

===============================

PyVolume

Python Docker Volume driver.

Supports pluggable implementations, currently there are three written.

Implements:

• '/Plugin.Activate'
• '/VolumeDriver.Create'
• '/VolumeDriver.Remove'
• '/VolumeDriver.List'
• '/VolumeDriver.Path'
• '/VolumeDriver.Mount'
• '/VolumeDriver.Unmount'
• '/VolumeDriver.Get'
• '/VolumeDriver.Capabilities'

for Docker Volume.

and

• '/'
• '/shutdown'

for management.

The volume manager (common to all drivers) uses Flask for routing and handles multiple invocations of Mount and Unmount for same volume as per docker specifications. It also passes options passed through API to the drivers. Cleanup is also handled on shutdown.

Current Implementations

• Ephemeral FileSystem
• SSHFS FileSystem
• Zookeeper FileSystem
  • This uses docker-zkfuse for using zkfuse and mounts zkfuse from container to host through shared mounting of volume from host to container.

Installing

1. Install the package.

pip install -r requirements.txt --user
python2 setup.py install --prefix=/usr/local

After this pyvolume should be available as /usr/local/bin/pyvolume.

2. Copy the pyvolume.json to /etc/docker/plugins/

Dependencies:

Installation

1. Python 2.7 and python related dependencies - pip, virtualenv etc.
2. sshfs for SSHFileSystem (default).

Integration Testing Dependencies.

1. ssh-add and sshfs.
2. curl
3. util-linux (for mount etc.)
4. Python-related tools such as virtualenv.

Usage

Permissions

• Make sure you have write permissions for base mount directory which is /mnt by default.
  • If not, make sure to chown as the user you run pyvolume as.
• For ZookeeperFileSystem, make sure the pyvolume's user can do docker run without sudo.
  • If not, add the pyvolume's user to docker group.

For SSH FileSystem

Arguments for docker volume create:

• remote_path: as host:directory (Required)
• ssh_config: path to ssh config directory if it is not default.
• sshfs_options: any options to pass to sshfs.

After Installing,

1. Make sure the ssh keys are available through the ssh-agent.

ssh-add -l

2. Start the pyvolume server.

    $ /usr/local/bin/pyvolume
    INFO:werkzeug: * Running on http://0.0.0.0:1331/ (Press CTRL+C to quit)

3. Create a docker volume.

    docker volume create -d pyvolume --name myvolume2 -o 'remote_path=server:/home/user' -o 'ssh_config=/home/rprabhu/.ssh.bkp/config.server'

4. Run docker as usual, providing the newly created volume name.

   docker run -it -v myvolume2:/data  busybox:latest sh

5. PROFIT!

For ZooKeeper FileSystem

Arguments for docker volume create:

• zookeeper_string - host:port (or a list of tuples) to running zookeeper instance. (Required)
• docker_opt - any options to pass to docker.

1. Start the pyvolume server.

    $ /usr/local/bin/pyvolume -t zookeeper
    INFO:werkzeug: * Running on http://0.0.0.0:1331/ (Press CTRL+C to quit)

3. Create a docker volume.

    docker volume create -d pyvolume --name zoo -o 'zookeeper_string=0.0.0.0:2181' -o 'docker_opt=--net=host'

This assumes that you have a local zookeeper running on host at 0.0.0.0:2181. Since it is running on host, you need '--net=host' as well.

Otherwise, if you have zookeeper running on $host:$port, following will do:

    docker volume create -d pyvolume --name zoo -o "zookeeper_string=$host:$port"

4. Run docker as usual, providing the newly created volume name.

   docker run -it -v zoo:/data  busybox:latest sh

and you can access the zookeeper znodes through /data/ after this.

Local Installation and Running

1. Look above for dependencies.

    $ make devenv
    $ source devenv/bin/activate

    $ ./devenv/bin/pyvolume --help

    usage: pyvolume [-h] [-t {sshfs,ephemeral}] [-H HOST] [-p PORT] [-m BASE]

    Arguments to volume router

    optional arguments:
    -h, --help            show this help message and exit
    -t {sshfs,ephemeral}, --driver {sshfs,ephemeral}
                            Type of driver to use
    -H HOST, --host HOST  Host to listen on
    -p PORT, --port PORT  Port to listen on
    -m BASE, --base BASE  Base directory to mount over

    $ ./devenv/bin/pyvolume
    INFO:werkzeug: * Running on http://0.0.0.0:1331/ (Press CTRL+C to quit)

Testing

Integration test.

1. Set the required environment variables.

   a. export SSH_CONFIG=/home/rprabhu/.ssh.bkp/config.server
   b. export REMOTE_PATH='server:/home/user'
   c. make itest

2. itest log - https://gist.github.com/ronin13/83d99b801202e63f07523c1c5b2be450

Unit test.

1. make test

    make test
    tox -e py37
    GLOB sdist-make: /home/rprabhu/repo/pyvolume/setup.py
    py37 create: /home/rprabhu/repo/pyvolume/.tox/py37
    py37 installdeps: -r/home/rprabhu/repo/pyvolume/requirements_dev.txt
    py37 inst: /home/rprabhu/repo/pyvolume/.tox/dist/pyvolume-0.1.0.zip
    py37 installed: You are using pip version 8.1.2, however version 9.0.0 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,alabaster==0.7.9,argh==0.26.2,Babel==2.3.4,bumpversion==0.5.3,cffi==1.8.3,click==6.6,coverage==4.1,cryptography==1.4,docutils==0.12,enum34==1.1.6,flake8==2.6.0,Flask==0.11.1,idna==2.1,imagesize==0.7.1,ipaddress==1.0.17,itsdangerous==0.24,Jinja2==2.8,MarkupSafe==0.23,mccabe==0.5.2,pathtools==0.1.2,pluggy==0.3.1,plumbum==1.6.2,py==1.4.31,pyasn1==0.1.9,pycodestyle==2.0.0,pycparser==2.17,pyflakes==1.2.3,Pygments==2.1.3,pytest==2.9.2,pytz==2016.7,pyvolume==0.1.0,PyYAML==3.11,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.4.8,tox==2.3.1,virtualenv==15.0.3,watchdog==0.8.3,Werkzeug==0.11.11
    py37 runtests: PYTHONHASHSEED='2628874551'
    py37 runtests: commands[0] | pip install -U pip
    Collecting pip
    Using cached pip-9.0.0-py2.py3-none-any.whl
    Installing collected packages: pip
    Found existing installation: pip 8.1.2
        Uninstalling pip-8.1.2:
        Successfully uninstalled pip-8.1.2
    Successfully installed pip-9.0.0
    py37 runtests: commands[1] | py.test
    ==================================================================================================== test session starts ====================================================================================================
    platform linux2 -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
    rootdir: /home/rprabhu/repo/pyvolume, inifile: tox.ini
    collected 3 items

    test_pyvolume_sshfs.py ...

    ================================================================================================= 3 passed in 0.09 seconds ==================================================================================================
    __________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________
    py37: commands succeeded
    congratulations :)

License

• Free software: MIT license

Credits

This package was created with Cookiecutter_ and the 'audreyr/cookiecutter-pypackage' project template.

• Cookiecutter: https://github.com/audreyr/cookiecutter
• audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage

pyvolume's Issues

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.