Coder Social home page Coder Social logo

ansible-docker's Introduction

DebOps logo DebOps

Your Debian-based data center in a box

GitHub CI GitLab CI CII Best Practices REUSE status RSS commits

The DebOps project provides a set of general-purpose Ansible roles that can be used to manage Debian or Ubuntu hosts. In addition, a default set of Ansible playbooks can be used to apply the provided roles in a controlled way, using Ansible inventory groups.

The roles are written with a high customization in mind, which can be done using Ansible inventory. This way the role and playbook code can be shared between multiple environments, with different configuration in to each one.

Services can be managed on a single host, or spread between multiple hosts. DebOps provides support for different SQL and NoSQL databases, web servers, programming languages and specialized applications useful in a data center environment or in a cluster. The project can also be used to deploy virtualization environments using KVM/libvirt, Docker or LXC technologies to manage virtual machines and/or containers.

You can find out more about DebOps features on the project's documentation page.

Quick start

Start a Docker container which acts as an Ansible Controller host with DebOps support, based on Debian Buster:

docker run -it --rm debops/debops
cd src/controller ; debops run common --diff

Or, create a Vagrant VM which acts as an Ansible Controller host:

git clone https://github.com/debops/debops
cd debops && vagrant up && vagrant ssh
cd src/controller ; debops run common --diff

You can use configuration in the src/controller subdirectory to try out DebOps against the container/VM, or create your own DebOps project directory using debops project init command.

More quick start tips can be found in the DebOps quick start guide.

Installation

You can install the DebOps Python package, which includes the DebOps roles and playbooks, as well as additional scripts which can be used to setup separate project directories and run Ansible in a convenient way. To install the Python package with Ansible and other required dependencies, run the command:

pip install --user debops[ansible]

Alternatively, DebOps roles are available on Ansible Galaxy as an Ansible Collection which can be installed using the ansible-galaxy command:

ansible-galaxy collection install debops.debops

Read the installation instructions in the DebOps documentation for more details about required software and dependencies.

Getting started

Ansible uses SSH to connect to and manage the hosts. DebOps enforces the SSH security by disabling password authentication, therefore using SSH keys to connect to the hosts is strongly recommended. This can be changed using the inventory variables.

During initial deployments you might find that the firewall created by DebOps blocked you from accessing the hosts. Because of that it's advisable to have an out-of-band console access to the host which can be used to login and troubleshoot the connection.

Create a new environment within a DebOps "project directory", add some hosts in the Ansible inventory and run the default DebOps playbook against them to configure them:

# Create a new environment
debops project init ~/src/projects/my-environment
cd ~/src/projects/my-environment

# Modify the 'ansible/inventory/hosts' file to suit your needs, for example
# uncomment the local host to configure it with DebOps

# Run the full playbook against all hosts in the inventory
debops run site

# Run the common playbook against specific host in the inventory
debops run common -l <hostname>

You should read the Getting Started with DebOps guide for a more in-depth explanation of how the project can be used to manage multiple hosts via Ansible.

Development

Create a fork of this repository and clone it to your workstation. Create a development DebOps environment and symlink the forked repository in it. Now you can create new playbooks/roles in the forked repository and see their results in the development environment.

git clone [email protected]:<username>/debops ~/src/github.com/<username>/debops
cd ~/src/github.com/<username>/debops
git remote add upstream https://github.com/debops/debops.git

debops project init ~/src/projects/debops-devel
cd ~/src/projects/debops-devel
ln -s ~/src/github.com/<username>/debops debops

You can pull latest changes to the project from the upstream repository:

cd ~/src/github.com/<username>/debops
git checkout master
git fetch upstream
git rebase upstream/master

Read the development guide file for more details about the DebOps development process.

Contributing

DebOps development is done via a distributed development model. New features and changes are prepared in a fork of the official repository and are published to the original repository via GitHub pull requests. PRs are reviewed by the DebOps developer team and if accepted, are merged in the main repository.

GPG-signed git commits are preferred to ensure authenticity.

Read the contributing guide file for more details about how to contribute to DebOps.

Licensing

The DebOps project is licensed under the GNU General Public License 3.0 or later. You can find full text of the license in the LICENSES/GPL-3.0-or-later.txt file.

Some files included with the DebOps project use a different license. The licenses are marked in these files using the SPDX license identifiers and can be found in the LICENSES/ subdirectory. They are also included in the project tarballs, Ansible Collections and Python packages. The project uses the REUSE Specification and its associated tool to check and verify copyright and license information in all files.

ansible-docker's People

Contributors

antoineco avatar drybjed avatar gasuketsu avatar reicru avatar routhinator avatar src386 avatar tallandtree avatar thiagotalma avatar ypid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ansible-docker's Issues

Support for iptables chain DOCKER-ISOLATION

Docker 1.10 and above implemented a new iptables chain "DOCKER-ISOLATION". With debops.ferm implementation, I need to restart the docker daemon after a firewall change managed by debops to be able to use docker networks again.

Support for DOCKER-ISOLATION is highly desired.

Error with upstream

I'm trying to install docker upstream on a few servers using the following playbook (pretty much identical to example one):

- name: Configure Backend Servers
  hosts: backend_hosts
  become: True
  vars:
     - ferm__ansible_controllers: ['0.0.0.0/0'] # Disable ferm ssh access filter
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Install apt-transport-https
      apt:
        name: apt-transport-https
        update_cache: yes
        
  roles:
    - role: "debops.etc_services"
      tags: [ 'role::etc_services' ]
      etc_services__dependent_list:
        - '{{ docker__etc_services__dependent_list }}'

    - role: "debops.ferm"
      ferm__dependent_rules:
        - '{{ docker__ferm__dependent_rules }}'
        
    - role: "debops.docker"
      docker__upstream: True
      tags: [ 'role::docker' ]

Unfortunately i run into this error:

RUNNING HANDLER [debops.ferm : Restart ferm] ***********************************
changed: [staging]
changed: [production]

RUNNING HANDLER [debops.docker : Restart docker] *******************************
fatal: [staging]: FAILED! => {"changed": false, "failed": true, "msg": "Warning: Unit file of docker.service changed on disk, 'systemctl daemon-reload' recomm
ended.\nWarning: Unit file of docker.service changed on disk, 'systemctl daemon-reload' recommended.\nJob for docker.service failed. See 'systemctl status doc
ker.service' and 'journalctl -xn' for details.\n"}
fatal: [production]: FAILED! => {"changed": false, "failed": true, "msg": "Warning: Unit file of docker.service changed on disk, 'systemctl daemon-reload' rec
ommended.\nWarning: Unit file of docker.service changed on disk, 'systemctl daemon-reload' recommended.\nJob for docker.service failed. See 'systemctl status
docker.service' and 'journalctl -xn' for details.\n"}

apt-transport-https missing

Debian 9 Stretch, ansible version:

root@debian9:~# ansible --version
ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

Playbook:

- hosts: localhost

  roles:
    - debops.docker

  vars:
    - docker__upstream: 'True'

(I use the docker__upstream var as a workaround for #51 )

Play :

root@debian9:~# ansible-playbook playbook.yml 
 [WARNING]: provided hosts list is empty, only localhost is available


PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debops.docker : Get upstream APT GPG key] ********************************
changed: [localhost]

TASK [debops.docker : Configure upstream APT repository] ***********************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException: E:Le pilote pour la méthode /usr/lib/apt/methods/https n'a pu être trouvé., W:Is the package apt-transport-https installed?, E:Impossible de récupérer https://download.docker.com/linux/debian/dists/stretch/InRelease  , E:Le téléchargement de quelques fichiers d'index a échoué, ils ont été ignorés, ou les anciens ont été utilisés à la place.
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_kXX5UG/ansible_module_apt_repository.py\", line 538, in <module>\n    main()\n  File \"/tmp/ansible_kXX5UG/ansible_module_apt_repository.py\", line 527, in main\n    cache.update()\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 464, in update\n    raise FetchFailedException(e)\napt.cache.FetchFailedException: E:Le pilote pour la méthode /usr/lib/apt/methods/https n'a pu être trouvé., W:Is the package apt-transport-https installed?, E:Impossible de récupérer https://download.docker.com/linux/debian/dists/stretch/InRelease  , E:Le téléchargement de quelques fichiers d'index a échoué, ils ont été ignorés, ou les anciens ont été utilisés à la place.\n", "module_stdout": "", "msg": "MODULE FAILURE"}
to retry, use: --limit @/root/playbook.retry

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=1

Workaround:

apt-get install apt-transport-https

systemd service template compiles incorrectly

I’m running a default setup of this role on a Debian jessie machine. All tasks run successfully. However, the docker service never starts.

systemctl status -l docker.service yields the following:

root@hannibal:~# systemctl -l status docker.service
● docker.service
   Loaded: loaded (/etc/systemd/system/docker.service; enabled)
   Active: failed (Result: exit-code) since Fri 2016-07-15 22:47:09 EDT; 42min ago
  Process: 31330 ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic (code=exited, status=1/FAILURE)
 Main PID: 31330 (code=exited, status=1/FAILURE)

Jul 15 22:47:09 hannibal docker[31330]: docker: 'daemon' is not a docker command. See 'docker --help'.
Jul 15 22:47:09 hannibal systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jul 15 22:47:09 hannibal systemd[1]: Unit docker.service entered failed state.

Digging into the /etc/systemd/system/docker.service file, I see that the script calls /usr/bin/docker daemon. This doesn’t exist for the version of Docker installed to my machine—1.6.2—and should be /usr/bin/docker -d.

Looking at the template which generates this service file, I see that it decides whether to use docker daemon or docker -d depending on the version of Docker installed. This makes sense, but doesn’t appear to be working: the version installed on my machine is 1.6.2, which is less than 1.8, so it should be using docker -d, not docker daemon. Clearly version_compare is failing to parse the version value properly.

Looking then at the place where the Docker version is registered for checking with version_compare, I see a command I can run to see what string is being passed to version_compare: dpkg-query -W -f='${Version}\n' '{{ ("docker-engine" if docker_upstream|d() else "docker.io") }}' | cut -d- -f1. Because I’m not using the upstream package, I sub in docker.io for the templated string, and get the following:

root@hannibal:~# dpkg-query -W -f='${Version}\n' 'docker.io' | cut -d- -f1
1.6.2~dfsg1

So it would appear that version_compare uses the string 1.6.2~dfsg1 to decide which script to call; though I haven’t dug any further than this, there might be something here which is causing this problem.

docker daemon failed to start on some systemd systems

Debian jessie in shipment has the following file: /lib/systemd/system/docker.service.d/custom.conf which is in conflict with /etc/docker/daemon.json

cat /lib/systemd/system/docker.service.d/custom.conf
[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=
ExecStart=/usr/bin/docker daemon $DOCKER_OPTS -H fd://
cat /etc/docker/daemon.json
{
    "graph": "/var/lib/docker",
    "hosts": [
        "fd://",
        ""
    ]
}

Therefore, the Docker service fail with the error:

 docker[25026]: time="2016-09-15T09:19:20-07:00" level=fatal msg="unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts: (from flag: [fd://], from file: [fd:// ])\n"

Stretch not detected

According to: https://docs.debops.org/en/latest/ansible/roles/debops.docker/defaults.html

docker__upstream: '{{ True
                  if (docker__distribution_release == "stretch")
                  else False }}'

if Stretch is detected, the docker upstream should be used.

However :

root@debian9:~# ansible-galaxy install debops.docker
- downloading role 'docker', owned by debops
- downloading role from https://github.com/debops/ansible-docker/archive/v0.4.1.tar.gz
- extracting debops.docker to /etc/ansible/roles/debops.docker
- debops.docker was installed successfully

root@debian9:~# cat playbook.yml 
    - hosts: localhost

      roles:
        - debops.docker

root@debian9:~# ansible-playbook playbook.yml 
[WARNING]: provided hosts list is empty, only localhost is available


PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debops.docker : Get upstream APT GPG key] ********************************
skipping: [localhost]

TASK [debops.docker : Configure upstream APT repository] ***********************
skipping: [localhost]

TASK [debops.docker : Make sure that systemd directory exists] *****************
ok: [localhost]

TASK [debops.docker : Make sure that docker.service.d directory exists] ********
ok: [localhost]

TASK [debops.docker : Remove other version if upstream is modified] ************
ok: [localhost] => (item=[u'docker-ce'])

TASK [debops.docker : Remove startup file(s) if present] ***********************
skipping: [localhost] => (item=/etc/systemd/system/docker.service.d/http-proxy.conf) 
skipping: [localhost] => (item=/etc/docker/daemon.json) 
skipping: [localhost] => (item=/etc/default/docker) 
skipping: [localhost] => (item=/lib/systemd/system/docker.service) 
skipping: [localhost] => (item=/etc/systemd/system/docker.service) 

TASK [debops.docker : Install required packages] *******************************
failed: [localhost] (item=[u'apt-transport-https', u'ca-certificates', u'curl', u'gnupg2', u'software-properties-common', u'docker.io', u'aufs-tools', u'python-pip', u'python-setuptools', u'python-virtualenv', u'virtualenv', u'bridge-utils']) => {"failed": true, "item": ["apt-transport-https", "ca-certificates", "curl", "gnupg2", "software-properties-common", "docker.io", "aufs-tools", "python-pip", "python-setuptools", "python-virtualenv", "virtualenv", "bridge-utils"], "msg": "No package matching 'docker.io' is available"}
to retry, use: --limit @/root/playbook.retry

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=1

The playbook tries to install docker.io, meaning not using the upstream.

Ansible version:

root@debian9:~# ansible --version
ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

Workaround:

  vars:
   - docker__upstream: 'True'

docker-py

Hi there,

Shouldn't this also pip install docker-py?

Jun

Permit bunch of ports on docker host

Hi, Maciej!

Thanks again for you great debops project.

I need some feature in debops.docker that I'd like to discuss. In one of your latest commits ferm configuration is defined like thit:

  - type: 'accept'
    dport: [ '{{ docker_tcp_port }}' ]
    saddr: '{{ docker_tcp_allow }}'
    accept_any: False
    weight: '50'
    role: 'docker'
    name: 'rest_api_rules'

But in my case I'd like to allow not just single docker_tcp_port but a list of ports (Swarm manager, Consul ports etc).

I certainly could redefine docker_ferm_dependent_rules completely, but this is too excessive - in my opinion.

The other way is to extend docker_tcp_port like this (roughly):

  - type: 'accept'
    dport: [ '{{ docker_tcp_port }}', '{{ rest_api_ports }}' ]
    saddr: '{{ docker_tcp_allow }}'
    accept_any: False
    weight: '50'
    role: 'docker'
    name: 'rest_api_rules'

And later I can redefine rest_api_ports in my playbook.

What do you think about this?

Support for Latest version of docker-ce

Docker split their distributions into docker-ce and docker-ee. New gpg keys are applied. If you use the old gpg keys and old package names, the latest version (docker-ce) is installed without error message.
But it would be better to add the new gpg key and latest package names. Also it should be possible to install the docker-ee version. If I've time, I'll make the changes and look at the other issues as well.

Support extra daemon.json config for storage-driver options.

Would like to be able to get the following into daemon.json:

{
  "storage-driver": "devicemapper",
   "storage-opts": [
     "dm.thinpooldev=/dev/mapper/docker-thinpool",
     "dm.use_deferred_removal=true",
     "dm.use_deferred_deletion=true"
   ]
}

But I don't see a way to either configure the storage-driver or a var to pass extra blocks to the json file.

docker__storage_driver default setting not correct for debian jessie

The default setting for docker__storage_driver does not work for Debian Jessie.

Docker daemon does not start. Error message from systemctl -l status docker:
level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."

Can be resolved by removing "storage-driver": "overlay" from /etc/docker/daemon.json.

Error at Get upstream APT GPG key

I've been trying with debops and I was stuck at installing docker. It throws gpg: keyserver receive failed: No dirmngr
The only settings I have changed are the variables associated with the sshd port. No custom playbooks or roles installed. The remote host is Debian 9.

/group_vars/all/sshd

sshd__ferm_ports:
  - 33335
sshd__ports:
  - 33335

fail2ban_jails:
  - name: 'sshd'
    enabled: 'true'
    port: 33335

hosts file:

[debops_all_hosts]
myserver ansible_host={{someip}} ansible_port=33335

[debops_service_docker]
myserver

[debops_fail2ban:children]
debops_all_hosts

I wonder if the new port has anything to do with the docker failure.
But this thread says installing dirmngr would fix the error. Does debops already install that?

TASK [debops.docker : Get upstream APT GPG key] ***********************************************************************
fatal: [myserver]: FAILED! => 
{"changed": false, "cmd": "/usr/bin/apt-key adv --keyserver hkp://pool.sks-keyservers.net --recv xxxxxxxxxxxxxxxxxxxxxxxxx", "failed": true, 
"msg": "Error fetching key xxxxxxxxxxxxxxxxxxxxxxxxxxxx from keyserver: hkp://pool.sks-keyservers.net", "rc": 2, 
"stderr": "Warning: apt-key output should not be parsed (stdout is not a terminal)\ngpg: connecting dirmngr at '/tmp/apt-key-gpghome.F2jcVDWshd/S.dirmngr' failed: IPC connect call failed\ngpg: keyserver receive failed: No dirmngr\n", "stderr_lines": ["Warning: apt-key output should not be parsed (stdout is not a terminal)", "gpg: connecting dirmngr at '/tmp/apt-key-gpghome.F2jcVDWshd/S.dirmngr' failed: IPC connect call failed", "gpg: keyserver receive failed: No dirmngr"], "stdout": "Executing: /tmp/apt-key-gpghome.F2jcVDWshd/gpg.1.sh --keyserver hkp://pool.sks-keyservers.net --recv xxxxxxxxxxxxxxxxxxxxxxxxxxx\n", "stdout_lines": ["Executing: /tmp/apt-key-gpghome.F2jcVDWshd/gpg.1.sh --keyserver hkp://pool.sks-keyservers.net --recv xxxxxxxxxxxxxxxxxxxxx"]}

'Restart docker' task raise error with v0.2.1.

Hello.

If I run playbook include debops.docker role then it raise error at 'Restart docker' task.

RUNNING HANDLER [debops.docker : Restart docker] *******************************
fatal: [ctrl]: FAILED! => {"changed": false, "failed": true, "msg": "Warning: Unit file of docker.service changed on disk, 'systemctl daemon-reload' recommended.\nWarning: Unit file of docker.service changed on disk, 'systemctl daemon-reload' recommended.\nJob for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.\n"}

I tested this with bento/debian-8.6, bento/ubuntu-16.04 vagrant boxes.

Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.