debops / debops

DebOps - Your Debian-based data center in a box

DebOps

Your Debian-based data center in a box

The DebOps project provides a set of general-purpose Ansible roles that can be used to manage Debian or Ubuntu hosts. In addition, a default set of Ansible playbooks can be used to apply the provided roles in a controlled way, using Ansible inventory groups.

The roles are written with a high customization in mind, which can be done using Ansible inventory. This way the role and playbook code can be shared between multiple environments, with different configuration in to each one.

Services can be managed on a single host, or spread between multiple hosts. DebOps provides support for different SQL and NoSQL databases, web servers, programming languages and specialized applications useful in a data center environment or in a cluster. The project can also be used to deploy virtualization environments using KVM/libvirt, Docker or LXC technologies to manage virtual machines and/or containers.

You can find out more about DebOps features on the project's documentation page.

Quick start

Start a Docker container which acts as an Ansible Controller host with DebOps support, based on Debian Buster:

docker run -it --rm debops/debops
cd src/controller ; debops run common --diff

Or, create a Vagrant VM which acts as an Ansible Controller host:

git clone https://github.com/debops/debops
cd debops && vagrant up && vagrant ssh
cd src/controller ; debops run common --diff

You can use configuration in the src/controller subdirectory to try out DebOps against the container/VM, or create your own DebOps project directory using debops project init command.

More quick start tips can be found in the DebOps quick start guide.

Installation

You can install the DebOps Python package, which includes the DebOps roles and playbooks, as well as additional scripts which can be used to setup separate project directories and run Ansible in a convenient way. To install the Python package with Ansible and other required dependencies, run the command:

pip install --user debops[ansible]

Alternatively, DebOps roles are available on Ansible Galaxy as an Ansible Collection which can be installed using the ansible-galaxy command:

ansible-galaxy collection install debops.debops

Read the installation instructions in the DebOps documentation for more details about required software and dependencies.

Getting started

Ansible uses SSH to connect to and manage the hosts. DebOps enforces the SSH security by disabling password authentication, therefore using SSH keys to connect to the hosts is strongly recommended. This can be changed using the inventory variables.

During initial deployments you might find that the firewall created by DebOps blocked you from accessing the hosts. Because of that it's advisable to have an out-of-band console access to the host which can be used to login and troubleshoot the connection.

Create a new environment within a DebOps "project directory", add some hosts in the Ansible inventory and run the default DebOps playbook against them to configure them:

# Create a new environment
debops project init ~/src/projects/my-environment
cd ~/src/projects/my-environment

# Modify the 'ansible/inventory/hosts' file to suit your needs, for example
# uncomment the local host to configure it with DebOps

# Run the full playbook against all hosts in the inventory
debops run site

# Run the common playbook against specific host in the inventory
debops run common -l <hostname>

You should read the Getting Started with DebOps guide for a more in-depth explanation of how the project can be used to manage multiple hosts via Ansible.

Development

Create a fork of this repository and clone it to your workstation. Create a development DebOps environment and symlink the forked repository in it. Now you can create new playbooks/roles in the forked repository and see their results in the development environment.

git clone [email protected]:<username>/debops ~/src/github.com/<username>/debops
cd ~/src/github.com/<username>/debops
git remote add upstream https://github.com/debops/debops.git

debops project init ~/src/projects/debops-devel
cd ~/src/projects/debops-devel
ln -s ~/src/github.com/<username>/debops debops

You can pull latest changes to the project from the upstream repository:

cd ~/src/github.com/<username>/debops
git checkout master
git fetch upstream
git rebase upstream/master

Read the development guide file for more details about the DebOps development process.

Contributing

DebOps development is done via a distributed development model. New features and changes are prepared in a fork of the official repository and are published to the original repository via GitHub pull requests. PRs are reviewed by the DebOps developer team and if accepted, are merged in the main repository.

GPG-signed git commits are preferred to ensure authenticity.

Read the contributing guide file for more details about how to contribute to DebOps.

Licensing

The DebOps project is licensed under the GNU General Public License 3.0 or later. You can find full text of the license in the LICENSES/GPL-3.0-or-later.txt file.

Some files included with the DebOps project use a different license. The licenses are marked in these files using the SPDX license identifiers and can be found in the LICENSES/ subdirectory. They are also included in the project tarballs, Ansible Collections and Python packages. The project uses the REUSE Specification and its associated tool to check and verify copyright and license information in all files.

debops's Issues

[debops.opendkim] Integration with postfix fails on ubuntu

It seems that the ansible role does not edit the /etc/default/opendkim on ubuntu when postfix is installed on the same host.

After I update the SOCKET line with SOCKET="local:/var/spool/postfix/opendkim/opendkim.sock" the integration was working.
I can fix the problem it is ok to update this file on ubuntu.

[debops.elastic_co] Proxy support in Stretch

Right now it's not possible to add APT repository GPG key when machine doesn't have direct access to the internet. Ansible apt_key module uses apt-key command which uses gpg2 in Stretch. gpg2 uses dirmngr and you can set proxy configuration in dirmngr.conf but it's not possible with apt-key (it creates temporary configuration).

This issue applies not only to this role but to many other like nodejs or nginx. Simple grep -r keyserver: | wc -l in my not so up to date configuration shows 26 occurrences.

There is a numer of potential solutions:

1. Don't use apt_key and do all keyring manipulation manually
2. Don't use id in apt_key, use data and place whole key in defaults
3. Add an variable which would allow to use data instead of id. id would be used by default (so the key would be fetched form key server) but it will be possible switch to data if needed.

Replace comma with dot in debops.users when condition

debops/ansible/roles/debops.users/tasks/resources.yml

Line 35 in a34cdca

(item.dest|d() or item,path|d()) and (item.src|d() or item.content|d()))

debops/ansible/roles/debops.users/tasks/resources.yml

Line 52 in a34cdca

(item.dest|d() or item,path|d()) and (item.src|d() or item.content|d()))

item,path|d() should be item.path|d()

global compress parameter

It could be useful to be able to globally switch off logs compression - for example when your filesystems with logs are already being compressed.

debops.console Use /etc/update-motd.d to generate motd file

According to Debian's wiki, /etc/motd may be automatically update by /etc/update-motd.d content (such script,…).

Possible security hole in debops.authorized_keys role?

I might be missing something but I am concerned that the authorized_key task "fails over" to the root user for the "user" parameter if the specified user isn't found in the passwd DB. Wouldn't this logic allow an erroneous key to give full root access?

debops/ansible/roles/debops.authorized_keys/tasks/main.yml

Line 48 in a34cdca

user: '{{ item.name if item.name in getent_passwd.keys() else "root" }}'

My apologies if I'm being dense.

Segmentation fault upon attempting local installation

On Debian 9.3 "Stretch" x86_64:

$ which pip
/usr/bin/pip
$ pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
$ pip install --user --upgrade debops --log spk-debops.log
Collecting debops
Collecting argparse (from debops)
  Using cached argparse-1.4.0-py2.py3-none-any.whl
Collecting ansible (from debops)
Collecting netaddr (from debops)
  Using cached netaddr-0.7.19-py2.py3-none-any.whl
Collecting passlib (from debops)
  Using cached passlib-1.7.1-py2.py3-none-any.whl
Collecting PyYAML (from ansible->debops)
Collecting paramiko (from ansible->debops)
  Using cached paramiko-2.4.0-py2.py3-none-any.whl
Collecting jinja2 (from ansible->debops)
  Using cached Jinja2-2.10-py2.py3-none-any.whl
Collecting cryptography (from ansible->debops)
  Using cached cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl
Collecting setuptools (from ansible->debops)
  Using cached setuptools-38.4.0-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko->ansible->debops)
  Using cached PyNaCl-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting bcrypt>=3.1.3 (from paramiko->ansible->debops)
  Using cached bcrypt-3.1.4-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyasn1>=0.1.7 (from paramiko->ansible->debops)
  Using cached pyasn1-0.4.2-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2->ansible->debops)
Collecting six>=1.4.1 (from cryptography->ansible->debops)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography->ansible->debops)
  Using cached cffi-1.11.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting enum34; python_version < "3" (from cryptography->ansible->debops)
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting idna>=2.1 (from cryptography->ansible->debops)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography->ansible->debops)
  Using cached asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography->ansible->debops)
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography->ansible->debops)
Installing collected packages: argparse, PyYAML, six, pycparser, cffi, pynacl, enum34, idna, asn1crypto, ipaddress, cryptography, bcrypt, pyasn1, paramiko, MarkupSafe, jinja2, setuptools, ansible, netaddr, passlib, debops
Successfully installed MarkupSafe-1.0 PyYAML-3.12 ansible-2.4.2.0 argparse-1.4.0 asn1crypto-0.24.0 bcrypt-3.1.4 cffi-1.11.2 cryptography-2.1.4 debops-0.6.1 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 jinja2-2.10 netaddr-0.7.19 paramiko-2.4.0 passlib-1.7.1 pyasn1-0.4.2 pycparser-2.18 pynacl-1.2.1 setuptools-38.4.0 six-1.11.0
Segmentation fault

The resulting content of spk-debops.log is here.

If this turns out to be due to pip or something else other than debops, fine to close this issue.

debops.nginx failes with lookup/ferm__fix_dependent_rules.j2 error

Hi,

running debops service/nginx the role fails with:

the template file lookup/ferm__fix_dependent_rules.j2 could not be found for the lookup

This happens on the Task: "Create default welcome page if enabled" If disabled this with nginx_webroot_create: False everything works fine.

I didn't understand yet, what leads to this error. Any Idea?

Design the GitLab CI test infrastructure using LXC

GitLab CI can be used with Docker to test roles and playbooks, but Docker containers are very limited since there's no real service management. LXC containers seem to be better suited to test DebOps environments, since they can behave like normal hosts with multiple services, an init system, etc.

There's an existing GitLab Runner LXC executor proposal which might be useful in the future. For now, DebOps could develop its own scripts used with the GitLab Runner shell executor to manage the LXC containers and run jobs in them manually.

Todo:

• Review the debops.lxc role and update it to work properly on Debian Stretch
• Develop the test scripts that prepare the LXC container test environment
• Update all the DebOps roles needed to self-host the LXC container test environment

debops.snmpd: request for more information/docs

I setup test VMs with debops.snmpd and now want to connect to them by using the generated agent account (ansible/secret/snmp/credentials/admin/username ...)

Even after disabling iptables and tcpwrappers I can't get a single snmpwalk-command return with results:

root@deb_snmp:~# snmpwalk -v3 -a SHA -A G5DcxEVCQgteStJ34ukmCyYvoDSASABWOxRifErfnw7u9F6f -X 1rOM8Zn823oEAgzxOJ9otjoR70RLqLm2w88TkdX7qX93Dbir -l authPriv -u 8540294092a75037 localhost -x AES
Timeout: No Response from localhost 

A little additional explanation would be helpful, or maybe I even hit a bug here?
I am completely new to SNMP, so it might be my lack of knowledge here.

OWASP Core Rule Set (CRS) installed by default

In order to improve security of DebOps web hosting plan, e.g. CRS support of Apache server and Ngnix server DebOps roles, implementation of ModSecurity Core Rule Set (CRS) default installation into DebOps seems appropriate.

How this benefit can be achieved - either by a new, separate DebOps role or by adding CRS to DebOps roles Apache server and Ngnix server - is not clear at the moment and needs technical clarification.

OWASP Core Rule Set (CRS) on Debian tracker:
Debian modsecurity-crs
Debian modsecurity-apache

Contacts for questions:

• Debian package maintainer:
  Alberto Gonzalez Iniesta, email: [email protected]
• ModSecurity Core Rule Set main contributor:
  Christian Folini on Github and on OWASP, email: [email protected]

Error deploying Gitlab Pages on Stretch (and potentially Jessie)

Hi team,

There seems to be an problem when deploying GitLab Pages on Stretch. But the problem may potentially appear on Jessie too.

The error occurs on task "Get gitlab-pages dependencies".
The error string is : "domain.go:16:2: use of internal package not allowed"

The problem looks to be the following: Go prevents using internal packages that are not in a subdirectory of the project, itself located in /var/local/git/gitlab-pages. Here the dependencies are stored in the parent directory of the project (i.e. /var/local/git/go).

Does it works by some of you guys ? Because, considering the problem covered above, I don't see how it can...

I would be happy to write the patch if you guys think it is relevant.

Best,
Julien

Make Python 3-only host environment a viable option

At the moment many DebOps roles install python* APT packages as needed - that is, Python 2.7 environment is used by default. However I suspect that an environment based solely on Python 3 should be possible - no Python 2.7 APT packages installed, usage of Py3 virtualenv when needed, etc.

In case that this isn't yet possible, there should be some local facts that define the default Python environment used on a host (most likely managed by a debops.python role, similar to other "environment" roles like debops.php), and other roles could use this fact to install and use Python2/3 APT packages and services.

First place to start would most likely be bootstrapping, ie. debops.bootstrap, debops.preseed, debops.lxc roles. These could install Python 3 environment by default and hopefully the rest of the roles will follow suit. The debops.python role could be executed very early in the common playbook, check the current target of the /usr/bin/python symlink and set the default Python version used on the host via Ansible local facts.

Ansible compatibility should be ensured. Since Ansible does not support Python 3.4, which is available on Debian Jessie, Python 2 should be default on older Debian releases. Python 3 can be the default on Debian Stretch onwards (Python 3.5 is available).

Design the directory structure in the repository

When everything will be in one repository, the directory structure of the current project can be rearranged to make certain things like management of documentation and tests easier. I'm currently thinking about a directory structure similar to this:

├── ansible/
│   ├── playbooks/
│   │   ├── service/
│   │   │   └── core.yml
│   │   ├── bootstrap.yml
│   │   └── site.yml
│   └── roles/
│       ├── debops.ansible_plugins/
│       └── debops.core/
├── docs/
│   └── index.rst
├── lib/
│   └── debops/
├── tests/
│   ├── common/
│   │   └── ansible/
│   │       └── inventory/
│   │           ├── group_vars/
│   │           │   └── all/
│   │           │       └── core.yml
│   │           └── hosts
│   └── lib/
│       └── run-test
├── COPYRIGHT
├── .gitlab-ci.yml
├── LICENSE
├── README.rst
└── .travis.yml

debops.nsswitch/debops.libvirt: add support for libvirt and libvirt_guest services for nsswitch hosts database

It would be nice to have built-in support for the libvirt and libvirt_guest services for the nsswitch hosts database. They're the most effort-free way of letting a hypervisor resolve hostnames of VMs that use the built-in dnsmasq-based NAT provided by libvirt.

Upstream documentation: https://libvirt.org/nss.html

Both services are provided by the libnss-libvirt package in Stretch.

A configuration I am using now to enable the libvirt service:

nsswitch__group_database_map:
    'hosts':
      - 'files'
      - 'libvirt'
      - 'dns'
      
nsswitch__group_services: [ 'libvirt' ]

It works, but overrides the default hosts map, which is more feature-rich and could change in the future. The point of including them in the role would be to remove the need for a map override, and to ensure that the libnss-libvirt package is installed when these services are enabled.

As it's up to the user to decide if they need the libvirt or the libvirt_guest service (or both) for their deployment, both should probably be disabled by default even for debops_service_libvirt hosts.

Setp "Configure TLS certificates" fail because debops.dhparam is missing

The step "Configure TLS certificates" is failing because debops.dhparam is not defined.

failed: [relpda] (item={'key': u'olcTLSDHParamFile', 'value': u''}) => {"details": "{'info': 'olcTLSDHParamFile: value #0 invalid per syntax', 'desc': 'Invalid syntax'}", "failed": true, "item": {"key": "olcTLSDHParamFile", "value": ""}, "msg": "Attribute action failed."}

I understand slapd_dhparam_file is empty and LDAP is refusing an empty value for "olcTLSDHParamFile".

[debops.postgresql_server] No way to create/move data_directory of cluster

It would be great to have some options to have the createcluster.conf modified, and then the rest of the role taking those values into consideration, or to provide a way to inject the data_directory into the pg_createcluster command (--datadir

) as well as the --logfile <path/filename>

ifupdown apply handler problem

Hello,

while testing out the tinc playbook which requires ifupdown underneath I came across the following issue which I can't seem to be able to get my head around and I suspect this is a bug in the handler script.

My testing environment is Ubuntu 16.04.3 (laravel/homestead) on the nodes built by Vagrant (see the attached Vagrantfile in case you want to reconstruct it).

The aim is to create mesh network out of 3 nodes:

all:
  children:
    debops_service_tinc:
      hosts:
        first:
        second:
        third:
    debops_service_ifupdown:
      hosts:
        first:
        second:
        third:

I'm applying the minimal set roles in my playbook in order to get the basic config up:

- role: debops.tinc/env
- role: debops.ifupdown/env
- role: debops.secret
  secret__directories: '{{ tinc__env_secret__directories }}'
- role: debops.tinc
- role: debops.ifupdown

My nodes are configured as follows (IP naturally differs):

ifupdown__host_interfaces:
  'tun0':
    type: 'tunnel'
    inet: 'static'
    addresses: [ '192.0.2.21/24' ]

And the variables I set are the following:

# Define 'elasticnet' VPN network
tinc__host_networks:
  'elasticnet': # VPN network name
    port: '655' # Default port number
    interface: 'tun0' # Align with the ifupdown__*_interfaces value
    link_type: 'static' # No interest in DHCP

# By default network 'mesh0' is created. We don't want that.
tinc__default_networks: {}

# Assume no default interfaces for ifupdown
ifupdown__default_interfaces_map: {}

# Testing on a private network
tinc__host_addresses: '{{ tinc__host_addresses_fqdn +
                          tinc__host_addresses_ip_public +
                          tinc__host_addresses_ip_private }}'

The error I'm getting is the following (I did some formatting for better readability):

RUNNING HANDLER [debops.ifupdown : Apply ifupdown configuration] ****
fatal: [first]: FAILED! => {"changed": true, "failed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.1.11 closed.\r\n",
"stdout": "...",
"stdout_lines": ["Detected interfaces to reconfigure: eth0,eth1,tun0",
  "Found active systemd ifup@ instances: eth0",
  "The 'tun0' interface is inactive",
  "Bringing down 'eth1' interface",
  "The 'eth1' interface is still active, shutting down networking service",
  "Bringing down 'eth0' interface due to desynchronized networking",
  "Bringing down '' interface due to desynchronized networking",
  "Bringing down 'eth0' interface",
  "Error: Script was working on 'eth0' network interface when it lost knowledge about the network interface state. The '/etc/network/interfaces.d/' might be desynchronized. Exiting to avoid loss of connectivity, investigate the issue."]} 

I don't think it's a correct behavior of the script if it tries to apply changes on any other interface than 'tun0' which is defined in the config file.

I'm attaching my full configuration.
tinc_basic_example.tar.gz

Steps to reproduce:

1. install vagrant
2. unpack the attachment
3. in the project folder execute

ansible-galaxy install -r requirements.yml
vagrant up
ansible-playbook -i inventory.yml site.yml

Thanks in advance.

[debops.core] Change keyserver to hkps://hkps.pool.sks-keyservers.net

Prerequisites:

• Configure Root CA for apt-key
• Ansible apt_key support

Refs:

• https://gpgtools.tenderapp.com/discussions/problems/28674-hkps
• https://www.privacy-handbuch.de/handbuch_32s.htm (German)

Ansible apt_key does not officaially allow to pass GPG options to apt-key adv but it’s source code suggests that it does not properly qoute parameters passed to apt-key adv so :) Anyway, real support would be needed.

initial debops site deploy fails on some VPS services with IPv6 enabled

ferm fails to start on some ipv6 enabled vps on various vps providers.

/var/log/syslog shows:
Starting Firewall: fermip6tables: Protocol wrong type for socket.

On the involved system i had to install a more recent kernel to get this problem solved.

debops.nginx there is no way to create a custom log_format

The current configuration does not allow changing the format of the log.
Because format must be specified before the line indicating the path to the log.

log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request_method $scheme://$host$request_uri $server_protocol" '
                '$status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;

I think I can create a variable with the right conditions.
in this file

debops/ansible/roles/debops.nginx/templates/etc/nginx/nginx.conf.j2

Lines 40 to 44 in 9222ec6

	# Logging
	access_log {{ nginx_http_access_log | default(nginx_log_path + '/access.log') }};
	error_log {{ nginx_http_error_log | default(nginx_log_path + '/error.log') }};

[debops.postgresql] postgresql_user: syntax

Hi.
I was trying to create postgresql user with

- name: Setup BARMAN db user
  become: yes
  become_user: postgres
  postgresql_user:
    state: present
    ssl_mode: prefer
    name: "{{barman_db_user}}"
    password: "{{barman_db_password}}"
    role_attr_flags: SUPERUSER,CREATEROLE,CREATEDB

This task can not be completed (if i will replay role, it will pass somehow)

But if i will use this syntax, everything goes fine.

- name: Setup BARMAN db user
  postgresql_user:
    state=present
    ssl_mode=prefer
    name={{barman_db_user}}
    password={{barman_db_password}}
    role_attr_flags=SUPERUSER,CREATEROLE,CREATEDB
  become: yes
  become_user: postgres

[debops.nullmailer] nullmailer shouldn't install itself only to be removed by postfix in the same run

Test case

My remote server is a clean Ubuntu 16.04.
Here's an excerpt from inventory/hosts

[debops_all_hosts]
EXAMPLE.domain.com

[debops_service_postfix]
EXAMPLE.domain.com

What happens

When I run $ debops initially, the nullmailer role installs nullmailer and sets up its configuration.

But a little while later, the postfix role purges nullmailer.

What I expected to happen

The nullmailer role detects that this host will have postfix installed and does not install itself.

Maybe you could check if the current host is part of debops_service_postfix?
But some systems don't directly depend on the postfix service but have it as a dependency.
Could you have postfix run before nullmailer?

debops.console remounting of /proc fails on lxc-container

If the target host is an lxc-host the common playbook fails on the role debops.console. Remounting of /proc with "hidepid" option fails with read-only filesystem error.

The solution could be - as far as I understand - to check for lxc and either don't apply the hidepid option - or don't remount the /proc. For the latter we would wait for restarting lxc.

I don't know much about ansible but I think, not to apply hidepid on lxc could be achieved by following patch:

--- main.yml.orig	2017-11-05 18:29:16.995788357 +0100
+++ main.yml	2017-11-05 18:30:34.000000000 +0100
@@ -105,9 +105,10 @@
 
   # Enable hidepid= only on specific hosts
 - include: proc_hidepid.yml
-  when: (ansible_local|d() and ((ansible_local.root|d() and 'ignore-cap12s' in ansible_local.root.flags) or
-         (ansible_local.cap12s|d() and (not ansible_local.cap12s.enabled | bool or
-          (ansible_local.cap12s.enabled | bool and 'cap_sys_admin' in ansible_local.cap12s.list)))))
+  when: ( ansible_virtualization_type not in [ "lxc", "openvz" ] and 
+        (ansible_local|d() and ((ansible_local.root|d() and 'ignore-cap12s' in ansible_local.root.flags) or
+        (ansible_local.cap12s|d() and (not ansible_local.cap12s.enabled | bool or
+        (ansible_local.cap12s.enabled | bool and 'cap_sys_admin' in ansible_local.cap12s.list))))))
 
 - name: Make sure that Ansible fact directory exists
   file:

for having "hidepid" in fstab but not remount /proc in proc_hidepid.yml we maybe instead have to ensure to restart the whole container.

But the following does not work on debian. It's remounted anyway - but what is responsible for remounting? The remount should only should match on ubuntu. But on debian it's remounted anyway - but where the remount command is executed?

--- proc_hidepid.yml.orig	2017-10-29 20:58:18.000000000 +0100
+++ proc_hidepid.yml	2017-11-05 23:03:23.103390032 +0100
@@ -24,8 +24,9 @@
     line: 'mount -o remount,hidepid={{ console_proc_hidepid_level }},gid={{ console_proc_hidepid_group }} /proc'
     insertbefore: 'exit 0'
     state: 'present'
-  when: ((console_proc_hidepid is defined and console_proc_hidepid) and
-         (ansible_distribution in [ 'Ubuntu' ] and ansible_distribution_release in [ 'trusty' ]))
+  when: (ansible_virtualization_type not in [ "lxc", "openvz" ]) and
+        ((console_proc_hidepid is defined and console_proc_hidepid) and
+        (ansible_distribution in [ 'Ubuntu' ] and ansible_distribution_release in [ 'trusty' ]))
 
 - name: Create the systemd-logind configuration directory
   file:

Similar issue exists for debops.sysctl and the shared memory options.

'i386' Dictionary Key in apt__architecture_map is unnecessary

The 'i386' key-value pair is superfluous based on this comment: "Only definitions that are different from the detected ones are listed here, otherwise the value of ansible_architecture is used."

debops/ansible/roles/debops.apt/defaults/main.yml

Line 186 in 4e91e72

'i386': 'i386'

[debops.apt_proxy] The 'python3-future' package requires 'jessie-backports' Debian repository

The debops.apt_proxy role contains a Python script which allows easy handling of unreachable APT caches, implented in 5bdb2c148c4. This script apparently relies on code provided by the python3-future APT package, which is available on Debian Jessie via the jessie-backports repository.

In certain situations, particularly initial host deployment, where the APT proxy support is enabled, playbook run can be aborted due to the python3-future package not being available when jessie-backports repository hasn't been configured yet. A solution would be to drop the python3-future from the script requirements.

Split 'debops.nginx' role into frontend and backend roles

At present, the debops.nginx role is used in multiple application playbooks (ownCloud/Nextcloud, phpIPAM, phpMyAdmin, NetBox, GitLab and more). The way it's used forces the application servers to be installed and running on the same host as the nginx server. Separating the webserver and application server into two hosts, to for example set up multiple backend intances behind one nginx central server, or even multiple load balanced nginx servers acting as proxies, is possible but requires creation of separate playbooks for different hosts.

I think that separating the existing debops.nginx role into debops.nginx_frontend and debops.nginx_backend, similar to how the MariaDB and PostgreSQL roles are designed, might be benefical. The purpose of the new roles would be:

• The debops.nginx_frontend acts as a server role. It can be installed either on the same host as the application server (current situation) or on a separate host. If the server is located on a separate host, connection to backend nginx servers (or applications that support it themselves) is protected by the proxy over TLS connections. The frontend role manages the main nginx configuration, that is /etc/nginx/nginx.conf, dynamic modules, support for Passenger, etc. The server host can support multiple separate application backends.

• The debops.nginx_backend role acts as a client role. This role is used in the playbook as a dependency for the application roles. It can detect if nginx-as-frontend is installed on the same host and back off from the main configuration, or configure a very simple and streamlined nginx instance to act as a proxy server with TLS support for the applications that don't support TLS connections directly. This role manages upstream and server configuration for a particular application, either locally or on the remote frontend server using Ansible's delegate_to mechanism. The role should support delegating to multiple hosts, so that load-balanced nginx frontend servers can have the same configuration.

I'm not sure about the specific naming yet, but I would like to avoid collisions with terms used frequently in the nginx context, like server, upstream, client, etc.

This is currently just a proposal. There are probably some issues that need to be ironed out. More extensive modification of the debops.nginx role will most likely happen after the DebOps v1.0.0 is released, so there's still time for discussion.

Create a 'debops.baseline' role

Different VPS providers offer Debian or Ubuntu base images as a starting point. Access to them is usually handled by the debops.bootstrap role which can set up an admin account, sudo access, etc.

One issue some of the VPS-provided images have is different initial services installed on the new VM. Usually this is NFS support (rpcbind) or some other services. Between the debops.bootstrap role and the DebOps common playbook + the rest of the playbooks, DebOps could provide an optional debops.baseline role which can remove known unwanted services, reset changed files, etc., providing a common starting point for the rest of the playbook between various VPS providers.

debops.librenms: template missing

TASK [debops.nginx : Create default welcome page if enabled]

[WARNING]: Unable to find 'lookup/ferm__fix_dependent_rules.j2' in expected paths.

original message: the template file lookup/ferm__fix_dependent_rules.j2 could not be found for the lookup

---

$ find ~/.local/share/debops/debops/ | grep fix_dep
~/.local/share/debops/debops/ansible/roles/debops.ferm/templates/lookup/ferm__fix_dependent_rules.j2

debops.docker fails with 'dict object' has no attribute 'search'

executing debops using the roles from the monorepo the role debops.docker fails with an error:

FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: {{
ansible_local.resolver.search if (ansible_local|d() and
ansible_local.resolver|d() and ansible_local.resolver.search|d()) else
ansible_dns.search }}: 'dict object' has no attribute 'search'"}

FYI: there are no search domains defined in /etc/resolv.conf

when statements should not include jinja2 templating delimiters

Got a warning running the gitlab playbook:

TASK [debops.redis : Divert the original Redis config files]

[WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found:
('/etc/redis/{{ item }}.dpkg-divert' not in redis__register_diversions.stdout_lines)

debops.postfix should install libsasl2-modules when using smtp_sasl_auth_enable

postfix: ['client']

should also install libsasl2-modules in a way.

Currently I can't find a way to configure debops.postfix so that the sasl-modules get installed, a grep in the debops-roles shows that only debops.saslauthd installs these modules.

IMO a postfix server with an upstream relayhost shouldn't need a local saslauthd.

Right now I define it within:

postfix__maincf:

• smtp_sasl_auth_enable: 'yes'

which might be wrong or improveable?

[debops.cryptsetup] Cannot create encrypted swap partition

Hello,

I've been trying over and over again to set up an encrypted swap partition with example provided and lots of different options, it always fails to go through.
If I only use variables from the example, it created an ext4 filesystem, looks like the swap: True has no impact on deployment at all, and when trying to specify swap mount options, then it fails stating it doesn't recognize 'swap' as type to mount (which makes sense on its own, but shouldn't happen with the swap: True option.

Am I missing something?

For extra info, I could successfully encrypt and format other partitions for ext4 filesystems.

debops.dovecot: Add support for "iterate_query"

Very simple PR incoming supporting this:
https://wiki2.dovecot.org/AuthDatabase/SQL#User_iteration

I just read about the GPG signed commits which is a very good idea, will set it up for future PRs.

[debops.postgresql_server] Fails at "Update default admin password"

file "pg-server.yml":

---

- name: Manage PostgreSQL server
  hosts: all
  become: True
  roles:
    - role: debops.postgresql_server
      tags: [ 'role::postgresql_server' ]
      postgresql_server__upstream: true
      postgresql_server__auto_backup: false
#      postgresql_server__admin_password: "{{ pg_pass }}"
      postgresql_server__trusted: [ "root", "postgres" ]
      postgresql_server__allow: [ "0.0.0.0/0" ]
      postgresql_server__listen_addresses: [ '*' ]
      postgresql_server__cluster_main:
        name: 'main'
        port: '5432'
        hba:
          - type: 'host'
            database: 'all'
            user: 'all'
            address: [ '0.0.0.0/0' ]
            method: 'md5'

run command:

ansible-playbook pg-server.yml -vvv -u root -i 192.168.1.108, -e pg_pass=123abc -t role::postgresql_server

error messages:

TASK [debops.postgresql_server : Update default admin password] ****************
task path: /etc/ansible/roles/debops.postgresql_server/tasks/secure_installation.yml:3
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 98, in run
    item_results = self._run_loop(items)
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 281, in _run_loop
    res = self._execute(variables=task_vars)
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 443, in _execute
    self._task.post_validate(templar=templar)
  File "/usr/lib/python2.7/dist-packages/ansible/playbook/task.py", line 248, in post_validate
    super(Task, self).post_validate(templar)
  File "/usr/lib/python2.7/dist-packages/ansible/playbook/base.py", line 373, in post_validate
    value = templar.template(getattr(self, name))
  File "/usr/lib/python2.7/dist-packages/ansible/template/__init__.py", line 427, in template
    disable_lookups=disable_lookups,
  File "/usr/lib/python2.7/dist-packages/ansible/template/__init__.py", line 383, in template
    disable_lookups=disable_lookups,
  File "/usr/lib/python2.7/dist-packages/ansible/template/__init__.py", line 583, in do_template
    res = j2_concat(rf)
  File "<template>", line 7, in root
  File "/usr/lib/python2.7/dist-packages/ansible/template/__init__.py", line 162, in resolve
    val = super(AnsibleContext, self).resolve(key)
  File "/usr/lib/python2.7/dist-packages/jinja2/runtime.py", line 153, in resolve
    return self.parent[key]
  File "/usr/lib/python2.7/dist-packages/ansible/template/vars.py", line 92, in __getitem__
    raise type(e)(to_native(variable) + ': ' + e.message)
AttributeError: {{ lookup('password', secret + '/credentials/' + ansible_fqdn + '/postgresql/default/' + postgresql_server__user + '/password length=' + postgresql_server__password_length + ' chars=ascii,digits,.-_~!$&()*+;=') }}: {{ secret__root + "/" + ((secret__levels + "/") if secret__levels else "") + secret__name }}: {{ inventory_dir | realpath }}: 'NoneType' object has no attribute 'startswith'
fatal: [192.168.1.108]: FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

=====

If uncomment the playbook to include this variable:

    postgresql_server__admin_password: "{{ pg_pass }}"

then I get this error:

<192.168.1.108> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/cnliou/.ansible/cp/ansible-ssh-%h-%p-%r -tt 192.168.1.108 '/bin/sh -c '"'"'sudo -H -S -n -u postgres /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rdazpzxrtfvawqsbhmxlxoixveemcdwt; /usr/bin/python /tmp/ansible-tmp-1487766984.54-82653947905827/postgresql_user.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<192.168.1.108> ESTABLISH SSH CONNECTION FOR USER: root
<192.168.1.108> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/cnliou/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.108 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1487766984.54-82653947905827/ > /dev/null 2>&1 && sleep 0'"'"''
failed: [192.168.1.108] (item=(censored due to no_log)) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

Am I doing anything wrong?
Thanks!

[debops.gitlab] manage secrets.yml with debops

according to https://gitlab.msu.edu/help/raketasks/backup_restore.md#storing-configuration-files it's important to backup gitlab's config/secrets.yml file, cause it's used to encrypt the database.

Therefore it might be good, if debops creates this file.

When updating gitlab of course the old file should be kept intact and - if possible - the current secrets saved as ansible secret

debops.postgresql grant a role only select privileges to public schema

I found the https://www.github.com/debops/debops/blob/master/ansible/roles/debops.postgresql/tasks/main.yml#L117 Grant public schema permissions task but this gives ALL privileges. Is there a way to grant a role SELECT privileges only to the public schema? This would be for a read only user I need to create.

[debops.core] Check IP address of current Ansible Controller fails on second run

When fact caching is turned on:

[ansible defaults]
remote_user = ansible
gathering = smart
fact_caching = jsonfile
fact_caching_connection = fact_cache
fact_caching_timeout = 86400

debops fails on second run on freshly installed system:

TASK: [debops.core | Check IP address of current Ansible Controller] ********** 
fatal: [dnscache2] => One or more undefined variables: 'dict object' has no attribute 'SSH_CLIENT'

FATAL: all hosts have already failed -- aborting

Clearing cache for this host solves the issue. Third and following runs complete without problems.

debops.nginx docs/defaults.rst doesn't exist

I just wanted to add something to docs and I found that the site
https://github.com/debops/debops/blob/master/docs/ansible/roles/debops.nginx/defaults.rst

doesn't exist. But https://docs.debops.org/en/latest/ansible/roles/debops.nginx/defaults.html does.

``nginx_http_access_log``
  Optional, string.
  custom access log configuration in /etc/nginx/nginx.conf
  For example to configure nginx for logging by syslog or change filenames. Overwrites nginx_log_path.


``nginx_http_error_log``
  Optional, string.
  custom error log configuration in /etc/nginx/nginx.conf
  For example to configure nginx for logging by syslog or change filenames. Overwrites nginx_log_path.

padlock always yields: gpg: WARNING: no command supplied

I have setup encfs support via debops-padlock as documented in https://docs.debops.org/en/latest/debops-tools/scripts/debops-padlock.html

Whenever I run the debops command there are some warnings from gpg displayed:

gpg: WARNING: no command supplied.  Trying to guess what you mean ...

Offer only TLSv1.2 encryption protocol by default

The TLSv1.2 protocol is the preferred encryption method in 2017 and it should be the default where possible. However, some older clients might still need to use older protocols (TLSv1, TLSv1.1) for various reasons.

To make management of offered protocols easier, the debops.pki role should set up an Ansible local fact with list of enabled encryption protocols, by default only TLSv1.2. Then, other Ansible roles can use that local fact to configure their applications. if needed, system administrators can easily enable older versions of the TLS protocol in one place, or selectively per role.

debops.cryptsetup uses `dd` option that is unsupported on macos

I'm using debops.cryptsetup from a macos controller and have found that the command to generate the encryption key on the ansible controller makes use of the dd option iflag=fullblock. This option isn't available to the default macos version of dd.

I can think of three ways to address this:

1. Remove the iflag=fullblock option. From the man page this option will accumulate full blocks of input. It seems to me this is a "nice to have" option rather than a requirement, but I'm not certain.
2. Use something other than dd. I'm thinking openssl might be a suitable alternative.
3. Determine whether the controller is darwin or linux in a prior task, then conditionally include or exclude the option.

It seems to me option 1 may acceptable, though I'm happy to explore the other options too and raise a PR.

debops.sysctrl failes because applying kernel parameters doesn't work

Applying debops.sysctrl fails on lxc-hosts (on debian stretch) because the role tries to apply the kernel parameter for shared memory:

- name: Apply kernel parameters if they were modified
  command: '{{ "sysctl --system"
               if (sysctl__register_system.stdout != "")
               else ("sysctl -e -p " + sysctl__config_file) }}'
  when: sysctl__register_config|changed

This fails because of:

sysctl -e -p /etc/sysctl.d/10-debops_sysctl.conf 
sysctl: setting key "kernel.shmall": read-only filesystem
sysctl: setting key "kernel.shmmax": read-only filesystem

We could disable either the shared memory configuration or avoid to apply the kernel parameters during runtime. Does this make sense?

diff --git a/ansible/roles/debops.sysctl/tasks/main.yml b/ansible/roles/debops.sysctl/tasks/main.yml
index 684c3253..44346581 100644
--- a/ansible/roles/debops.sysctl/tasks/main.yml
+++ b/ansible/roles/debops.sysctl/tasks/main.yml
@@ -28,7 +28,7 @@
   command: '{{ "sysctl --system"
                if (sysctl__register_system.stdout != "")
                else ("sysctl -e -p " + sysctl__config_file) }}'
-  when: sysctl__register_config|changed
+  when: sysctl__register_config|changed and ( ansible_virtualization_type not in [ "lxc", "openvz" ]

[debops.rsnapshot] Scripts in /etc/rsnapshot/{hosts} does not get regenerated

I have rsnapshot with debops servers and some external server running. There was a error with the my fqdn so the mail info in the runner script was wrong. I deleted all /etc/rsnapshot/{hosts} directories, but the does not get regenerated.

Small bug, but worth to get documented.

no action detected in task

Hi all,

So I'm doing my first steps with debops an run into the error below...
Unfortunately I couldn't find anything related in the docs.

While the error says 'gunicorn', it probably comes from elswhere, as I do not have
[debops_service_gunicorn] anywhere in my inventory.

I looked into debops/ansible/playbooks/srv/gunicorn.yml, couldn't see any errors there, though.
I commented out the role from /srv/all.yml, and got hit with the same error for debops.opendkim.

git log tells me I#m using commit 0876f86

Any ideas where to dig further?

8<- - - - -

$ debops
Running Ansible playbook from:
/home/marco/.local/share/debops/debops/ansible/playbooks/site.yml ...
ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/home/marco/.local/share/debops/debops/ansible/roles/debops.gunicorn/tasks/main.yml': line 11, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Configure Green Unicorn on older OS releases
  ^ here


The error appears to have been in '/home/marco/.local/share/debops/debops/ansible/roles/debops.gunicorn/tasks/main.yml': line 11, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Configure Green Unicorn on older OS releases
  ^ here

8<- - - - -

debops.docker fails at Configure Docker systemd options

debops.docker role fails at task Configure Docker systemd options.

TASK [debops.docker : Configure Docker systemd options] *************************************************************************************************************************
fatal: [host-01]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: {{ ansible_local.resolver.search if (ansible_local|d() and ansible_local.resolver|d() and ansible_local.resolver.search|d()) else ansible_dns.search }}: 'dict object' has no attribute 'search'"}

When I look at host-01 /etc/resolv.conf there is no search field.

root@host-01:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver xxx.xxx.xxx.xxx
nameserver xxx.xxx.xxx.xxx

How can I get the search field in the resolv.conf? I tried adding var dns_search: mydomain.com to main.yml, but resolv.conf still doesn't have the search field. I also tried adding ansible_dns.search to main.yml but that also did not work.

If I add search mydomain.com to resolv.conf manually the task works.

Version info:

ansible 2.4.2.0
python version = 2.7.12

Provide default for lxc_template_debootstrap_release

debops/ansible/roles/debops.lxc/defaults/main.yml

Line 212 in 79236c0

lxc_template_debootstrap_release: '{{ ansible_distribution_release }}'

Another item that is likely due to launching controller from within OSX.

The default behavior if no _release is specifically provided results in an empty ..._release variable and then this error, note the space before the comma.

Invalid release , valid ones are: squeeze wheezy jessie sid

lxc_template_debootstrap_release: '{{ ansible_distribution_release | default("stretch") }}'

I can submit a PR with the above change if you agree.

Next on the list is to figure out why the command below doesn't like stretch as a distro release. If anyone has any ideas.
$ lxc-create -n obj01 -t debops -f /etc/lxc/external.conf debootstrap is /usr/sbin/debootstrap Invalid release stretch, valid ones are: squeeze wheezy jessie sid lxc-create: lxccontainer.c: create_run_template: 1297 container creation template for obj01 failed lxc-create: tools/lxc_create.c: main: 318 Error creating container obj01

debops.php5 and debops.mysql roles required but missing

Hi,

today I executed

pip install --upgrade debops && debops-upgrade

afterwards running debops I ran into missing the two roles 'debops.php5' and 'debops.mysql'. After I added the missing roles into my local debops repository via ansible galaxy everything went smooth again.

debops.postfix: example for a group-related secret

I try to set use a specific sasl-sender user/pw via a postfix-lookup-table as you suggested.

It would be great to add an example to:

https://docs.debops.org/en/latest/ansible/roles/debops.postfix/defaults-detailed.html#postfix-lookup-tables

which shows how to use a file lookup with such a table that reads in a commonly used secret (= file within ansible/secret containing that password).

debops.apt Add a way to manage apt.conf.d files

Something like apt__conf, apt__group_conf and apt__host_conf to be able to add personnal configuration (e.g : to be able to remount ro partitions).