debops / debops-tools

Your Debian-based data center in a box

DebOps

Your Debian-based data center in a box

A collection of Ansible playbooks, scalable from one container to an entire data center.

DebOps is a framework

• 117+ highly extensible roles with sane defaults
• Tuned for production and works great for development
• Built for modularity so extending it is simple
• Custom scripts to tie everything together

We believe in the UNIX philosophy; one tool should only do one thing very well. DebOps has many playbooks and roles but it is just a set of focused tools to help you run and manage your infrastructure.

In fact all of the DebOps playbooks and roles can be ran with Ansible directly.

Installation

Dependencies

DebOps requires a dependency that is not already installed by Ansible. Install netaddr however you see fit:

$ pip install netaddr
$ apt-get install python-netaddr
$ yum install python-netaddr

DebOps scripts

The easiest way to install DebOps is:

$ sudo pip install https://github.com/debops/debops-tools/archive/master.zip
$ debops-update

If you want to have more control on the installation process, you can use:

$ git clone https://github.com/debops/debops-tools
$ sudo pip install ./debops-tools
$ debops-update

Please see the Installation Guide for more details.

Getting started

Here is a short intro how to use DebOps. Please have a look at the Getting Started Guide for more detailed information.

Make your first project

$ debops-init ~/myproject

Add a host to your inventory

Take a peek at ~/myproject/ansible/inventory/hosts.

Verify it

$ ssh yourhost
$ debops-task all -m setup

Run the DebOps playbooks

$ debops

What do you want to learn more about?

_

• Playbooks and roles
• Custom scripts

• DebOps guides and troubleshooting

  • Using linux containers
  • Creating a local APT server to use backports
  • Solving common problems

Do you want to contribute?

Sounds great, check out the contributing guide for the details.

Authors

Maciej Delmanowski

• Email: [email protected]
• Twitter: @drybjed
• Github: drybjed

Nick Janetakis

• Email: [email protected]
• Twitter: @nickjanetakis
• Github: nickjj

Hartmut Goebel

• Email: [email protected]
• Website: http://www.crazy-compilers.com

Robin `ypid` Schneider

• Email: [email protected]
• Website: https://me.ypid.de/
• GitHub: ypid

debops-tools's Issues

gen_ansible_cfg in bin/debops disabled

Unless I'm mistaken, the call to gen_ansible_cfg was accidentally commented out in 96e29e2 (#134). Doesn't seem relevant to the rest of the commit and it breaks generating the ansible.cfg file for me.

.debops.cfg not applied to ansible.cfg

Hello everyone,

i'm trying to use DebOps (for debops-gitlab) for one of my new playbooks. I'm on OSX and i've found an issue ( #117 ) about the problem i'm facing. The issue is the same as in it but i'm not able to use another feature of debops (which is .debops.cfg file) so i decided to open a new issue.

Basically, as mentioned in #117, i should symlink Application Support/debops folder as /usr/local/share/debops (or whatever other path that suits me) and change that setting in the .debops.cfg file. The thing is, even after changing the paths in .debops.cfg [ansible default] block, running $ debops ... command still produces wrong paths in ansible.cfg.

I've set the new paths in .debops.cfg, in ~/.Library/Application Support/debps.cfg and even in /etc/debops.cfg but debops is still using some default values i have no idea where taken from.

Have you any idea what am i doing wrong ? Any help much appreciated.

How to install into $HOME or `/opt/debops`.

For keeping the roles and playbooks up-to-date, I would prefer to install debops into $HOME or into something like /opt/debops (which is then owned by the admin-group). In the scripts I've seen some kind of preparation for this, but I could not figure out how it is meant to be used.

Please update the README and/or Makefile for how to achieve this. Thanks.

ERROR: set_fact is not a legal parameter in an Ansible task or handler

Hi guys great project!
Just installed via apt + pip (ubuntu lts 14.04) following the docs, but i get when i try to start debops in myproject dir.
What kind of problem could it be?

ERROR: set_fact is not a legal parameter in an Ansible task or handler

OSX - debops command fails with Ansible from Homebrew

On Mac OS X, the command debops seems to work only when Ansible is installed from pip.
With Ansible installed through Homebrew, this is the error:

> debops
Traceback (most recent call last):
  File "/usr/local/bin/debops", line 36, in <module>
    import ansible
ImportError: No module named ansible

It works when uninstalling from Homebrew and use pip instead, but I wonder, is there any way around this, without installing Ansible from pip?

boostrap_ansible does not work on recent ubuntu: E: ansible: depends-on-obsolete-package depends: python-support (>= 0.90) => use dh_python2 instead

when trying to build ansible:

dh_perl -pansible 
dh_shlibdeps -pansible    
dh_gencontrol -pansible  
dpkg-gencontrol: warning: package ansible: unused substitution variable ${python:Versions}
dpkg-gencontrol: warning: package ansible: unused substitution variable ${python:Depends}
# only call dh_scour for packages in main
if grep -q '^Component:[[:space:]]*main' /CurrentlyBuilding 2>/dev/null; then dh_scour -pansible ; fi
dh_md5sums -pansible 
dh_builddeb -pansible 
dpkg-deb: building package `ansible' in `../ansible_1.9.0-0.git201503252207~unstable_all.deb'.
 dpkg-genchanges -b >../ansible_1.9.0-0.git201503252207~unstable_amd64.changes
dpkg-genchanges: binary-only upload (no source code included)
 dpkg-source -I --after-build ansible-1.9.0
dpkg-buildpackage: binary-only upload (no source included)
Now running lintian...
E: ansible changes: bad-distribution-in-changes-file unstable
W: ansible: copyright-refers-to-versionless-license-file usr/share/common-licenses/GPL
W: ansible: copyright-without-copyright-notice
W: ansible: description-synopsis-starts-with-article
E: ansible: depends-on-obsolete-package depends: python-support (>= 0.90) => use dh_python2 instead
Finished running lintian.
#############################################
Ansible DEB artifacts:
deb-build/unstable/ansible_1.9.0-0.git201503252207~unstable_amd64.changes
#############################################

I expected it to work on my ubuntu 15.04 because of the comment in the file:

# bootstrap-ansible.sh: download and build Ansible on Debian/Ubuntu host

debops-update : no such file or directory

I've been following through the the installation instructions here.

When I get to debops-update I get the output below. I'm a bit naive to this, I tried creating the directory mentioned in the second line, but I get the same error, looks like something to do with the install_path

---

levi@debianOffice:~$ debops-update
DebOps playbooks have not been found, installing into /home/levi/.local/share/debops/debops-playbooks

Traceback (most recent call last):
File "/usr/local/bin/debops-update", line 220, in
main(args.project_dir)
File "/usr/local/bin/debops-update", line 201, in main
clone_git_repository(PLAYBOOKS_GIT_URI, 'master', install_path)
File "/usr/local/bin/debops-update", line 129, in clone_git_repository
repo_uri, destination])
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(_popenargs, *_kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception

OSError: [Errno 2] No such file or directory

debops-update fails

debops-update is failing between 9/76 and 15/76

so, I dont know, whether github is to blame for the "timeout" or not.
I was pinging github in parallel and got roundtriptimes from 99 to 104 sec on this 25Mbit/s cable connection. so, the network-path seems to be ok.
I had this error on different occasions, but today, it's not giving me one successful update out of 20 attempts within an hour.
even edited the debops-update script to use http instead of https didn't change anything.

I get tow diff. errormsg.:

debops-update 
DebOps playbooks have been found in /home/guenter/.local/share/debops/debops-playbooks

Updating https://github.com/debops/ansible-apt [master] (1/76)
Updating https://github.com/debops/ansible-apt_preferences [master] (2/76)
Updating https://github.com/debops/ansible-auth [master] (3/76)
Updating https://github.com/debops/ansible-backporter [master] (4/76)
Updating https://github.com/debops/ansible-bootstrap [master] (5/76)
Updating https://github.com/debops/ansible-boxbackup [master] (6/76)
Updating https://github.com/debops/ansible-console [master] (7/76)
Updating https://github.com/debops/ansible-debops [master] (8/76)
Updating https://github.com/debops/ansible-dhcpd [master] (9/76)


fatal: unable to access 'https://github.com/debops/ansible-dhcpd/': gnutls_handshake() failed: Error in the pull function.
fatal: ambiguous argument 'FETCH_HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/usr/local/bin/debops-update", line 220, in <module>
    main(args.project_dir)
  File "/usr/local/bin/debops-update", line 212, in main
    fetch_or_clone_roles(roles_path, GALAXY_REQUIREMENTS)
  File "/usr/local/bin/debops-update", line 121, in fetch_or_clone_roles
    update_git_repository(destination_dir)
  File "/usr/local/bin/debops-update", line 147, in update_git_repository
    fetch_sha = subprocess.check_output(['git', 'rev-parse', 'FETCH_HEAD']).strip()
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['git', 'rev-parse', 'FETCH_HEAD']' returned non-zero exit status 128

and

...
Updating http://github.com/debops/ansible-etc_services [master] (15/76)
fatal: unable to access 'https://github.com/debops/ansible-etc_services/': Failed to connect to github.com port 443: Connection timed out
fatal: ambiguous argument 'FETCH_HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/usr/local/bin/debops-update", line 221, in <module>
    main(args.project_dir)
  File "/usr/local/bin/debops-update", line 213, in main
    fetch_or_clone_roles(roles_path, GALAXY_REQUIREMENTS)
  File "/usr/local/bin/debops-update", line 122, in fetch_or_clone_roles
    update_git_repository(destination_dir)
  File "/usr/local/bin/debops-update", line 148, in update_git_repository
    fetch_sha = subprocess.check_output(['git', 'rev-parse', 'FETCH_HEAD']).strip()
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['git', 'rev-parse', 'FETCH_HEAD']' returned non-zero exit status 128

Vagrantfile controller bootstrap

I've been attempting to come up with a Vagrantfile that can provide a bootstrapped compatible controller using just your playbooks but there is still a bunch of work to be done (hashicorp/vagrant#3396) to make everything "just-work".
Ideally, Ansible + Vagrant will be cross platform compatible with a local Ansible internally (https://github.com/podarok/vansible) & (hashicorp/vagrant#2103) but that isn't happening just yet. I've bootstrapped a vagrant host via shell provisionment: "apt-get virtualenv python-dev" => "virtualenv --easyinstall ansible" => "source ansible/bin/activate && pip install --upgrade ansible" and been using that.

So far, I've been playing around with this for quite some time trying to come up with a decent means to create a local controller via bootstrap.yml since the example Vagrantfile below produces a couple different unattended consequences out of the box. I'm just throwing this out there to get some thoughts on how to do this elegantly as I think having a local virtual machine locked down with correct cryptography as a controller would be ideal.

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "chef/debian-7.6"
  config.ssh.forward_agent = true
  config.vm.define "local_controller" do |controller|
    controller.vm.provision "ansible" do |ansible|
      ansible.playbook = "bootstrap/debops-playbooks/playbooks/site.yml"
      ansible.groups = {
        "ansible_controllers" => "local_controller"
      }
    end
  end
end

Produces a secret directory structure out of the intended $PWD tree:

 tree .vagrant/
.vagrant/
├── machines
│   ├── controller
│   │   └── virtualbox
│   │       ├── action_provision
│   │       ├── action_set_name
│   │       ├── id
│   │       ├── index_uuid
│   │       └── synced_folders
│   ├── default
│   │   └── virtualbox
│   └── local_controller
│       └── virtualbox
└── provisioners
    └── ansible
        ├── inventory
        │   └── vagrant_ansible_inventory
        └── secret
            └── pki
                ├── ca
                │   ├── certs
                │   └── crl
                ├── hosts
                │   └── packer-debian-7.6-amd64
                │       ├── crl
                │       ├── csr
                │       │   └── packer-debian-7.6-amd64.csr
                │       └── signed
                └── wildcard
                    ├── certs
                    ├── crl
                    └── private

24 directories, 7 files

Being able to destroy the local controller and bring back up a fresh one while retaining all cryptography (GPG, monkeysphere, x.509 PKI chain, Vault, EncFs) would be rather useful.

debops-task script traceback

Looks like something is missing.

drybjed@helios ~/src/projects/helios/ % debops-task test -m setup
Traceback (most recent call last):
  File "/usr/local/bin/debops-task", line 5, in <module>
    pkg_resources.run_script('debops==0.1.0', 'debops-task')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1401, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/debops-0.1.0-py2.7.egg/EGG-INFO/scripts/debops-task", line 45, in <module>

TypeError: find_up() takes exactly 2 arguments (1 given)

Although, debops-task script is not essential - after one run of debops, Ansible configuration file is generated and you can use ansible command directly. So I suppose we could just remove debops-task and not worry about it.

debops v0.2.0 fails to run on Yosemite

$ debops
Traceback (most recent call last):
File "/usr/local/bin/debops", line 162, in
main(sys.argv[1:])
File "/usr/local/bin/debops", line 97, in main
config = read_config(debops_root)
File "/Library/Python/2.7/site-packages/debops/config.py", line 60, in read_config
configfiles = _configfiles + [os.path.join(debops_root, DEBOPS_CONFIG)]
TypeError: unsupported operand type(s) for +: 'NoneType' and 'list'

That is from a dir with a .debops.cfg, ansible.cfg is not created.

Ansible Best Practices

http://docs.ansible.com/playbooks_best_practices.html

Having DebOps repositories follow a 'develop/stage/production' git branching system would be rather ideal. It would also allow debop-init to structure inventories accordingly and use the repository branch to determine the hosts to run against.

ansible-playbook cannot be interrupted with Ctrl-C

When debops passes control to ansible-playbook, pressing Control+C does not stop the playbook execution. Is there something we could do to fix this?

network configuration for servers with static ips

Currently, ifupdown only configures network, if there is no mention of the word static in /etc/network/interfaces
Often, root servers have a static configuration provided by the hosting provider.
With such a setup, setting the server up as a kvm host will fail with surprising errors.
First, no br2 interface is created.
I am not sure, what is the right location to fix that.
It depends on what is a an assumption to which a server has to apply.
one could make ifupdown fail if there is static configuration.
Then one could deactivate ifupdown.
Then subnetwork could fail if ifupdown fails.
Or something like that.

Can't overwrite default inventory hosts file in .debops.cfg

When trying to overwrite the default hostfile var in .debops.cfg, after generating ansible.cfg the value is still the default value:

Contents of .debops.cfg:

[ansible defaults]
hostfile = ./provisioning/vagrant_ansible_inventory

Value of hostfile in generated ansible.cfg:

# Ansible configuration file generated by DebOps, all changes will be lost.
# You can manipulate the contents of this file via `.debops.cfg`.

[defaults]
(other vars)
hostfile = /Users/daniel/devel/dvigueras/vagrant-debops/ansible/inventory
(other vars)

I've found that if I delete line 84 in debops: https://github.com/debops/debops/blob/master/bin/debops#L84 the file gets generated correctly:

# Ansible configuration file generated by DebOps, all changes will be lost.
# You can manipulate the contents of this file via `.debops.cfg`.

[defaults]
(other vars)
hostfile = ./provisioning/vagrant_ansible_inventory
(other vars)

And even ansible works, but debops isn't able to connect to the vm:

$ ansible -m ping all
vagrantdebops | success >> {
    "changed": false,
    "ping": "pong"
}

$ debops-task -m ping all
No hosts matched

Best Practise Configuration (or: how to use 'playbooks-paths'?)

For a while I'm trying to find the best way how to easily extend DebOps with my custom roles. However, I'm failing to find a satisfying configuration.

So far my approach was to create a custom <project>/ansible/playbooks/site.yml where I include the upstream site.yml at the end. The dirty hack in this configuration is that the upstream site.yml has to be given with a concrete path. E.g.

  # This playbook contains the debops setup.
- include: /var/lib/debops/.local/share/debops/debops-playbooks/playbooks/site.yml

Is there a variable which can be used in the YAML to get the upstream playbooks path?

Problem: Using playbooks-paths
When looking through the documentation I found that there is a playbooks-paths variable which is documented as:

List of comma-separated paths where playbooks can be found. debops script will search these
paths looking for playbooks to execute.

So I tried to move my custom site.yml to a separate path which is listed in playbooks-paths (btw. the parsing only works when the definition is newline separated, not comma separated):

playbooks-paths: ~/my-playbooks
                 %(install-path)s/playbooks

The problem now is, that because the site.yml is first found in my custom location, the roles_paths is expanded to only include role directories relative to this playbook path. This makes it impossible to run upstream roles.

So I deleted the site.yml from my custom directory, which now fixes the role lookup. But no custom playbooks are run anymore, what defeats the purpose of the configuration.

Problem: Expanding roles_paths
Alternatively I tried to add the %(install-paths)s/roles path to the roles_paths variable in .debops.cfg which resulted in this error:

Traceback (most recent call last):
  File "/usr/local/bin/debops", line 174, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/bin/debops", line 111, in main
    config = read_config(project_root)
  File "/usr/local/lib/python2.7/dist-packages/debops/config.py", line 113, in read_config
    for sect in cfgparser.sections())
  File "/usr/local/lib/python2.7/dist-packages/debops/config.py", line 113, in <genexpr>
    for sect in cfgparser.sections())
  File "/usr/lib/python2.7/ConfigParser.py", line 655, in items
    for option in options]
  File "/usr/lib/python2.7/ConfigParser.py", line 691, in _interpolate
    self._interpolate_some(option, L, rawval, section, vars, 1)
  File "/usr/lib/python2.7/ConfigParser.py", line 723, in _interpolate_some
    option, section, rest, var)
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
        section: [ansible defaults]
        option : roles_path
        key    : install-path
        rawval : /roles

Obviously %(install-path) cannot be correctly expanded when used under [ansible defaults].

Questions:
I don't really understand how people are using these configuration statements.

• Is anyone else having problems with this?
• How are you managing multiple projects which are using common code (playbooks, roles) outside of DebOps?
• Is someone using playbooks-paths?
• Is anything wrong with always adding the upstream roles location to the roles_paths?

Add system-wide configuration file

/etc/debops/debops.cfg should be used as the default system-wide configuration file. It could specify an address and branch of default debops-playbooks git repository, which could allow to easily install playbook and roles from different repositories instead of the official one.

Create an alias that redirects messages to /dev/null

e.g blackhole or devnull

This is already possible by setting
postfix_local_aliases: { 'blackhole': ['/dev/null'] }

but it would be nice to have by default

Refactor repo and rename to debops-tools.

I propose the name "DebOps Tools" for this repository to avoid misunderstanding what this repo is about. Maybe a new debops/debops repo can be created as a landing page and general issue tracker.

Include fails when done from hooks

Hi,

I want to split functionality used in hooks into several files (hook called from debops.users). For that I would like to use include from hook.
But it doesn't work this way. I extracted smallest possible example to reproduce problem: https://www.dropbox.com/s/fjc1m2n6qwtkqbs/debops-test.zip?dl=0

Short description (in case it will be problems with download):
Have two playbooks (test_fail, test_success) and hook spitted into two files (post_main, prezto).

First (test_fail) done the way it is used in all DebOps playbooks will not print Success message - fail. Second (test_success) work fine, but I don't understand why and unfortunately it is not how it done in debops, so I can't use it. Can you please give any suggestion about why first case fail and how can I split playbook called from debops hook? Thanks!

test_fail.yml:

---

- hosts: localhost
  tasks:
    - name: DebOps post_tasks hook
      include: "{{ lookup('task_src', 'users/post_main.yml') }}"

test_success.yml:

---

- hosts: localhost
  tasks:
    - name: DebOps post_tasks hook
      include: "{{ p_path }}"
  vars:
    p_path: "{{ lookup('task_src', 'users/post_main.yml')}}"

post_main.yml:

---

- name: Debug
  debug: msg="{{ lookup('task_src', 'users/prezto.yml') }}"

- name: prezto
  include: "{{ lookup('task_src', 'users/prezto.yml') }}"

prezto.yml:

---

- debug: msg="Success"

Port ansigenome readme to RST

Use SPDX license identifiers

Ref: ansible/galaxy-issues#169 (not yet approved)

I am already doing this, ref:

• https://github.com/debops/ansible-cryptsetup/blob/master/meta/main.yml
• https://github.com/debops/ansible-apt_cacher_ng/blob/master/meta/main.yml

And Galaxy does not have any problem with it.

bootstrap-ansible.sh does not work with recent tags (v1.9.4-1)

I executed it like this:

./misc/scripts/bootstrap-ansible.sh v1.9.4-1

The "-1" in the version mixup the deb install, and nothing get installed.

Encrypted storage for secrets at rest

Problem

At the moment secret data is stored in inventory.secret/ directory inside the project directory (relative to the Ansible directory, location can be changed using debops.secret role). But this data is stored in plaintext, and can be easily accessed during normal work on a host, even if user has encrypted home directory with eCryptfs, created automatically by many distributions.

Possible solution

We could offer an encrypted directory using EncFS, with encryption key saved in a file encrypted using GnuPG:

• at rest, data is stored encrypted in inventory.secret.encfs/ directory
• when debops script starts and finds encrypted directory, it tries to decrypt it using gpg command (GnuPG is useful because we can encrypt the key for multiple administrators at the same time, each one using his own GPG key, no problem with sharing encrypted data between them)
• when EncFS-encrypted directory is "opened", ansible-playbook takes over and runs the playbook
• when playbook run is finished, trap in debops script unmounts the encrypted directory with secrets encrypted again

Issues

• EncFS encryption is potentially insecure - but the reason to use it is to protect data at rest, we should expect that underlying filesystem is encrypted anyway with different encryption method, LUKS, eCryptfs
• encrypted secrets cannot be used easily "raw" with ansible-playbook directly - but this is a debops only feature, and optional, so if someone does not needed it, he/she is not forced to use it
• why not just use ansible-vault? At the moment we cannot generate random passwords with it, storing files securely or transferring them via Ansible controller to other hosts with ansible-vault is cumbersome borderlining on impossible

Questions, other suggestions, comments?

mysql configuration fails when hostname has a capital letter

For some reason configuring mysqld does not work when the hostname is not all lower case.
The work-around is to edit /etc/hostname and /etc/hosts to remove capital letters.
This may very well be a in bug in Debian's mysql setup. But I file it here so that it can be tracked.

Refactor Python modules

Refs:

• https://packaging.python.org/

Use {{ ansible_managed }} instead of 'This file is managed by Ansible, all changes will be lost'?

What do you think?

Currently it is only used in a few places.

Is your debops.plan related to this?

Change "his program" to "This Program" in all roles.

The problem from debops/debops-playbooks#198 seems to be present in all roles.

ag '\bhis program' -a

I would recommend to fix this in using sed. #120

Look for global debops.cfg in /etc

Can we make debops look for /etc/debops.cfg on all systems that aren't windows? It is a standard place for global information. On OSX it ONLY looks in /Library/Application Support...

Instructions in Getting Started don't work, path to 'core' missing?

I'm following the instructions in Getting Started but I'm hitting this error when I run debops

Running Ansible playbook from:
/home/user/.local/share/debops/debops-playbooks/playbooks/site.yml ...
ERROR: cannot find role in /home/user/.local/share/debops/debops-playbooks/playbooks/roles/debops.core or /home/user/.local/share/debops/debops-playbooks/playbooks/debops.core or /etc/ansible/roles/debops.core

Here's the contents of playbooks/

app  callback_plugins  env  filter_plugins  hw  library  lookup_plugins  net  service  srv  sys  tools  virt  app.yml  bootstrap.yml  common.yml  core.yml  env.yml  hw.yml  net.yml  site.yml  srv.yml  sys.yml  virt.yml

Anything I can do to fix this?

make misc/scripts/bootstrap_ansible more proxy friendly

Currrently, it uses

project="git://github.com/ansible/ansible.git"

which is not friendly for people behind a proxy, because setting up git to use the git protocol via a proxy is much more painful than setting it up to use HTTP through a proxy.

The HTTP URL seems to work well, so I suggest to use that instead.

Debops system-wide installation option(s)

DebOps used to do a system-wide installation in to /usr/local. Since v0.2.0, it no longer does and instead installs in $HOME/.local. This might work well when you have a single person working with DebOps, but once you have more than one, the system-wide installation is required.

A suggestion I have is to have debops default to using a global /etc/debops.conf file whose location can be overridden by an environment variable, or a command line arg... or both. In there you can set what the prefix is for the installation path. That way you can support user and system-wide installations.

debops fails common task: install 'openntpd' --no-install-recommends' failed: /etc/openntpd/ntpd.conf: Permission denied

In ubuntu 14.01 ntpd.yml task fails lunching debops playbook.
I think is due to this bug: https://bugs.launchpad.net/ubuntu/+source/openntpd/+bug/458061
I solved the problem as written in comment bug 24 running commands not in the same order but still...

passlib must be installed to encrypt vars_prompt values

I don't know why running

TASK: [debops.console | Enforce root password]

pass lib was not installed on the node but running:

sudo pip install passlib

solved the problem.
Btw, I'm a bit afraid what will be doing the task.

su doesn't require password

Hi there,

From an admin/sudo user, if you type "su" and hit enter it drops you straight into a root shell.

Feature or bug?

Ubuntu Studio Trusty installation fails

I am running into a mess of Python 3.4 related issues with Ubuntu Studio Trusty when installing debops

had a much better time on mainline Trusty

am thinking of running a VirtualBox of Jessie as a bastion server just to get this thing to run !

cheers

-N

Wrapping ansible, abstracting configuration, and making things even more useful

This issue is intended to continue the discussion here: https://github.com/debops/ansible-dnsmasq/pull/20#issuecomment-160771596

I read the wiki page and liked the idea

The ideas I really liked about the approach of DebOps is the wrapping of ansible, so that it, with only a few system requirements, can be used in a virtualenv, with everything needed to configure an inventory contained entirely in a directory. It seemed that the scripts seem to be too tied into the default debops playbooks and roles, and I really wanted to take the wrapping idea further, and abstract how the playbooks and roles were cloned. I started a project called Demosthenes, to wrap ansible in a similar way using a demos command. Right now, I have just been rewriting parts of debops and ripping out apple and windows controller support, since I have no need for it. I'm just in a bit of a hurry, and need to convert all the salt states and formulae into roles and playbooks, using sensible roles already in the galaxy if appropriate.

Ansible really solves a couple of long standing problems I have experienced with using salt. However, with salt, since the minions were continuously running root processes subscribing to a service, the issue of privilege escalation didn't need to be confronted. I am very uncomfortable with a line like this in /etc/sudoers:

root@pokey:~# cat /etc/sudoers.d/admins 
Defaults: %admins env_check += "SSH_CLIENT"
%admins ALL = (ALL:ALL) NOPASSWD: SETENV: ALL

I would rather have a script in /usr/local/bin that adds the above file at the beginning of a playbook run, then removes it upon end or error. Alternatively, using the --ask-sudo-pass(I might be dyslexic here) option in a local config would be very useful. I don't have the same user password on every machine. Having it to where I can login remotely and be root without a password is not something I really want to do.

I also had this principle, when designing my salt states, that was basically there was no state after the bootstrap. If a machine was already bootstrapped, it could be included in the inventory and nothing would happen to it until variables were set that matched the machine. The default state was a meta-state where you could just include pieces from it, and still be able to configure other services and states. I couldn't find a way to separate (common/core)(I don't remember which) into pieces.

I ramble on, but I created, many years ago, and have kept going, a fully automated network install system, paella. The original code is on sourceforge, but I moved to berlios since they provided subversion support early. I have been doing a lot of netboot installs over the years, and I have a pyramid webserver that provides the preseeds from a mako template. The preseed bootstraps salt and an initscript starts a state run on reboot, then removes the boot script. I made a video, that may be a bit boring, but it will give you an idea of what I've been working with.

Anyway, I've decided to use ansible instead of salt, and the debops way of wrapping ansible and using a local configuration is really nice. Also, I really think that making a separate ansible wrapper that is a bit more agnostic about how things are laid out would be really great. If there is an easy way that I'm missing to perform some of this using how things currently exist in debops, please let me know. Also, I named Demosthenes, who was a famous orator from Greeze, from Valentine Wiggin of fictional variety that actually used an ansible.

Which .ansigenome.conf is used?

https://github.com/debops/debops/tree/master/misc/ansigenome/templates contains the templates but I am unable to reproduce the README.md because ansigenome_info is in meta/ansigenome.yml.

Related to nickjj/ansigenome#23

Consider using requirements.yml and with ansible-galaxy to handle roles

Newer versions of ansible have a nice requirements format, making the debops-update script more flexible.

I made a function here making use of it: https://github.com/umeboshi2/demosthenes/blob/master/demosthenes/scripts/demos_update.py#L103

A potential downside is that roles in version control are tarballed and extracted, rather than cloned into the roles_path.

I took the time to make a simple conversion of the galaxy requirements files in the debops-playbooks repo. debops/debops-playbooks#217

Check code signatures

With the debops-keyring and the DebOps Code Signing Policy in place. The DebOps Tools should bootstrap and check signatures using the keyring.

Benefits:

• After initial trust is established on first debops update run, GitHub is no longer part of the TCB.

Depends on: #165

Missing python-passlib dependency

Hi there,

By default the pip install doesn't bring in python-passlib which is required to generate the root password for machines.

Thanks,
~ B

running common tasks on a node denied other hosts to login on the server

I ran debops on a test server and finally finished all task with no errors, but my colleagues where suddenly denied to access the server.
I like the harding debops made, and make sense to allow only controllers host to log as root but how I can re-enable access to my colleagues without having them to run debops on their machine as controllers. They are still afraid about it.

Core Infrastructure Initiative (CII) Best Practices badge

Refer to: https://twit.tv/shows/floss-weekly/episodes/389 and https://bestpractices.coreinfrastructure.org

I am currently going thought the questions.

Project URL: https://bestpractices.coreinfrastructure.org/projects/237

Move `debops.keyring` Python module into this repo

running debops as per the documentation does not work

I followed the documentation on http://docs.debops.org/en/latest/getting-started.html#your-first-project

I cloned the debops repository and did a python setup.py install --user. debops-update seems to run fine. But I needed to patch GIT_GIT_URI = GIT_URI, because I am behind a corporate proxy.

I.e.

debops-init /tmp/test-debops/
cd /tmp/test-debops
cat >  ansible/inventory/hosts <<EOF
[gitlab]
vagrant_gitlab ansible_ssh_host=192.168.121.59 ansible_ssh_private_key=~/.vagrant.d/insecure_private_key ansible_ssh_user=vagrant
EOF
debops-task all -m setup   # works fine
mkdir ansible/inventory/host_vars/vagrant_gitlab
cat > ansible/inventory/host_vars/vagrant_gitlab/vars.yml <<EOF
# Set custom timezone on the server
ntp_timezone: 'Europe/Paris'

# Protect the SSH service by specifying list of hosts/networks which can
# access it (by default access is allowed from anywhere, but firewall blocks
# too many connection attempts in a short amout of time)
sshd_host_allow: [ '192.168.178.0/24' ]

# Specify a mail server to send all mail through (it needs to accept the
# incoming messages from your host)
postfix_relayhost: 'mail.intern.example.com'

# Set a default admin e-mail address where all messages to root account will
# be forwarded
postfix_default_local_alias_recipients: [ '[email protected]' ]
EOF

Currently, the readme states:

Run the DebOps playbooks

$ debops

When I do that, I get

>env http_proxy= https_proxy= debops
Running Ansible playbook from:
/home/muelli/.local/share/debops/debops-playbooks/playbooks/site.yml ...
ERROR: set_fact is not a legal parameter in an Ansible task or handler
>

I expected to be able to run debops as per the documentation.

I'm running ansible 1.7.2.

ansible/inventory/group_vars/all/apt_lxc.yml must be stored as a dictionary/hash

I get this error when running a playbook on a freshly created debops project.
This file currently contains only comments. Obviously Ansible does not like this. I'm afraid, the only way to avoid this error message is to remove the file.

mkdir /tmp/testtesttest
cd /tmp/testtesttest
debops-init
echo localhost >> ansible/inventory/hosts
cat > site.yml <<EOF

---
- gather_facts: false
  hosts: localhost
  tasks:
  - name: Check if host is in group
    command: echo "yes"
EOF
debops ./site.yml

installation on Trusty

before I bork my re-installed mainline Trusty laptop again

what should I be doing with pip install of debops?

I am a bit confused, because I am not sure the instructions are consistent

in some places sudo is used

on this you indicate use sudo https://github.com/debops/debops

on this, no clue, but I assumed no sudo

niccolox@trustyinx:~/Projects|⇒ pip install debops
Downloading/unpacking debops
Downloading debops-0.4.3.tar.bz2
Running setup.py (path:/tmp/pip_build_niccolox/debops/setup.py) egg_info for package debops

Downloading/unpacking netaddr (from debops)
Downloading netaddr-0.7.18-py2.py3-none-any.whl (1.5MB): 1.5MB downloaded
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7 (from debops)
Installing collected packages: debops, netaddr
Running setup.py install for debops
changing mode of build/scripts-2.7/debops from 664 to 775
changing mode of build/scripts-2.7/debops-defaults from 664 to 775
changing mode of build/scripts-2.7/debops-init from 664 to 775
changing mode of build/scripts-2.7/debops-padlock from 664 to 775
changing mode of build/scripts-2.7/debops-task from 664 to 775
changing mode of build/scripts-2.7/debops-update from 664 to 775
error: could not create '/usr/local/lib/python2.7/dist-packages/debops': Permission denied
Complete output from command /usr/bin/python -c "import setuptools, tokenize;file='/tmp/pip_build_niccolox/debops/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-w3JbU6-record/install-record.txt --single-version-externally-managed --compile:
running install

running build
running build

running build_py

creating build

creating build/lib.linux-x86_64-2.7

creating build/lib.linux-x86_64-2.7/debops

copying debops/config.py -> build/lib.linux-x86_64-2.7/debops

copying debops/init.py -> build/lib.linux-x86_64-2.7/debops

creating build/lib.linux-x86_64-2.7/debops/cmds

copying debops/cmds/init.py -> build/lib.linux-x86_64-2.7/debops/cmds

running build_scripts

creating build/scripts-2.7

copying and adjusting bin/debops -> build/scripts-2.7

copying and adjusting bin/debops-defaults -> build/scripts-2.7

copying and adjusting bin/debops-init -> build/scripts-2.7

copying and adjusting bin/debops-padlock -> build/scripts-2.7

copying and adjusting bin/debops-task -> build/scripts-2.7

copying and adjusting bin/debops-update -> build/scripts-2.7

changing mode of build/scripts-2.7/debops from 664 to 775

changing mode of build/scripts-2.7/debops-defaults from 664 to 775

changing mode of build/scripts-2.7/debops-init from 664 to 775

changing mode of build/scripts-2.7/debops-padlock from 664 to 775

changing mode of build/scripts-2.7/debops-task from 664 to 775

changing mode of build/scripts-2.7/debops-update from 664 to 775

running install_lib

creating /usr/local/lib/python2.7/dist-packages/debops

error: could not create '/usr/local/lib/python2.7/dist-packages/debops': Permission denied

---

Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;file='/tmp/pip_build_niccolox/debops/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-w3JbU6-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_niccolox/debops
Storing debug log for failure in /home/niccolox/.pip/pip.log
niccolox@trustyinx:~/Projects|⇒

Custom playbook can't find "task_src" lookup plugin

I'm trying to build a wordpress application role in a separate role/playbook. I'm using the phpmyadmin role as a base. So you get dependencies that look like:

dependencies:

  - role: debops.php5
    php5_packages: [ 'php5-mysqlnd', 'php5-mcrypt', 'php5-gd', 'php5-dev' ]
    php5_pools: [ '{{ wordpress_php5_pool }}' ]
    when: wordpress_dependencies is defined and wordpress_dependencies
    tags: [ 'mysql', 'wordpress' ]

  - role: debops.nginx
    nginx_servers: [ '{{ wordpress_nginx_server }}' ]
    nginx_upstreams: [ '{{ wordpress_nginx_upstream_php5 }}' ]
    when: wordpress_dependencies is defined and wordpress_dependencies
    tags: [ 'mysql', 'wordpress', 'nginx' ]

  - role: debops.secret

The role is in its own playbook called wordpress.yml that follows the application.yml standard.

---

- name: Manage WordPress service
  hosts: 'wordpress'
  sudo: True

  roles:
    - { role: wordpress, tags: wordpress }

I get this error when I run debops wordpress:

ERROR: Failed to template {{ lookup('task_src', 'nginx/pre_main.yml') }}: lookup plugin (task_src) not found

It finds the plugin if I just assign the host to the group debops_nginx and run debops. It only happens if I use a custom role/playbook where I use debops.nginx as a dependency. The playbook runs fine without it.

I tried copying the task_src.py plugin into my debops project, but it's still not picked up. I also tried overwriting the lookup_plugins in .debops.cfg to point to it. That didn't work either. It doesn't seem like lookup_plugins is picked up at all with a custom playbook.

locked out after running debops

I just followed the instructions, ran debops, everything finished successfully. ( my goal was to install gitlab using this utility here ).

Now, whenever I want to ssh to my host I get the error message Permission denied (publickey).
Any tasks run via debops fail for the same reason.

Locally, some folders have been created in my project-folder ( ansible/secret/credentials, ansible/secret/dhparam, ansible/secret/pki )

I did not expect the default playbook of this package to instantly lock me out of my system.

What happened here? What can I do to access my host again?

atd_default_allow in atd role does not use bootstrap__admin_name

Having a bunch of machines with different admin account names that does not have python or anything else installed that ansible need. Am running my first bootstrap without a ansible_user set, specifying on command line --user that I know works against just this debian host with --become parameter and --ask-pass.

Am expecting debops bootstrap to use this credentials to setup this host with chosen bootstrap__admin_name, bootstrap__admin_sshkeys and bootstrap__domain in my case.
After bootstrap is done I add to hosts file ansible_user={{ bootstrap__admin_name }} and I can use depops.

Problem I have found is that atd_default_allow in atd role is by default using ansible_ssh_user and not bootstrap__admin_name. Meaning wrong username is added to /etc/at.allow.