Coder Social home page Coder Social logo

debops-playbooks's Introduction

debops_logo DebOps playbooks

CII Best Practices

This repository contains all the playbooks used by DebOps.

Here are a few services that are available

Fully loaded ready to go applications

GitLab GitLabCI Etherpad DokuWiki ownCloud phpIPAM Mailman

Databases

PostgreSQL MariaDB Redis Memcached Elasticsearch

Programming languages

Ruby Golang Java NodeJS PHP

Web application deployment

nginx Apache RubyOnRails

Service monitoring and logging

LibreNMS monit rsyslog

Networking

dnsmasq DHCP Radvd ferm postfix SMS SSH NFS Samba Tinc

Virtualization

LXC Docker libvirt

Backup and encryption

Safekeep BoxBackup encFS cryptsetup SKS Monkeysphere

Security

PKI dhparam slapd

Overview of how playbooks work within DebOps

There's a common.yml playbook which gets ran on every host except localhost. This includes standard services like sshd and ferm. The full list can be found here.

Additional plays are then ran based on what groups the host is in. For example if you wanted to setup a Gitlab instance you would add a host to the [debops_service_gitlab] group in your inventory.

That carries over for things like postgresql, ruby or any service for the most part. You can also install roles as dependencies rather than use inventory groups.

The dependency approach makes sense in a lot of places, especially for setting ports through ferm or perhaps installing a database for a role that demands that database.

You can view all of the services and plays here.

View a dependency graph

Dependency graph

The dependency graph was generated using: ansigenome.

Status page

If you want to keep tabs on each role's status then check out our status page.

debops-playbooks's People

Contributors

anbuku avatar anzil avatar barraponto avatar bfabio avatar do3cc avatar drybjed avatar erethon avatar ganto avatar gasuketsu avatar htgoebel avatar it-praktyk avatar jstruebel avatar le9i0nx avatar machty avatar neo-tahi avatar nickjj avatar noplanman avatar pedroluislopez avatar pniederlag avatar ramitsurana avatar scibi avatar tallandtree avatar thiagotalma avatar umeboshi2 avatar ypid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

debops-playbooks's Issues

Remove debops.asible role

  • this role can currently be replaced by bootstrap-ansible.sh script in https://github.com/debops/debops/ repository
  • ansible should be installed through APT from .deb packages. Currently a backported 1.7.2 version is available in Debian, newer version should be provided via a local APT repository (depends on #65).

ruby_version backport error: No package matching 'ruby2.1'

I get the below error when using debops.rails_deploy and debops.ruby to install ruby 2.1 via backport. I'm running Ubuntu 14.04 LTS (Trusty Tahr). I've based my ansible scripts on the rails deploy example here: https://github.com/debops/ansible-rails_deploy/tree/master/docs/examples/ansible

$ debops custom -v -t dash-api
TASK: [debops.ruby | Install Ruby packages] *********************************** 
<ec2-X-X-X-X.compute-1.amazonaws.com> ESTABLISH CONNECTION FOR USER: ubuntu
<ec2-X-X-X-X.compute-1.amazonaws.com> REMOTE_MODULE apt name=ruby,ruby2.1,libruby,libruby2.1,ruby-dev,ruby2.1-dev,ruby-all-dev,rubygems-integration state=latest
failed: [api-staging01] => (item=ruby,ruby2.1,libruby,libruby2.1,ruby-dev,ruby2.1-dev,ruby-all-dev,rubygems-integration) => {"failed": true, "item": "ruby,ruby2.1,libruby,libruby2.1,ruby-dev,ruby2.1-dev,ruby-all-dev,rubygems-integration"}
msg: No package matching 'ruby2.1' is available

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
debops.apt_preferences | Create APT preferences ------------------------- 3.23s
debops.etc_services | Make sure /etc/services.d directory exists -------- 2.23s
debops.etc_services | Create /etc/services.d/00_ansible ----------------- 2.19s
debops.ruby | Install Ruby packages ------------------------------------- 2.10s
debops.nodejs | Get npm global temporary directory ---------------------- 2.01s
debops.nodejs | Install NodeJS packages --------------------------------- 2.00s
debops.backporter | Update APT cache to get current package information --- 1.89s
debops.etc_services | Assemble /etc/services.d -------------------------- 1.65s
debops.nodejs | Install npm --------------------------------------------- 1.61s
debops.etc_services | Divert original /etc/services --------------------- 1.60s
           to retry, use: --limit @/Users/jon/custom.retry

api-staging01              : ok=28   changed=0    unreachable=0    failed=1   
---
# playbooks/custom.yml

- name: Deploy apiserver
  hosts: debops_rails_deploy
  sudo: true

  roles:
    - { role: debops.rails_deploy, tags: apiserver }
---
# inventory/group_vars/all/debops_ruby.yml
# Common debops.ruby config

# Specify version of Ruby to install:
# - 'apt':      (default) will install Ruby packages automatically depending on
#               what's available in APT at the time
#
# - 'backport': will enable backporting of packages from the next distribution
#               (currently on Debian Wheezy it's required to install Ruby 2.1
#               packages)
#
ruby_version: 'backport'

Proposed role: debops.hhvm

HHVM, or the HipHop Virtual Machine, is a virtual machine for PHP developed by Facebook to improve the performance of PHP applications. Unlike the regular PHP runtime, HHVM uses a just-in-time compiler to convert scripts into native machine code. As a result, third-party benchmarks have shown as much as a 3x load time reduction over PHP-FPM 5.4 for tasks like loading a regular Drupal website.
Read more at: https://www.vultr.com/docs/installing-hhvm-and-nginx-apache-on-ubuntu-debian-mint

Hardening recommendations for common playbook

I'm finalizing a DebOps project that runs in a separate playbook. I just have a question about the common playbook that runs when you just call debops before hand.

Do you add extra configuration when you create your own projects? Or do the role defaults follow common sysadmin best practices?

They look good to me, but I'm not a sysadmin per say. I'd like to offer some good defaults. I looked at the examples, but couldn't see much beyond subnet limitations for sshd.

No RabbitMQ?

Would be great if RabbitMQ can be installed through debops :)

Where to add example configuration snippits?

Good evening

I am using a lot of DebOps roles lately and have created some configuration snippets which I think could also come in handy for others. Is there a good place to add them somewhere.

Example. Using the debops.apt_preferences role, I configured apt preferences for my systems to handle various sources in the sources.list:

apt_preferences_list:
  - package: '*'
    suffix: '_Debian'
    raw: |
      Explanation: Configure the installed release higher as the default (500) so that those packages are preferred.
      Package: *
      Pin: release o=Debian,n={{ ansible_distribution_release }}
      Pin-Priority: 600

      Explanation: The default priority of packages from backports is 100 which is even lower then testing and unstable (500).
      Explanation: Prefer backports over testing and unstable but donโ€™t automatically upgrade to it.
      Package: *
      Pin: release o=Debian Backports,n={{ ansible_distribution_release }}-backports
      Pin-Priority: 400

      Explanation: Install packages from testing if no package with the same name is available in release archives or backports or other archives.
      Package: *
      Pin: release o=Debian,a=testing
      Pin-Priority: 300

      Explanation: Only install packages from unstable if explicitly asked for or the package is pined.
      Package: *
      Pin: release o=Debian,a=unstable
      Pin-Priority: -1

      Explanation: Only install packages from experimental if explicitly asked for or the package is pined.
      Package: *
      Pin: release o=Debian,a=experimental
      Pin-Priority: -1

I also worked a bit on ownCloud tuning. I would propose to add an directory to each role for example configurations which might be useful for that role. What do you think?

Typo in message

TASK: [debops.pki | Make sure that Ansile local facts directory exists] *******

Ansile => Ansible

cannot use lookup on paths with a space

I am trying to use the MySQL role and debops uses the lookup function to locate a password.
That, however, does not work when the directory contains spaces.

TASK: [debops.mysql | debug var=secret] *************************************** 
ok: [default] => {
    "var": {
        "secret": "/home/muelli/vcs/repository/2015 Projects Deliverables/some more spaces/ansible/../secret"
    }
}

TASK: [debops.mysql | Manage MySQL users] ************************************* 
fatal: [default] => Failed to template {{ item.password | default(lookup("password", secret|replace(" ", "\\ ") + "/credentials/" + ansible_fqdn + "/mysql/" + item.name + "/password length=" + mysql_password_length)) }}: need more than 1 value to unpack

FATAL: all hosts have already failed -- aborting

The playbook is as simple as that:

- hosts: all
  sudo: yes

  vars:
    # Hm, mysql seems to be locked
    tcpwrappers: False

  pre_tasks:
    ### This is more logic for detecting whether mysql is already installed
    - name: Check if foo is installed
      command: dpkg-query -l mysql-server
      failed_when: False
      changed_when: False
      register: deb_check

    - debug: var=ohai_ip_address
    - command: echo Always restart mysql prior to making changes...
      # Because when we already run with --skip-grant-tables then we
      # cannot update users' passwords.
      changed_when: True
      # This does not necessarily help when mysqld is not yet installed :(
      failed_when: False
      # So we add more logic here
      when: deb_check.stdout.find('no packages found') != -1
      notify: ['Restart mysql']

  roles:
    # Include SSHd so that SSH access (for ansible) will be allowed
    - debops.sshd
    - role: debops.mysql
      mysql_mysqld_bind_address: '0.0.0.0'
      mysql_root_password: root
      mysql_users:
        - name: root
          host: '%'
          password: 'root'
          priv: mysql.*:ALL
          append_privs: yes
          state: present


and I run ansible like this:

env ANSIBLE_FILTER_PLUGINS=~/.ansible/plugins/filter_plugins/:/usr/share/ansible_plugins/filter_plugins:~/vcs/debops.playbooks/playbooks/filter_plugins/   ANSIBLE_LOOKUP_PLUGINS=~/.ansible/plugins/lookup_plugins/:/usr/share/ansible:~/vcs/debops.playbooks/playbooks/:~/vcs/debops.playbooks/playbooks/library/database/ldap/:~/vcs/debops.playbooks/playbooks/library/database/:~/vcs/debops.playbooks/playbooks/library/database/   ANSIBLE_ROLES_PATH=~/vcs/:~/vcs/ansible-hydra/roles   ANSIBLE_LIBRARY=~/vcs/debops.playbooks/playbooks/library/  ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false PYTHONUNBUFFERED=1 ANSIBLE_SSH_ARGS='-o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/home/muelli/.vagrant.d/insecure_private_key --user=vagrant --connection=ssh --inventory-file=ansible/vagrant_ansible_inventory  -vv --limit='all' ansible/playbook.yml

I expected it to work fine, because it does in a directory without spaces.

Regeneration of dhparams files forces constant snapshot backups to Ansible Controller

In debops.pki role, Diffie-Hellman parameters are regenerated daily, which means that they change. On an Ansible run, automatic archive of current /etc/pki/ directory is made and sent to Ansible Controller, which means that this step will be performat at least once per day, which makes this cumbersome when inventory is kept in git repository (constant bloat).

Backup command should exclude /etc/pki/*/dhparams/ directories as well as /etc/pki/*/dhparam.pem symlinks. The same needs to be done in command that creates pki-signatures.txt file. This should fix the problem.

New hook format proposal

This is a proposal to alter the pre_tasks and post_tasks with using functionality native to ansible. I tested a dozen different methods and this seems to be the best option. The goal is to remove the dependency on task_src so it is must easier to use from galaxy while still preserving the functionality.

Playbook

- name: "Test"
  hosts: local
  sudo: False
  remote_user: vagrant
  vars:
    debops_pre_tasks_users: 'hook.yml'
  roles:
    - { role: test, tags: [test] }
  tasks:
    - debug: var=debops_pre_tasks_users

File: test/hook.yml

- debug: msg="Hook Fired"

File: test/main.yml

- set_fact:
    debops_pre_tasks_users_fact: 'hook.yml'
    debops_fire_hooks: True

# - debug: msg="Firing test 0"
# - name: DebOps pre_tasks hook
#   include: "{{ lookup('task_src', 'users/pre_main.yml') }}"
# ERROR: Failed to template {{ lookup('task_src', 'users/pre_main.yml') }}: lookup plugin (task_src) not found

# - debug: msg="Firing test 1"
# - include: '{{ debops_pre_tasks_users_fact }}'
#   when: debops_pre_tasks_users_fact is defined
# ERROR: file could not read: /Users/patrickheeney/Dev/infrastructure/playbooks/roles/test/tasks/{{ debops_pre_tasks_users_fact }}

# - debug: msg="Firing test 2"
# - include: '{{ lookup("file", debops_pre_tasks_users_fact) }}'
#   when: debops_pre_tasks_users_fact is defined
# ERROR: file could not read: /Users/patrickheeney/Dev/infrastructure/playbooks/roles/test/tasks/{{ lookup("file", debops_pre_tasks_users_fact) }}

- debug: msg="Firing test 3"
- include: hook.yml
  when: debops_fire_hooks is defined and debops_fire_hooks

- debug: msg="Firing test 4"
- include: "{{ debops_pre_tasks_users_fact | default('empty.yml') }}"

- debug: msg="Firing test 5"
- include: "{{ debops_pre_tasks_users_doesnt_exist | default('empty.yml') }}"

- debug: msg="Firing test 6"
- include: "{{ debops_pre_tasks_users | default('empty.yml') }}"

- debug: msg="Firing test 7"
- include: "{{ debops_pre_tasks_users }}"
  when: debops_pre_tasks_users is defined

File: test/empty.yml

- debug: msg="Empty"

This successfully fires test 5, 6, 7. Test 4 doesn't work when using facts. I also explored using symlinks and file module to check if it exists before including, but this approach is much better. Test 7 looks to be the most ideal, however it fails when debops_pre_tasks_users is not defined anywhere. Test 6 seems to work in all scenarios I tested, which is similar to other popular playbooks that utilize hooks.

This same principle could be applied for template_src to be something like:

#  template:
#    src: '{{ lookup("template_src", "etc/ssh/authorized_keys_lookup.d/" + item + ".j2") }}'
#    dest: '/etc/ssh/authorized_keys_lookup.d/{{ item }}'

  template:
    src: '{{ debops_template_src_authorized_lookup_folder | default("etc/ssh/authorized_keys_lookup.d/") + item + ".j2" }}'
    dest: '/etc/ssh/authorized_keys_lookup.d/{{ item }}'

I tested this with:

- debug: msg="Firing test 8"
- debug: msg="{{ debops_template_src_authorized_lookup_folder | default("etc/ssh/authorized_keys_lookup.d/") + item + ".j2" }}"
  with_items:
    - "test"

There are other popular roles using this format like https://github.com/ansistrano/deploy . I only tested setting vars within the playbook. There may be other ways to set the variable to have it recognized by the time it is included.

I don't think ansible 2.0 will solve this with the block module as the playbook searches for the lookups and it won't be found. If include + with_items or with_first_found starts working again, then it will be another way to expand upon this syntax.

If this proposal is accepted, I can help submit pull requests to individual repositories. Debops would likely need to create a naming structure to be consistent with hook variables and template variables.

debops.mysql: warnings after Wheezy -> Jessie upgrade

After an upgrade and reboot, in journalctl logs MySQL emits a warning:

[Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in afuture release. Please use the full name instead.
[Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.

debops.mysql role does not modify original /etc/mysql/my.cnf, but maybe it could update these two options using lineinfile if jessie is detected as the operating system of a host.

No mongodb?

Would be great to have support for installing MongoDB :)

debops.reprepro does not support distribution upgrades

When a host is upgraded from Wheezy to Jessie, debops.reprepro role replaces distributions configuration file with new version configured for Jessie, which breaks repository symlinks. Instead, conf/distributions and any other files should be generated from parts and assembled, to allow older distributions to still exists while newer are added automatically.

Add build server infrastructure

This functionality will replace debops.backporter role and it should allow to easily backport existing packages using package sources (.dsc) and allow to build custom software packages and deploy them from local APT repositories.

To do this, DebOps needs to support at least pbuilder and rebuildd applications. Build packages can then be pushed to local APT repositories managed by reprepro directly, without using Ansible Controller host as the intermediary.

Support for proper GPG keysigning vs automated GPG keysigning needs to be taken into account. Can packages be built on remote servers, then moved to a "staging" area when they are signed by the developer, and after signing moved to APT repository via a script?

Useful links:

RFC: Drop role dependencies from roles

Some of you already know this but if not, @patrickheeney and other users call for better support for standalone role usage in DebOps, especially when users come from Ansible Galaxy. There is a way to do this, but it comes with a cost: role dependencies could be dropped from the roles themselves, and moved to the role plays (most of the roles have their own plays). The problem is, that for the DebOps playbook to work the same as right now, each play would need to have the complete copy of its dependency chain, preferably with all the variables from various roles used the same way.

Right now this dependency chain is resolved by Ansible itself and not really visible, moving it to the playbooks themselves would make it visible, but any changes would need to be propagated across all affected playbooks. After all roles are modified, from the user perspective playbook should work exactly the same as it works today (not 100% sure, but most likely). Custom playbooks and roles will also need to have full dependency chains present, otherwise roles will work partially and for example webserver won't have the needed firewall rules set if it hasn't been installed on a host previously.

It will take some time to rewrite all the roles and playbooks, and I would most likely do this after the playbooks are split into separate files (in the middle of the process right now). I'm inclined to do the conversion since individual roles will be more flexible and it will be easier to switch different services around, for example creating a set of playbooks that use apache2 instead of nginx as a webserver, but it will require manual maintenance of the dependency chain between various roles instead of relying on Ansible to do this for us.

Comments?

Move ansigenome metadata to separate file

Ansible v2 introduces stricter tests for role syntax and existing ansigenome metadata are breaking ansible-playbook:

Running Ansible playbook from:
/home/drybjed/.local/share/debops/debops-playbooks/playbooks/site.yml ...
ERROR! 'ansigenome_info' is not a valid attribute for a RoleMetadata

The error appears to have been in
'/home/drybjed/.local/share/debops/debops-playbooks/roles/debops.apt_preferences/meta/main.yml':
line 3, column 18, but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


allow_duplicates: True
                 ^ here

Since Ansible should not look in other files located in meta/ directories, I will move ansigenome metadata to meta/ansigenome.yml file in all roles.

Both `ferm_ansible_controllers` and `tcpwrappers_ansible_controllers` need to be set

After setting ferm_ansible_controllers, I still was not able to access my server. I then found that I additionally need to set tcpwrappers_ansible_controllers. Both ferm and tcpwrappers are common roles, active by default.

Possible solutions I see:

  1. Merge both variables into debops_ansible_controllers (or so)
  2. Make both variables default to debops_ansible_controllers (which defaults to an empty list)
  3. If bot roles are active, do not activate tcpwrappers for sshd.

standalone usage of roles w/ ferm dep locks down iptables

@drybjed per our convo in #debops. In using a role such as debops/ntp or debops/mysql which has a dependency on ferm, using the default settings locks me out of my VM. In looking at the docs for the ferm role, i see that overriding the default with ferm: false gets me the desired behavior.

I guess the issue is that if I've found one of these roles on ansible-galaxy and am not using the full debops experience, applying any role with a dep of ferm is going to lock the user out of their VM

Add OpenDCIM role

OpenDCIM is an open source Data Center Infrastructure Management application. This might require support of Basic Authentication in nginx to work properly.

Proposed role: debops.ids

Provide default intrusion detection systems like debops.ossec + debops.audit? Leverage ELK stack for audit views and ossec for notifications (email + script)?

Should:

  • Audit user logins
  • Audit known activities (DDOS, Synflood, Auth attempts)
  • Audit custom activities (Watch this file in /opt/secret for changes)
  • Notify for known activities (MD5 change of core lib or executable)
  • Notify for custom activities (if desired)
    ...

debops.gitlab: After an upgrade to Jessie, GitLab refuses to start or perform operations

For example, running a backup rake in GitLab shows:

rake aborted!
LoadError: libicui18n.so.48: cannot open shared object file: No such file or directory - /var/local/git/gitlab/vendor/bundle/ruby/2.1.0/extensions/x86_64-linux/2.1.0/charlock_holmes-0.6.9.4/charlock_holmes/charlock_holmes.so
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/charlock_holmes-0.6.9.4/lib/charlock_holmes.rb:1:in `<top (required)>'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/gitlab-grit-2.6.7/lib/grit.rb:79:in `<top (required)>'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `block in require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:214:in `load_dependency'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/dependencies.rb:229:in `require'
/var/local/git/gitlab/vendor/bundle/ruby/2.1.0/gems/gitlab_git-5.8.0/lib/gitlab_git.rb:4:in `<top (required)>'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:76:in `require'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:76:in `block (2 levels) in require'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:72:in `each'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:72:in `block in require'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:61:in `each'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:61:in `require'
/var/lib/gems/2.1.0/gems/bundler-1.7.12/lib/bundler.rb:134:in `require'
/var/local/git/gitlab/config/application.rb:6:in `<top (required)>'
/var/local/git/gitlab/Rakefile:5:in `require'
/var/local/git/gitlab/Rakefile:5:in `<top (required)>'
(See full trace by running task with --trace)

This might be related to charlock_holmes gem, after a quick googling I've found this GitLab issue which points to this apparent fix. This should be investigated further to find a way to implement the fix in debops.gitlab and other similar roles.

Proposed role: debops.canvas

Automatically stand up and configure an Instructure Canvas installation. This would be huge in the education sector - Canvas is absolutely wrecking every other LMS in popularity an adoption right now.

Create 'root.yml' playbook which prepares base system

This playbook should be executed before common.yml playbook ("pre-common" play included there will be moved to root.yml).

This playbook is meant to prepare and maintain settings which other playbooks and roles can use. It will have a task that creates a set of important directories, like /etc/ansible/fact.d/ or other missing and required directories. It will also configure a set of local facts in ansible_local.root branch which other roles can use for global settings, like base data directory, base home directory for system services, and so on.

Missing sudo

I have install a debops environment on my MacBook Air.
I try to setup a host under a minimal Debian Jessie.

MacBookAir jg $ debops
Running Ansible playbook from:
/Users/jg/Library/Application Support/debops/debops-playbooks/playbooks/site.yml ...

PLAY [Gather default and custom facts] ****************************************

GATHERING FACTS ***************************************************************
ok: [myhost.mydomain.local]

TASK: [Check IP address of Ansible Controller] ********************************
ok: [myhost.mydomain.local]

PLAY [Prepare base environment] ***********************************************

TASK: [Create root directories] ***********************************************
failed: [myhost.mydomain.local] => (item={'path': '/etc/ansible/facts.d', 'fact': 'facts'}) => {"failed": true, "item": {"fact": "facts", "path": "/etc/ansible/facts.d"}, "parsed": false}
/bin/sh: sudo: command not found
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/jg/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: auto-mux: Trying existing master
debug1: mux_client_request_session: master session id: 2
Shared connection to xxx.xxx.xxx.xxx closed.

Proposed role: debops.ralph

Ralph is an asset management software for datacenters, which could replace phpIPAM as data source and provide data for future DNS server (Ralph can generate DHCP and DNS entries).

This role might require Django support added as well.

How to reset `ferm_ansible_controllers` and `tcpwrappers_ansible_controllers`, etc?

This is a follow up to #53.

I tried to reset the settings by doing the following:

  • Delete /etc/ansible/facts.d/*
  • Delete /etc/hosts.allow.d/10_ansible_controller
  • Remove definitions for ferm_ansible_controllers and tcpwrappers_ansible_controllers from the host-vars.

According to debops/ansible-sshd@e3ea220 I would assume that this will allow ssh-access from all clients. But it does not: The (former deleted) files contain only my current client, while I would expect it to contains ALL (or similar).

Explicitly setting ssh_host_allow and sshd_tcpwrappers_default did not help.

Update debops.debops role

debops.debops should manage a separate account on a server which can be set up as a central Ansible Controller for a cluster. It should deploy DebOps scripts, playbooks and roles to a system-wide path, create a cron script that will update the roles and playbooks.

Dependency on [debops.ansible]{https://github.com/debops/ansible-role-ansible/) role should be dropped, debops.debops role can either install ansible package from Debian (old version), or use bootstrap-ansible.sh script to install development version of Ansible.

Overhaul of ldap_attr and ldap_entry modules

These modules are quite old and don't support new Ansible features, like `no_log', properly. I'm sure that individual module parameters like 'bind_pw' and 'userPassword' (if it's detected) should be automatically censored, because right now they leak passwords into logs.

Unfortunately my knowledge of Python is abysmal, so maybe someone more experienced in Ansible module development could look at them and fix the issues?

Make group `distribution_$distri` and `distribution_$distri_$release` avalable.

I use these to define apt pinnings for Debian which make no sense on other distros like Ubuntu. For this I have in a pre task:

- name: Group hosts by distribution
  group_by:
    key: 'distribution_{{ ansible_distribution }}'
  changed_when: False

- name: Group hosts by distribution and release
  group_by:
    key: 'distribution_{{ ansible_distribution }}_{{ ansible_distribution_release }}'
  changed_when: False

Would it make sense to add this to DebOps?

See https://docs.ansible.com/ansible/playbooks_best_practices.html#operating-system-and-distribution-variance

Support secure connections to remote databases (MySQL, PostgreSQL)

Support for encrypted connections to remote databases is needed for high availability and failover if necessary. Check if it can be done with TLS support in database servers (support in client applications might be required) or investigate an easy way to do it with stunnel managed by a separate role.

Proposed role: debops.monitoring

Provide out of the box monitoring framework. Perhaps using the following?

Server and Service Monitoring: Sensu
System Metrics: Collectd
App Metrics: Statsd
Metrics visualization: Grafana
Metrics Storage and Collection: Influxdb
Alerts and Notification routing: Sensu
Integrations: Pagerduty, Hubot...?

Use DebOps "apt_codename" instead of "ansible_distribution" throughout playbooks for Raspbian support

If bootstrap.yml installs "lsb-release" and apt_codename is setup from within "debops.apt" to be used elsewhere, please use it consistently within all dependencies. Raspbian shows up as Debian in "ansible_distribution" but correctly shows up in "ansible_lsb" as a separate distribution. I will submit a pull request for all modules impacted if desired.

Debian:

        "ansible_distribution": "Debian",
        "ansible_distribution_major_version": "7",
        "ansible_distribution_release": "wheezy",
        "ansible_distribution_version": "7.6",
...
        "ansible_lsb": {
            "codename": "wheezy",
            "description": "Debian GNU/Linux 7.6 (wheezy)",
            "id": "Debian",
            "major_release": "7",
            "release": "7.6"
        },

Raspbian:

        "ansible_distribution": "Debian",
        "ansible_distribution_major_version": "jessie/sid",
        "ansible_distribution_release": "jessie/sid",
        "ansible_distribution_version": "jessie/sid",
...
        "ansible_lsb": {
            "codename": "jessie",
            "description": "Raspbian GNU/Linux testing (jessie)",
            "id": "Raspbian",
            "major_release": "testing",
            "release": "testing"
        },

Proposed role: debops.cron

This role should help manage cron and crontabs.

  • allow creation of custom scripts in /etc/cron.d/ with support for variables like MAILTO (which are lacking in cron Ansible module) from a dependency
  • perhaps add an option to create or remove entries from user contabs using lineinfile
  • randomize cron hourly/daily/weekly/monthly times to avoid huge hit on the whole data center early in the morning (some ideas: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=373152)

Reduce dependencies for individual usage

I found debops via ansible galaxy and I first attempted to use a bunch of the roles in my playbooks. However I later find out they all have a unique set of dependencies that really require the whole debops package. This doesn't work in my use case as I need to stay with stock ansible as close as possible. Debops is one of the best collections of ansible roles that I have come across and I would like to help contribute to making these roles better versus forking them.

It is my hope that you would consider going "native" on some of the functionality. This may involve some alternative approaches to functionality or collectively working with ansible to try and get some missing functionality in core. The end goal is to be able to use these roles independently and in insolation of the playbooks repository. These are some of the problems I encountered when using devops.

Pre / post role hooks

I definitely understand the use case for these, but I don't know that debops should cater to it. This seems like it could easily be accomplished with stock ansible with something like:

roles:
  - { role: custom-pre-mysql }
  - { role: debops.mysql, tags: [mysql] }
  - { role: custom-post-mysql }

If you want something to happen before or after a role, you just include your own role, before or after. If you want to override functionality, you can do it in the post role.

Alternatively, maybe you can add some feature detection:

- name: DebOps pre_tasks hook
  include: "{{ lookup('task_src', 'mysql/pre_main.yml') }}"
  when: debops_env == 'native'

That way it should work without it, and when ran in debops environment you can pass in the extra vars, put it in group vars, hosts, etc.

Template src

If I understand this correctly the goal is to be able to override templates to provide your own? I have seen this in some roles, but not others. So it seems this may be legacy or new functionality? that is not consistent across all roles yet?

I am not familiar with ansible internals but this seems like something better suited at the ansible level to be able to override any templates. For example {{ lookup("template", "etc/fail2ban/jail.local.d/default.local.j2") }} should have a way in the config to search custom paths for that file. if this doesn't exist, and hasn't been proposed, then we should voice our support or submit a pull request to introduce this functionality to core.

In the interim, you could always convert it to a var like fail2ban_template_jail_local: 'etc/fail2ban/jail.local.d/default.local.j2'. This would allow overrides. On templates with loops, you could specify a folder instead template: src={{ fail2ban_template_jail_local + item.name }}.

Also, maybe you can consider the feature detection mentioned above? One that loads your template_src when in the right environment, otherwise use a normal template that doesn't allow it to be overwritten.

Other role dependencies

I was also wondering about mysql for example, requiring secret, ferm, and tcpwrappers. It seems these dependencies offer some security or other features, but are not actually required by the role. Is there another way to have these dependencies installed and configured when they are part of a playbook that has them, but not a hard dependency on the individual role.

I am also not sure how this works if we try to override all the ferm_input_list, but if there are 5 roles with ferm as a dependency that add more inputs, do the group vars override all of that or also add to it?

Maybe the dependencies can be removed and replace with documentation suggesting the installation of ferm and its inputs when using mysql? Or maybe there is a way to make the dependencies optional when you want to use mysql but not ferm or tcpwrappers?

Manage 'default_server' option in nginx for HTTP and HTTPS separately

Currently nginx selects 1 server configuration automatically and sets it as a "default server", which is then saved in Ansible local facts and can be changed only by removing that fact manually. This is needed for idempotent nginx support from multiple roles - each role calls debops.nginx role with different set of nginx server configuration variables, so local fact is needed to remember which server should be the default.

This is also important for keeping ipv6only=off option sane - because it can be present only once for each connection type in all server configurations. To make that easier to manage, both default_server and ipv6only=off are enabled in only one server configuration.

However in a rare case that only one type of service is configured (HTTP, for example) and another role adds a HTTPS service, this might result in a failed nginx state. New HTTPS service does not have ipv6only=off option applied to it, and since no other HTTPS service was configured previously, nginx listens only for IPv6 connections on HTTPS, not IPv4+IPv6.

Because of that, debops.nginx role should track the default server for HTTP and HTTPS connections separately.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.