stackstorm / ansible-st2

Ansible Roles and Playbooks to deploy StackStorm

Ansible-st2

Ansible roles and playbooks to deploy StackStorm.

StackStorm is event-driven automation platform written in Python. With over 50+ integrations like GitHub, Docker, Nagios, NewRelic, AWS, Ansible it allows you to wire together your existing infrastructure into complex Workflows with auto-remediation and many more. Aka IFTTT orchestration for Ops.

Supported platforms

• Ubuntu Focal (20.04)
• RHEL 7 and CentOS 7
• RHEL 8 and Rocky Linux 8

If you are using the provided Vagrantfile, note that it uses Focal by default.

In order to access StackStorm Web UI, please do not forget to ensure that the http and https ports are allowed by the firewall.

Requirements

A supported version of Ansible and its dependencies. At least 2 GB of memory and 3.5 GB of disk space is required, since StackStorm is shipped with RabbitMQ, Mongo, Redis and nginx.

Installation

# Deploy StackStorm
ansible-playbook stackstorm.yml

Variables

Below is the list of variables which you can redefine in your playbook, or inventory, to customize the StackStorm deployment:

Variable	Default	Description
st2repo
st2repo_name	stable	StackStorm PackageCloud repository to install. stable, unstable, staging-stable, staging-unstable
st2
st2_version	latest	StackStorm version to install. present to install available package, latest to get automatic updates, or pin it to numeric version like 2.2.0 or with revision like 2.2.0-1
st2_config	{}	Hash with StackStorm configuration settings to set in st2.conf ini file.
st2_system_user	stanley	System user from which st2 will execute local/remote shell actions.
st2_system_user_in_sudoers	yes	Add st2_system_user to the sudoers (recommended for most st2 features to work).
st2_ssh_key_file	/home/{{st2_system_user}}/.ssh/{{st2_system_user}}_rsa	Path to st2_system_user SSH private key. It will be autogenerated by default.
st2_auth_enable	yes	Enable StackStorm standalone authentication.
st2_auth_username	testu	Username used by StackStorm standalone authentication.
st2_auth_password	testp	Password used by StackStorm standalone authentication.
st2_save_credentials	yes	Save credentials for local CLI in /root/.st2/config file.
st2_rbac_enable	no	Enable RBAC.
st2_rbac	See st2_rbac variable in role defaults	RBAC roles and assignments. This is a dictionary with three keys roles, assignments, and mappings (based on LDAP group membership). roles, assignments, and mappings are arrays. Each element in the array follows the exact YAML schema for roles, assignments defined in ST2 documentation. mappings are slightly different, they use a name and a mapping key, see ./roles/StackStorm.st2/defaults/main.yml for an example.
st2_ldap_enable	no	Enable LDAP authentication backend.
st2_ldap	See st2_ldap variable in role defaults	Settings for LDAP authentication backend. st2_ldap is a dictionary and has one item backend_kwargs. backend_kwargs should be provided as exactly listed in ST2 documentation for LDAP configuration.
st2_packs	[ st2 ]	List of packs to install. This flag does not work with a --python3 only pack.
st2_python_packages	[ ]	List of python packages to install into the /opt/stackstorm/st2 virtualenv. This is needed when deploying alternative auth or coordination backends which depend on Python modules to make them work.
st2web
st2web_version	latest	st2web version to install. present to install available package, latest to get automatic updates, or pin it to numeric version like 2.2.0 or with revision like 2.2.0-1.
st2web_ssl_certificate	null	String with custom SSL certificate (.crt). If not provided, self-signed certificate will be generated.
st2web_ssl_certificate_key	null	String with custom SSL certificate secret key (.key). If not provided, self-signed certificate will be generated.
st2web_nginx_config	null	String with a custom nginx configuration file (st2.conf). If not provided, the default st2.conf will be used.
st2chatops
nodejs_major_version	14	The default fits st2chatops version >= 3.5.0.
st2chatops_version	latest	st2chatops version to install. present to install available package, latest to get automatic updates, or pin it to numeric version like 2.2.0 or with revision like 2.2.0-1.
st2chatops_st2_api_key		st2 API key to be updated in st2chatops.env using "st2 apikey create -k" in a task
st2chatops_hubot_adapter		Hubot Adapter to be used for st2chatops. Default is shell, but should be changed to one of the supported adapters.[Required]
st2chatops_config	{ }	Based on adapter in st2chatops_hubot_adapter, provide hash for the adapter settings, to update st2chatops.env. For example, for Slack hubot adapter: st2chatops_config: HUBOT_SLACK_TOKEN: xoxb-CHANGE-ME-PLEASE
st2chatops_version	latest	st2chatops version to install. Use latest to get automatic updates or pin it to numeric version like 2.2.0.

Examples

Install latest stable StackStorm with all its components on local machine:

ansible-playbook --inventory localhost, --connection local stackstorm.yml

To install StackStorm on a remote machine named stackstorm.example.com, create and reference an inventory:

echo "stackstorm.example.com" > inventory
ansible-playbook --inventory inventory stackstorm.yml

Keeping the latest version is useful to update StackStorm by re-running the playbook, since it will reinstall (upgrade) st2 when there is new version available.

This is default behavior. If you do not want updates, consider pinning specific version and revision numbers.

Install a specific version of st2 with pinned revision number:

ansible-playbook stackstorm.yml --extra-vars "st2_version=2.2.0 st2_revision=8"

Installing behind a proxy

If you are installing from behind a proxy, you can use environment variables http_proxy, https_proxy, and no_proxy in the playbook. For the st2smoketests, you will need to disable proxy for localhost.

environment:
  http_proxy: http://proxy.example.net:3128
  https_proxy: http://proxy.example.net:3128
  no_proxy: 127.0.0.1,localhost

Development

There are a few requirements when developing on ansible-st2.

These are the platforms we must support (must pass end-to-end testing):

• Ubuntu Focal (20.04)
• CentOS 7
• Rocky Linux 8
• RHEL 7 (via AWS)
• RHEL 8 (via AWS)

The playbooks and roles must support Ansible idempotence. That is, re-running the stackstorm.yml playbook must end with the result changed=0.*failed=0 (provided that versions and revisions are pinned).

For local development, there is a Vagrantfile available. By default, the following command will set up an ubuntu20 box (ubuntu/focal64):

vagrant up

Other supported distributions:

vagrant up ubuntu20
vagrant up centos7
vagrant up rockylinux8

Other installation methods

You may be interested in other methods to deploy StackStorm:

• Configuration Management

  • Puppet Module

• Manual Instructions

  • Ubuntu Bionic (18.04)
  • Ubuntu Focal (20.04)
  • RHEL 8 / Rocky Linux 8
  • RHEL 7 / CentOS 7

Getting help

If you encounter any problems, our community is ready to help, feel free to:

• Ask questions in our public Slack channel
• Report issues, provide feature requests, or just give this project a ✮ star

Your questions, comments, and contributions are more than welcome and greatly appreciated!

ansible-st2's Issues

Add more Variables/Settings to Configure

Initial Minimum Viable implementation for installation of new packages was done in #42.

Real production use requires much more settings to configure.

• Ability to use unstable repo (?)
• Improve version/revision installation logic
• Ability to adjust st2.conf parameters #121
• nginx-related stuff (st2web)
  • Specify custom certificate #133
• MongoDB
  • Allow adjusting st2 config [database] group settings via Ansible variables #121
  • MongoDB role setting changes should inject into st2 config.
  • Add ability to use external MongoDB server. #17
• Much more to add [...]

Vagrantfile for more platforms

Since we started adding support for more platforms as part of #5 in #65 and #66 We'll need a way to bootstrap Vagrant for:

• Ubuntu14
• Ubuntu16
• CentOS6
• CentOS7

Example syntax:

vagrant up ubuntu14
vagrant up ubuntu16
vagrant up centos6
vagrant up centos7

Implementation examples:

• 1. Via Vagrant multi-machine https://www.vagrantup.com/docs/multi-machine/
     https://github.com/StackStorm/showcase-ansible-chatops/blob/master/Vagrantfile#L8-L21
     and
     https://github.com/StackStorm/showcase-ansible-chatops/blob/master/Vagrantfile#L51
• 2. https://github.com/StackStorm/st2puppet/blob/production/Vagrantfile#L40-L45

cc @Mierdin and @humblearner

Add functionality to get/set K/V pairs in st2 datastore

Similar to Pack Install #74 we'll need an abstraction for st2 key/value store.

For example define a list of K/V pairs in yaml which will be added in st2 datastore during Ansible provisioning.

See: https://docs.stackstorm.com/datastore.html

At a low level it could be implemented via Ansible plugins/modules (lookup/set):

• https://www.ansible.com/blog/how-to-extend-ansible-through-plugins
• http://docs.ansible.com/ansible/playbooks_lookups.html

Highlight v0.7.0 release via StackStorm Blog

Draft a StackStorm blog for ansible-st2 v0.7.0 release, highlight breaking changes, new features with usage examples and new roles like @lakshmi-kannan's bwc and @humblearner's st2chatops.

Idempotence

Second stackstorm.yml playbook re-run should end with the following: changed=0.*failed=0, meaning 0 resources changed.

Include Idempotence check in CI, see:
https://github.com/StackStorm/ansible-st2/blob/master/.kitchen.yml#L17-L18

EL7 Repo in PackageCloud

In testing EL7, I encountered this issue when running the Ansible playbook:

TASK [mongodb : yum | Install mongodb dependencies] ****************************
failed: [10.0.0.96] (item=[u'python-urllib3', u'pyOpenSSL', u'python-pyasn1', u'python-ndg_httpsclient']) => {"failed": true, "item": ["python-urllib3", "pyOpenSSL", "python-pyasn1", "python-ndg_httpsclient"], "msg": "Failure talking to yum: failure: repodata/repomd.xml from stackstorm: [Errno 256] No more mirrors to try.\nhttps://packagecloud.io/StackStorm/stable/el/7Server/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found"}
	to retry, use: --limit @/Users/oswaltm/Code/ansible-st2/stackstorm.retry

This is because the $releasever variable in the yum configuration evaluates to 7Server when on EL7 (whereas it equals simply 7 on CentOS7).

[ec2-user@ip-10-0-0-96 ~]$ cat /etc/yum.repos.d/st2.repo
[stackstorm]
baseurl = https://packagecloud.io/StackStorm/stable/el/$releasever/$basearch
name = Stackstorm Repo
[ec2-user@ip-10-0-0-96 ~]$ python -c 'import yum, pprint; yb = yum.YumBase(); pprint.pprint(yb.conf.yumvar, width=1)'
(...truncated...)
{'arch': 'ia32e',
 'basearch': 'x86_64',
 'releasever': '7Server',
 'uuid': '8b7c566e-243b-4985-a077-8e6dbeed7e16'}

This is expected behavior.

The question is, how should this get answered? Should we deal with this on the Ansible side, rewriting this config to include the actual hard-coded version like 7, instead of 7Server, or should we make a change on the packagecloud side that includes the expected path?

Refactor Mistral Role [EL]

• Rename mistral to st2mistral
• Tag all tasks in role
• Update meta file
• EL support
  • CentOS6
  • RHEL6
  • CentOS7
  • RHEL7
• Idempotence check

Depends on PostgreSQL Role [EL] #76

Publish on Ansible galaxy

Good "sign of quality" objective is to publish StackStorm roles on Ansible galaxy to get some more attention from the community.
At that point it should be tested/stable enough.

• Register Ansible Galaxy account
• Polish, Refactor, Follow best practices
• Integration Tests #15
• Ansible Galaxy Docs, Examples
• Decide: ship it as stackstorm.st2 role or st2, st2web, st2mistral, st2chatops roles
• Use Ansible Galaxy 3rd-party dependencies for postgresql, rabbitmq, mongo (We use our own roles now, but should leave an option for users to choose alternate external roles)

External role for RabbitMQ

Use external 3rd party Ansible Galaxy role for RabbitMQ (or create own).
Mark it as dependency for st2 role.

Improve st2mistral DB upgrade logic

According to https://docs.stackstorm.com/install/upgrades.html

Mistral DB upgrading may be required after every st2mistral package update:

# Stop related services
service mistral-api stop
service mistral stop

# Upgrade database
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate

# Restart related services
service mistral start
service mistral-api start

TODO

• Implement migration logic in st2mistral role
• Verify if service start/stop is a required step

Improve Security practices

From the:

• StackStorm/st2-packages#405
• StackStorm/st2docs#363

At the moment we don't even configure any username/passwords for external services like Mongo, RabbitMQ with st2 Ansible installation. Additionally, we're not sure whether those services are listening on 127.0.0.1 or are open to public net.

All of that needs better control and improvement.

• Ensure we can configure host for dependent services
• Services should run on 127.0.0.1 (PostgreSQL, Mongo, RabbitMQ) by default
• Configure/Set username/password for PostgreSQL, Mongo, RabbitMQ
• If not explicitly set, passwords for PostgreSQL, Mongo, RabbitMQ should be generated randomly and placed in st2.conf

PostgreSQL Role [EL]

External dependency ANXS.postgresql doesn't support EL platform family.
We need to write custom role to install PostgreSQL for Mistral.

• Tag all tasks in role
• Meta
• Get best practices from ANXS.postgresql code
• Ensure service is listening on 127.0.0.1 by default (configurable), see: #75
• EL support
  • CentOS6
  • RHEL6
  • CentOS7
  • RHEL7
• Idempotence check
• Update README

cc @humblearner based on your feedback

MySQL role: unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials

Existing MySQL role fails on some Ubuntu installations.

       TASK: [mysql | Change MySQL password] ***************************************** 
       <localhost> REMOTE_MODULE mysql_user login_password=VALUE_HIDDEN name=root password=VALUE_HIDDEN login_user=root
       <localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569 && echo $HOME/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569']
       <localhost> PUT /tmp/tmp4WsWvG TO /home/kitchen/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569/mysql_user
       <localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/kitchen/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569/mysql_user; rm -rf /home/kitchen/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569/ >/dev/null 2>&1']
       failed: [localhost] => {"failed": true}
       msg: unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials

       FATAL: all hosts have already failed -- aborting


       localhost                  : ok=11   changed=9    unreachable=0    failed=1  

Most probably using 3rd party Ansible Galaxy role for MySQL #8 will fix it.

allow st2_revision to be set to 'latest'

we'll probably have to pin to a specific version of stackstorm, but the revision is less important for us.

It'd be good to be able to set st2_version=2.2.1 st2_revision=latest to pin at whatever the latest 2.2.1 release is without having to know the exact revisions that've been released in advance

CI Tests

For better maintainability add automated tests.

• Test-kitchen
• CircleCI
  • Configuration
  • Parallelism
  • Badge
• Test Playbook run
• Write first minimal Rspec tests as proof of work (add more test coverage much later)

Pack Install

Implement a convenient ansible-friendly way (as Ansible module) to install st2 packs, as well as provide configurations/settings for them.

Decide:

• define config settings in yaml plays inline
• use /opt/stackstorm/configs/<pack>.yaml

https://docs.stackstorm.com/pack_configs.html

Alternative example from puppet-st2: https://github.com/stackstorm/puppet-st2#installing-and-configuring-packs

In short, it should work like similar pip or gem Ansible module, but for installing st2 packs.

Pass optional list of st2-packs to install

Add option to install additional StackStorm packs within Ansible installer.

Good feature, as seen at puppet-st2. Low priority, after v0.3.0.

Switch to MongoDB 3.2

Since StackStorm v1.6 MongoDB 3.2 is the default version,
see: https://docs.stackstorm.com/install/deb.html#install-dependencies for installation instructions.

Reflect the change in st2 Configuration Management repos.

Ansible playbook needs an update

[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will be removed in a future release.

Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{postgresql_env}}'). This feature
will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

Move DB creation from st2mistral to postgresql role

Since we have own postgresql role, - let it handle DB creation for st2mistral.

Use native Ansible modules postgresql_user and postgresql_db instead of init_mistral_db SQL template file.

So after all postgresql + st2mistral usage should look as following:

- name: Install st2mistral on a single node
  hosts: all
  roles:
    - name: Install PostgreSQL and configure databases
      role: postgresql
      vars:
        postgresql_databases:
          - name: "{{ st2mistral_db }}"
        postgresql_users:
          - name: "{{ st2mistral_db_username }}"
            pass: "{{ st2mistral_db_password }}"
            encrypted: yes

    - name: Install and configure StackStorm Mistral
      role: st2mistral
      vars:
        st2mistral_database_connection: postgresql://{{ st2mistral_db_username }}:{{ st2mistral_db_password }}@localhost/{{ st2mistral_db }}

Use best practices from https://github.com/ANXS/postgresql

Additionally, this will allow us to manage multi-node deployments #17 better in future.

Refactor st2web [EL]

• Tag all tasks in role
• Update meta file
• EL support
  • CentOS6
  • RHEL6
  • CentOS7
  • RHEL7
• Idempotence check
• New parameters to pass to st2web like hostname
  • Update README.md

Reflect 0.7.0 changes in st2docs

Add new features/changed vars from the v0.7.0 in
https://github.com/StackStorm/st2docs/blob/master/docs/source/install/ansible.rst docs.

Create CONTRIBUTING.md

Create a set of common rules, recommendations and conventions for development flow when contributing to ansible-st2.

CI fails on forked PRs

TravisCI is red for forked PRs, starting from st2chatops and bwc role implementations.

This is caused by private env variables defined in our Travis settings: HUBOT_SLACK_TOKEN and BWC_LICENSE which are not available to forked builds.

https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings

Similarly, we do not provide these values to untrusted builds, triggered by pull requests from another repository.

Find the way how to proceed this the right way in future, - because we can't turn down CI for contribution PRs.

Add CentOS6 CI in Travis

We started adding EL support in #65 and #65 as part of #5.
Working without CI is bad for quality, testing and reviewing.

There is a need to add CentOS6 integration Testing in Travis as we already do for Ubuntu Trusty and Ubuntu Xenial. The framework is already setup.

It's possible to make CentOS6 build non-voting to skip the failure for entire build (see travis.yml example for st2 repo) until we re-write all roles to support new platforms.

See:

• https://travis-ci.org/StackStorm/ansible-st2
• https://github.com/StackStorm/ansible-st2/blob/master/.travis.yml
• https://github.com/StackStorm/ansible-st2/blob/master/.kitchen.yml

cc @humblearner

ChatOps Support

Include st2chatops package installation

Literal version lookup fails

The "Lookup literal version" task is failing silently with "sort: unrecognized option `--version-sort'"
Leaving _st2_version set to "" and breaking following tasks

WebUI Installation

Add tasks to install StackStorm WebUI.

Options:

• st2_deploy.sh approach. Install by default, but add variable to opt-out: st2_install_webui.
• Use st2-puppet approach. Install webui in any case without option to opt-out.

Resources:

• https://github.com/StackStorm/puppet-st2/blob/master/manifests/profile/web.pp
• https://github.com/StackStorm/st2/blob/dffb8c5a2ad5180be9aece1c9d6954ea63c3ebea/tools/st2_deploy.sh#L605

Synchronize with recent Ansible

Fix the builds (since: https://circleci.com/gh/StackStorm/ansible-st2/31), check what's new in recent Ansible 2.0 version which broke the build, verify/test entire playbook story, check the backward compatibility.

Move from MySQL -> PostgreSQL for Mistral

Follow st2 mistral changes:

• StackStorm/st2#1860
• StackStorm/puppet-st2#46

Generate passwords for root and mistral PostgreSQL users

EL6 duplicated st2api st2stream st2auth services

This happens on EL6 (reproducible in CentOS6 Vagrant).
st2api, st2stream, st2auth got duplicated:

$ sudo st2ctl status
##### st2 components status #####
st2actionrunner PID: 699
st2actionrunner PID: 720
st2actionrunner PID: 740
st2actionrunner PID: 763
st2actionrunner PID: 789
st2actionrunner PID: 814
st2actionrunner PID: 841
st2actionrunner PID: 868
st2actionrunner PID: 896
st2actionrunner PID: 936
st2api PID: 29766
st2api PID: 29778
st2stream PID: 1034
st2stream PID: 1057
st2auth PID: 1085
st2auth PID: 1096
st2garbagecollector PID: 1133
st2notifier PID: 1167
st2resultstracker PID: 1202
st2rulesengine PID: 1241
st2sensorcontainer PID: 1277
st2chatops is not running.
mistral-server is not running.
mistral-api is not running.

This is dangerous behavior what could cause future problems like race conditions and zombie services.

Probably requires debugging and fixing st2-packages init files for rpm, see:

• https://github.com/StackStorm/st2-packages/blob/master/packages/st2/rpm/st2api.init
• https://github.com/StackStorm/st2-packages/blob/master/packages/st2/rpm/st2auth.init
• https://github.com/StackStorm/st2-packages/blob/master/packages/st2/rpm/st2stream.init

Extend existing ansible-lint rules

Ansible-lint (#52) works well.

In addition to existing ansible-lint rules there is another repo with set of pretty good checks, see:
https://github.com/tsukinowasha/ansible-lint-rules

Add them to existing checks.

General Plan

Ansible deployment repo definitely needs some polishing/testing/improvements for mass-use.

The big point is to polish & publish it on Ansible galaxy (public hub) to get some visibility in the community. Another good thing are real CI Integration tests, for better maintainability.

TODO:

Part 1

• Check for minimum requirements like memory (disk space for Mongo?)
• Version check before install (check if 13.5.99 version is available)
• stable, unstable support (besides of numeric 0.12.1 version)
• Vagrantfile
• Refactor all variables, add st2_ prefix (best practice for Ansible Galaxy)
• Upgrade to higher st2 version as well as downgrade should work
• Fix Upstart
• Upstart N action-runners
• Initial Docs

---

• Support New packages
• Add WebUI installation
• Use external/3rd party Ansible Galaxy roles for PostgreSQL/Mongo/RabbitMQ
• Tag all tasks, split into logical groups
• Idempotence (playbook re-run shouldn't change anything unless new st2 version is available)
• CI tests for better maintainability (Test-kitchen?, CircleCI?)
• Extend list of variables/options to configure
• Ability to configure with existing/remote installations of Mongo, RabbitMQ, PostgreSQL
• Publish on Ansible galaxy #12
  • Polish, Refactor, Follow best practices
  • Distribution
    • Ship it as single stackstorm.stackstorm https://galaxy.ansible.com/StackStorm/stackstorm/
    • Split into: st2, st2mistral, st2web, st2chatops independent roles (v1.0 ??)
  • Ansible Galaxy Docs, Examples
• More distros to support (currently Ubuntu is supported only)

Add logic for 'latest' st2_revision

When numeric st2_version is used, st2_revision is pinned to default val 1, which is not desired beavior.

Instead, when st2_revision is not set: is empty or latest, - let playbook install latest package revision number for respective version.

Optionally configure custom certificate, instead of generating self-signed [st2web]

Adjust st2web role to allow optionally specifying custom cert, instead of generating self-signed one every time.

(feature request) Adding tasks to set up cluster configuration in the RabbitMQ role

This is a much-needed project for us. I want to build StackStorm in production environment with this.

The current tasks in the RabbitMQ role just only install package and start service.
But the RabbitMQ servers that I expect are set up a cluster configuration.

I want some tasks to set up cluster configuration in the RabbitMQ role.

Thank you.

virtualenv is not installed

As reported in: StackStorm/st2contrib#421

CentOS7 CI in Travis

We started adding EL support in #65 and #65 as part of #5.
Working without CI is bad for quality, testing and reviewing.

There is a need to add CentOS7 integration Testing in Travis as we already do for Ubuntu Trusty and Ubuntu Xenial. The framework is already setup.

It's possible to make CentOS7 build non-voting to skip the failure for entire build (see travis.yml example for st2 repo) until we re-write all roles to support new platforms.

See:

• https://travis-ci.org/StackStorm/ansible-st2
• https://github.com/StackStorm/ansible-st2/blob/master/.travis.yml
• https://github.com/StackStorm/ansible-st2/blob/master/.kitchen.yml

cc @Mierdin

Split into `st2`, `st2web`, `st2chatops`

For simplicity reasons, at the moment we keep all the required roles (12) to install/configure StackStorm in one single repository ansible-st2, available under StackStorm.stackstorm Ansible Galaxy, see meta/main.yml workaround.

Alternative is to add more flexibility, distribute everything as different Ansible Galaxy roles (?)

Some arguments:

• StackStorm can be used in High Availability Deployment
• Could be installed/distributed across many hosts
• Can use external RabbitMQ, PostgreSQL, Mongo, ChatOps server
• Not all packages are needed for everyone

Disadvantages:

• Significant breaking change
• Increased complexity & support (we don't need that yet ?)
• Different repos, harder to maintain (?)
• Hard to test "all in one st2" because of components decoupling (CI, PRs)
• Less visibility (give me everything in one place) https://galaxy.ansible.com/StackStorm/stackstorm/

Must set log directory permissions to system user

After an ansible install from https://docs.stackstorm.com/install/ansible.html on a fresh ubuntu 14.04 vm none of the st2 services were started. This is because /var/log/st2 was owned by root:root, and needed to be owned by stanley:stanley ({{ st2_system_user }}).

Support more Linux Distros [EL]

Currently only ubuntu 14/16 installation story is covered as minimum viable example for contributions.

Add support for the following flavors (we build packages for each):

• Ubuntu Trusty
• Ubuntu Xenial
• RHEL/CentOS 7
• RHEL/CentOS 6
• Debian Wheezy
• Debian Jessie

We should add provision logic and integration tests for some of these platform, starting from ubuntu 14.

See notes #15 how we going to test them (Docker + Serverspec way as we do in st2-packages).

Use ansible-lint

https://github.com/willthames/ansible-lint

ansible-lint checks playbooks for practices and behaviour that could potentially be improved

PostgreSQL 3rd party role needs additional sudo flag

Reported in Slack community by @withrocks

mongodb installation failing at task: "Add mongodb key" (xenial)

TASK [mongodb : Add mongodb key] ***********************************************
fatal: [default]: FAILED! => {"changed": false, "cmd": "/usr/bin/apt-key adv --keyserver keyserver.ubuntu.com --recv 42F3E95A2C4F08279C4960ADD68FA50FEA312927", "failed": true, "msg": "Error fetching key 42F3E95A2C4F08279C4960ADD68FA50FEA312927 from keyserver: keyserver.ubuntu.com", "rc": 2, "stderr": "gpg: requesting key EA312927 from hkp server keyserver.ubuntu.com\ngpg: no valid OpenPGP data found.\ngpg: Total number processed: 0\ngpg: keyserver communications error: keyserver unreachable\ngpg: keyserver communications error: public key not found\ngpg: keyserver receive failed: public key not found\n", "stdout": "Executing: /tmp/tmp.MqF3MhC9hn/gpg.1.sh --keyserver\nkeyserver.ubuntu.com\n--recv\n42F3E95A2C4F08279C4960ADD68FA50FEA312927\n?: [fd 4]: read error: Connection reset by peer\ngpgkeys: HTTP fetch error 7: couldn't connect: eof\n", "stdout_lines": ["Executing: /tmp/tmp.MqF3MhC9hn/gpg.1.sh --keyserver", "keyserver.ubuntu.com", "--recv", "42F3E95A2C4F08279C4960ADD68FA50FEA312927", "?: [fd 4]: read error: Connection reset by peer", "gpgkeys: HTTP fetch error 7: couldn't connect: eof"]}
	to retry, use: --limit @/Users/ar/git/ansible-st2/stackstorm.retry

PLAY RECAP *********************************************************************
default                    : ok=1    changed=0    unreachable=0    failed=1

Configure with Existing Mongo, RabbitMQ, PostgreSQL installations

Allow user to install and configure StackStorm when 3rd party services like Mongo, RabbitMQ, MySQL already existing or remote HA clusters.

Think how to make it better.
Should be more clear after solving related #8 #9 #10.

Integration Tests

Before publishing on Ansible Galaxy, add some Integration/Smoke tests, increase infrastructure coverage.

Infrastructure Testing Frameworks

• Evaluate ServerSpec / InSpec - ruby-based. Famous, big community, nice looking and best in his class. But the team prefers Python (st2 main lang).
• Evaluate BATS - Bash Automated Testing System, which is a bit easier.
• Evaluate TestInfra - Python-based infra testing framework. Still pretty young, very small community. Intro.

Automation

• Molecule project - automation for testing Ansible roles
• TestKitchen - automation to spin up Vagrant or Docker or even AWS instance and test Ansible roles with Serverspec/BATS against just spinned up machines.

More info: http://stackoverflow.com/a/37006819/4533625
Continue work started in #7

"Register mistral actions" fails on `staging-unstable`.

Installation of staging-unstable breaks on Ubuntu with the following exception:

Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?

Yes, git IS installed on the box.

After it happens, re-running the playbook skips the Register mistral actions step.

TASK [st2mistral : Register mistral actions] ***********************************
skipping: [st2]

Playbook:

---
- name: Install StackStorm
  hosts: st2
  strategy: debug
  vars:
    st2_pkg_repo: staging-unstable

  roles:
    - mongodb
    - rabbitmq
    - postgresql
    - st2repos
    - st2
    - st2mistral
    - nginx
    - st2web
    - st2smoketests

TASK [st2mistral : Register mistral actions] ***********************************
fatal: [st2]: FAILED! => {"changed": true, "cmd": "/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate && touch /etc/mistral/mistral-db-manage.populate.ansible.has.run", "delta": "0:00:02.174987", "end": "2017-01-28 00:50:31.550711", "failed": true, "rc": 1, "start": "2017-01-28 00:50:29.375724", "stderr": "Traceback (most recent call last):\n  File \"/opt/stackstorm/mistral/bin/mistral-db-manage\", line 11, in <module>\n    sys.exit(main())\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/db/sqlalchemy/migration/cli.py\", line 130, in main\n    CONF.command.func(config, CONF.command.name)\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/db/sqlalchemy/migration/cli.py\", line 71, in do_populate\n    action_manager.sync_db()\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/services/action_manager.py\", line 82, in sync_db\n    register_action_classes()\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/services/action_manager.py\", line 128, in register_action_classes\n    _register_dynamic_action_classes()\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/services/action_manager.py\", line 87, in _register_dynamic_action_classes\n    for generator in generator_factory.all_generators():\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/actions/generator_factory.py\", line 32, in all_generators\n    mod_action_cls = importutils.import_class(mod_cls_name)\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_utils/importutils.py\", line 30, in import_class\n    __import__(mod_str)\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/actions/openstack/actions.py\", line 37, in <module>\n    muranoclient = importutils.try_import('muranoclient.v1.client')\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_utils/importutils.py\", line 103, in try_import\n    return import_module(import_str)\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_utils/importutils.py\", line 73, in import_module\n    __import__(import_str)\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranoclient/v1/client.py\", line 17, in <module>\n    from muranoclient.v1 import artifact_packages\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranoclient/v1/artifact_packages.py\", line 22, in <module>\n    from muranoclient.common import utils\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranoclient/common/utils.py\", line 20, in <module>\n    from muranopkgcheck import manager as check_manager\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranopkgcheck/__init__.py\", line 19, in <module>\n    'muranopkgcheck').version_string()\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/version.py\", line 457, in version_string\n    return self.semantic_version().brief_string()\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/version.py\", line 452, in semantic_version\n    self._semantic = self._get_version_from_pkg_resources()\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/version.py\", line 439, in _get_version_from_pkg_resources\n    result_string = packaging.get_version(self.package)\n  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/packaging.py\", line 671, in get_version\n    raise Exception(\"Versioning for this project requires either an sdist\"\nException: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?", "stdout": "", "stdout_lines": [], "warnings": []}

External role for PostgreSQL

Use external 3rd party Ansible Galaxy role to install PostgreSQL.
Mark it as dependency for st2 role.

Draft v0.6.0 release

• Put together all changes that were done as part of v0.6.0 release
• Document in Release CHANGELOG
• Release v0.6.0

External role for Mongo

Use external 3rd party Ansible Galaxy role for Mongo.
Mark it as dependency for st2 role.