stackstorm / ansible-st2 Goto Github PK
View Code? Open in Web Editor NEWAnsible Roles and Playbooks to deploy StackStorm
Home Page: https://galaxy.ansible.com/StackStorm/stackstorm/
License: Apache License 2.0
Ansible Roles and Playbooks to deploy StackStorm
Home Page: https://galaxy.ansible.com/StackStorm/stackstorm/
License: Apache License 2.0
https://github.com/willthames/ansible-lint
ansible-lint checks playbooks for practices and behaviour that could potentially be improved
The "Lookup literal version" task is failing silently with "sort: unrecognized option `--version-sort'"
Leaving _st2_version set to "" and breaking following tasks
Use external 3rd party Ansible Galaxy role to install PostgreSQL.
Mark it as dependency for st2
role.
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will be removed in a future release.
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{postgresql_env}}'). This feature
will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
For better maintainability add automated tests.
Create a set of common rules, recommendations and conventions for development flow when contributing to ansible-st2
.
Similar to Pack Install #74 we'll need an abstraction for st2
key/value store.
For example define a list of K/V pairs in yaml
which will be added in st2
datastore during Ansible provisioning.
See: https://docs.stackstorm.com/datastore.html
At a low level it could be implemented via Ansible plugins/modules (lookup/set):
According to https://docs.stackstorm.com/install/upgrades.html
Mistral DB upgrading may be required after every st2mistral
package update:
# Stop related services
service mistral-api stop
service mistral stop
# Upgrade database
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate
# Restart related services
service mistral start
service mistral-api start
st2mistral
rolestart/stop
is a required stepAs reported in: StackStorm/st2contrib#421
After an ansible install from https://docs.stackstorm.com/install/ansible.html on a fresh ubuntu 14.04 vm none of the st2 services were started. This is because /var/log/st2 was owned by root:root, and needed to be owned by stanley:stanley ({{ st2_system_user }}).
Ansible deployment repo definitely needs some polishing/testing/improvements for mass-use.
The big point is to polish & publish it on Ansible galaxy (public hub) to get some visibility in the community. Another good thing are real CI Integration tests, for better maintainability.
TODO:
memory
(disk
space for Mongo?)13.5.99
version is available)stable
, unstable
support (besides of numeric 0.12.1
version)st2_
prefix (best practice for Ansible Galaxy)N
action-runnersstackstorm.stackstorm
https://galaxy.ansible.com/StackStorm/stackstorm/st2
, st2mistral
, st2web
, st2chatops
independent roles (v1.0
??)Use external 3rd party Ansible Galaxy role for RabbitMQ (or create own).
Mark it as dependency for st2
role.
This is a much-needed project for us. I want to build StackStorm in production environment with this.
The current tasks in the RabbitMQ role just only install package and start service.
But the RabbitMQ servers that I expect are set up a cluster configuration.
I want some tasks to set up cluster configuration in the RabbitMQ role.
Thank you.
We started adding EL
support in #65 and #65 as part of #5.
Working without CI
is bad for quality, testing and reviewing.
There is a need to add CentOS6
integration Testing in Travis as we already do for Ubuntu Trusty
and Ubuntu Xenial
. The framework is already setup.
It's possible to make CentOS6
build non-voting to skip the failure for entire build (see travis.yml
example for st2
repo) until we re-write all roles to support new platforms.
See:
cc @humblearner
Include st2chatops
package installation
Before publishing on Ansible Galaxy, add some Integration/Smoke tests, increase infrastructure coverage.
More info: http://stackoverflow.com/a/37006819/4533625
Continue work started in #7
Installation of staging-unstable
breaks on Ubuntu with the following exception:
Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?
Yes, git IS installed on the box.
After it happens, re-running the playbook skips the Register mistral actions step.
TASK [st2mistral : Register mistral actions] ***********************************
skipping: [st2]
Playbook:
---
- name: Install StackStorm
hosts: st2
strategy: debug
vars:
st2_pkg_repo: staging-unstable
roles:
- mongodb
- rabbitmq
- postgresql
- st2repos
- st2
- st2mistral
- nginx
- st2web
- st2smoketests
TASK [st2mistral : Register mistral actions] ***********************************
fatal: [st2]: FAILED! => {"changed": true, "cmd": "/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate && touch /etc/mistral/mistral-db-manage.populate.ansible.has.run", "delta": "0:00:02.174987", "end": "2017-01-28 00:50:31.550711", "failed": true, "rc": 1, "start": "2017-01-28 00:50:29.375724", "stderr": "Traceback (most recent call last):\n File \"/opt/stackstorm/mistral/bin/mistral-db-manage\", line 11, in <module>\n sys.exit(main())\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/db/sqlalchemy/migration/cli.py\", line 130, in main\n CONF.command.func(config, CONF.command.name)\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/db/sqlalchemy/migration/cli.py\", line 71, in do_populate\n action_manager.sync_db()\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/services/action_manager.py\", line 82, in sync_db\n register_action_classes()\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/services/action_manager.py\", line 128, in register_action_classes\n _register_dynamic_action_classes()\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/services/action_manager.py\", line 87, in _register_dynamic_action_classes\n for generator in generator_factory.all_generators():\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/actions/generator_factory.py\", line 32, in all_generators\n mod_action_cls = importutils.import_class(mod_cls_name)\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_utils/importutils.py\", line 30, in import_class\n __import__(mod_str)\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/actions/openstack/actions.py\", line 37, in <module>\n muranoclient = importutils.try_import('muranoclient.v1.client')\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_utils/importutils.py\", line 103, in try_import\n return import_module(import_str)\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_utils/importutils.py\", line 73, in import_module\n __import__(import_str)\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranoclient/v1/client.py\", line 17, in <module>\n from muranoclient.v1 import artifact_packages\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranoclient/v1/artifact_packages.py\", line 22, in <module>\n from muranoclient.common import utils\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranoclient/common/utils.py\", line 20, in <module>\n from muranopkgcheck import manager as check_manager\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/muranopkgcheck/__init__.py\", line 19, in <module>\n 'muranopkgcheck').version_string()\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/version.py\", line 457, in version_string\n return self.semantic_version().brief_string()\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/version.py\", line 452, in semantic_version\n self._semantic = self._get_version_from_pkg_resources()\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/version.py\", line 439, in _get_version_from_pkg_resources\n result_string = packaging.get_version(self.package)\n File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/pbr/packaging.py\", line 671, in get_version\n raise Exception(\"Versioning for this project requires either an sdist\"\nException: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?", "stdout": "", "stdout_lines": [], "warnings": []}
From the:
At the moment we don't even configure any username/passwords for external services like Mongo, RabbitMQ with st2 Ansible installation. Additionally, we're not sure whether those services are listening on 127.0.0.1
or are open to public net.
All of that needs better control and improvement.
host
for dependent services127.0.0.1
(PostgreSQL, Mongo, RabbitMQ) by defaultst2.conf
Add option to install additional StackStorm packs within Ansible installer.
Good feature, as seen at puppet-st2
. Low priority, after v0.3.0
.
mistral
to st2mistral
Depends on PostgreSQL Role [EL] #76
In testing EL7, I encountered this issue when running the Ansible playbook:
TASK [mongodb : yum | Install mongodb dependencies] ****************************
failed: [10.0.0.96] (item=[u'python-urllib3', u'pyOpenSSL', u'python-pyasn1', u'python-ndg_httpsclient']) => {"failed": true, "item": ["python-urllib3", "pyOpenSSL", "python-pyasn1", "python-ndg_httpsclient"], "msg": "Failure talking to yum: failure: repodata/repomd.xml from stackstorm: [Errno 256] No more mirrors to try.\nhttps://packagecloud.io/StackStorm/stable/el/7Server/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found"}
to retry, use: --limit @/Users/oswaltm/Code/ansible-st2/stackstorm.retry
This is because the $releasever
variable in the yum configuration evaluates to 7Server
when on EL7 (whereas it equals simply 7
on CentOS7).
[ec2-user@ip-10-0-0-96 ~]$ cat /etc/yum.repos.d/st2.repo
[stackstorm]
baseurl = https://packagecloud.io/StackStorm/stable/el/$releasever/$basearch
name = Stackstorm Repo
[ec2-user@ip-10-0-0-96 ~]$ python -c 'import yum, pprint; yb = yum.YumBase(); pprint.pprint(yb.conf.yumvar, width=1)'
(...truncated...)
{'arch': 'ia32e',
'basearch': 'x86_64',
'releasever': '7Server',
'uuid': '8b7c566e-243b-4985-a077-8e6dbeed7e16'}
This is expected behavior.
The question is, how should this get answered? Should we deal with this on the Ansible side, rewriting this config to include the actual hard-coded version like 7
, instead of 7Server
, or should we make a change on the packagecloud side that includes the expected path?
Use external 3rd party Ansible Galaxy role for Mongo.
Mark it as dependency for st2
role.
Draft a StackStorm blog for ansible-st2
v0.7.0 release, highlight breaking changes, new features with usage examples and new roles like @lakshmi-kannan's bwc
and @humblearner's st2chatops
.
Reported in Slack community by @withrocks
Implement a convenient ansible-friendly way (as Ansible module) to install st2 packs, as well as provide configurations/settings for them.
Decide:
yaml
plays inline/opt/stackstorm/configs/<pack>.yaml
https://docs.stackstorm.com/pack_configs.html
Alternative example from
puppet-st2
: https://github.com/stackstorm/puppet-st2#installing-and-configuring-packs
In short, it should work like similar pip or gem Ansible module, but for installing st2 packs.
Second stackstorm.yml
playbook re-run should end with the following: changed=0.*failed=0
, meaning 0 resources changed.
Include Idempotence check in CI, see:
https://github.com/StackStorm/ansible-st2/blob/master/.kitchen.yml#L17-L18
Add new features/changed vars from the v0.7.0 in
https://github.com/StackStorm/st2docs/blob/master/docs/source/install/ansible.rst docs.
Adjust st2web
role to allow optionally specifying custom cert, instead of generating self-signed one every time.
st2web
like hostname
When numeric st2_version
is used, st2_revision
is pinned to default val 1
, which is not desired beavior.
Instead, when st2_revision
is not set: is empty or latest
, - let playbook install latest package revision number for respective version.
Add tasks to install StackStorm WebUI.
Options:
st2_deploy.sh
approach. Install by default, but add variable to opt-out: st2_install_webui
.st2-puppet
approach. Install webui
in any case without option to opt-out.Resources:
We started adding EL
support in #65 and #65 as part of #5.
Working without CI
is bad for quality, testing and reviewing.
There is a need to add CentOS7
integration Testing in Travis as we already do for Ubuntu Trusty
and Ubuntu Xenial
. The framework is already setup.
It's possible to make CentOS7
build non-voting to skip the failure for entire build (see travis.yml
example for st2
repo) until we re-write all roles to support new platforms.
See:
cc @Mierdin
Since we have own postgresql
role, - let it handle DB creation for st2mistral
.
Use native Ansible modules postgresql_user
and postgresql_db
instead of init_mistral_db
SQL template file.
So after all postgresql
+ st2mistral
usage should look as following:
- name: Install st2mistral on a single node
hosts: all
roles:
- name: Install PostgreSQL and configure databases
role: postgresql
vars:
postgresql_databases:
- name: "{{ st2mistral_db }}"
postgresql_users:
- name: "{{ st2mistral_db_username }}"
pass: "{{ st2mistral_db_password }}"
encrypted: yes
- name: Install and configure StackStorm Mistral
role: st2mistral
vars:
st2mistral_database_connection: postgresql://{{ st2mistral_db_username }}:{{ st2mistral_db_password }}@localhost/{{ st2mistral_db }}
Use best practices from https://github.com/ANXS/postgresql
Additionally, this will allow us to manage multi-node deployments #17 better in future.
This happens on EL6
(reproducible in CentOS6 Vagrant).
st2api
, st2stream
, st2auth
got duplicated:
$ sudo st2ctl status
##### st2 components status #####
st2actionrunner PID: 699
st2actionrunner PID: 720
st2actionrunner PID: 740
st2actionrunner PID: 763
st2actionrunner PID: 789
st2actionrunner PID: 814
st2actionrunner PID: 841
st2actionrunner PID: 868
st2actionrunner PID: 896
st2actionrunner PID: 936
st2api PID: 29766
st2api PID: 29778
st2stream PID: 1034
st2stream PID: 1057
st2auth PID: 1085
st2auth PID: 1096
st2garbagecollector PID: 1133
st2notifier PID: 1167
st2resultstracker PID: 1202
st2rulesengine PID: 1241
st2sensorcontainer PID: 1277
st2chatops is not running.
mistral-server is not running.
mistral-api is not running.
This is dangerous behavior what could cause future problems like race conditions and zombie services.
Probably requires debugging and fixing st2-packages
init files for rpm
, see:
Follow st2 mistral changes:
Generate passwords for root
and mistral
PostgreSQL users
Currently only ubuntu 14/16
installation story is covered as minimum viable example for contributions.
Add support for the following flavors (we build packages for each):
We should add provision logic and integration tests for some of these platform, starting from ubuntu 14
.
See notes #15 how we going to test them (Docker + Serverspec way as we do in st2-packages
).
TravisCI is red for forked PRs, starting from st2chatops
and bwc
role implementations.
This is caused by private env variables defined in our Travis settings: HUBOT_SLACK_TOKEN
and BWC_LICENSE
which are not available to forked builds.
https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings
Similarly, we do not provide these values to untrusted builds, triggered by pull requests from another repository.
Find the way how to proceed this the right way in future, - because we can't turn down CI for contribution PRs.
External dependency ANXS.postgresql doesn't support EL platform family.
We need to write custom role to install PostgreSQL
for Mistral.
ANXS.postgresql
code127.0.0.1
by default (configurable), see: #75cc @humblearner based on your feedback
Since StackStorm v1.6
MongoDB 3.2
is the default version,
see: https://docs.stackstorm.com/install/deb.html#install-dependencies for installation instructions.
Reflect the change in st2 Configuration Management repos.
Fix the builds (since: https://circleci.com/gh/StackStorm/ansible-st2/31), check what's new in recent Ansible 2.0
version which broke the build, verify/test entire playbook story, check the backward compatibility.
we'll probably have to pin to a specific version of stackstorm, but the revision is less important for us.
It'd be good to be able to set st2_version=2.2.1 st2_revision=latest to pin at whatever the latest 2.2.1 release is without having to know the exact revisions that've been released in advance
Since we started adding support for more platforms as part of #5 in #65 and #66 We'll need a way to bootstrap Vagrant for:
Example syntax:
vagrant up ubuntu14
vagrant up ubuntu16
vagrant up centos6
vagrant up centos7
Implementation examples:
cc @Mierdin and @humblearner
Good "sign of quality" objective is to publish StackStorm roles on Ansible galaxy to get some more attention from the community.
At that point it should be tested/stable enough.
stackstorm.st2
role or st2
, st2web
, st2mistral
, st2chatops
rolespostgresql
, rabbitmq
, mongo
For simplicity reasons, at the moment we keep all the required roles (12) to install/configure StackStorm in one single repository ansible-st2
, available under StackStorm.stackstorm
Ansible Galaxy, see meta/main.yml
workaround.
Alternative is to add more flexibility, distribute everything as different Ansible Galaxy roles (?)
Ansible-lint (#52) works well.
In addition to existing ansible-lint
rules there is another repo with set of pretty good checks, see:
https://github.com/tsukinowasha/ansible-lint-rules
Add them to existing checks.
TASK [mongodb : Add mongodb key] ***********************************************
fatal: [default]: FAILED! => {"changed": false, "cmd": "/usr/bin/apt-key adv --keyserver keyserver.ubuntu.com --recv 42F3E95A2C4F08279C4960ADD68FA50FEA312927", "failed": true, "msg": "Error fetching key 42F3E95A2C4F08279C4960ADD68FA50FEA312927 from keyserver: keyserver.ubuntu.com", "rc": 2, "stderr": "gpg: requesting key EA312927 from hkp server keyserver.ubuntu.com\ngpg: no valid OpenPGP data found.\ngpg: Total number processed: 0\ngpg: keyserver communications error: keyserver unreachable\ngpg: keyserver communications error: public key not found\ngpg: keyserver receive failed: public key not found\n", "stdout": "Executing: /tmp/tmp.MqF3MhC9hn/gpg.1.sh --keyserver\nkeyserver.ubuntu.com\n--recv\n42F3E95A2C4F08279C4960ADD68FA50FEA312927\n?: [fd 4]: read error: Connection reset by peer\ngpgkeys: HTTP fetch error 7: couldn't connect: eof\n", "stdout_lines": ["Executing: /tmp/tmp.MqF3MhC9hn/gpg.1.sh --keyserver", "keyserver.ubuntu.com", "--recv", "42F3E95A2C4F08279C4960ADD68FA50FEA312927", "?: [fd 4]: read error: Connection reset by peer", "gpgkeys: HTTP fetch error 7: couldn't connect: eof"]}
to retry, use: --limit @/Users/ar/git/ansible-st2/stackstorm.retry
PLAY RECAP *********************************************************************
default : ok=1 changed=0 unreachable=0 failed=1
v0.6.0
releasev0.6.0
Existing MySQL role fails on some Ubuntu installations.
TASK: [mysql | Change MySQL password] *****************************************
<localhost> REMOTE_MODULE mysql_user login_password=VALUE_HIDDEN name=root password=VALUE_HIDDEN login_user=root
<localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569 && echo $HOME/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569']
<localhost> PUT /tmp/tmp4WsWvG TO /home/kitchen/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569/mysql_user
<localhost> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/kitchen/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569/mysql_user; rm -rf /home/kitchen/.ansible/tmp/ansible-tmp-1440013362.89-221545053412569/ >/dev/null 2>&1']
failed: [localhost] => {"failed": true}
msg: unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials
FATAL: all hosts have already failed -- aborting
localhost : ok=11 changed=9 unreachable=0 failed=1
Most probably using 3rd party Ansible Galaxy role for MySQL #8 will fix it.
Initial Minimum Viable implementation for installation of new packages was done in #42.
Real production use requires much more settings to configure.
unstable
repo (?)st2.conf
parameters #121A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.