tenforwardconsulting / subspace

Ruby CLI for server maintenance, using Terraform and Ansible

Subspace

Subspace is a rubygem meant to make provisioning as easy as Capistrano makes deploying.

http://tvtropes.org/pmwiki/pmwiki.php/Main/SubspaceAnsible

It is powered by Ansible. Most of the roles require you to configure variables that the role uses.

Installation

First, install ansible (>2.0)

• OSX: brew install ansible
• Linux: apt-get install ansible

Add this line to your application's Gemfile:

gem 'subspace'

Or install it yourself from the command line: $ gem install subspace

Mitogen

[!CAUTION] Mitogen is currently broken! It doesn't support the latest ansible version.

Optionally, you can install a python/pip packaged called "Mitogen" which dramatically speeds up running ansible over ssh. See Here for details.

pip install mitogen

Subspace will try and detect if mitogen is present and use it can. If mitogen causes problems (sometimes it can cause problems depending on the system versions, and particaularly when brand new versions of anible come up and it hasn't updated), you can disable it:

DISABLE_MITOGEN=1 subspace provision env

Usage

subspace init

Initialize the project for subspace. Creates config/subspace with all necessary files.

Subspace 3 supports terraform. You will need to create an IAM user manually with administrative access to the target AWS environment for terraform.

Other Requirements

• JQ
• Terraform
• AWS CLI

Setup AWS Profile

aws configure --profile profile_name

Subspace expects the profile_name to be subspace-{project name}.

init Options

Check cli.rb init for all available options when initializing a new subspace project.

--env

The environment will default to dev unless you pass in --env [env name]

subspace bootstrap <environment>

Ensures the $HOME/.ssh directory is present and ensures python is installed. Python is required to be installed on the remote server for the provision command to work.

subspace provision <environment>

Runs the playbook at config/provision/<environment.yml>.

You can pass certain options through to the ansible-playbook command. See the provision command for the current list.

At the time of this writing, we pass through the ansible-playbook "tags" and "start-at-task" options. The tags option is probably the most useful.

e.g. To run only the alienvault tasks (all of which have been tagged with the 'alienvault' tag): subspace provision staging --tags=alienvault

subspace maintain <environment>

Runs the playbook at config/provision/<environment.yml> but only the tasks tagged with "maintenance".

You can pass certain options through to the ansible-playbook command. See the maintain command for the current list.

At the time of this writing, we pass through the ansible-playbook "limit" option.

e.g. To run only on the host "prod-web1": subspace maintain production --limit=prod-web1

subspace maintenance_mode <environment> --on

This sets up nginx to return all requests as 503 and serve only the static page at /u/apps/{{project_name}}/current/public/maintenance.html which must be checked into your project and deployed to the server.

--on and --off, defaults to off.

Only works for hosts using the nginx role, but you can pass in your entire environment. Running it on worker servers won't hurt anything.

MUST be turned off manually by running subspace maintenance_mode <environment> --off, even a deploy will not disable maintenance mode.

Tagged roles

Role	Tags	Comment
alienvault	alienvault	All tasks in the alienvault role have been tagged 'alienvault'
common	upgrade	runs apt-get update and apt-get upgrade
common	authorized_keys	updates the authorized_keys file for the deploy user
rails	appyml
monit	monit	All tasks in the monit role have been tagged 'monit'

subspace secrets <environment> [--edit] [--create]

The secrets command will manage encrypted secrets for different environments. The default action is simply to show the secrets defined for an environment. Pass --edit to edit them in the system editor (vim, etc).

This uses ansible-vault under the hood and requires a vault password file. You will need to get the .vault_pass from from a teammate out of band (secrets.10fw.ne, 1password, sticky-note, etc), and put it into config/provision/.vault_pass

These secrets are used during provisioning to populate variables in a few different places:

1. config/application.yml, which uses the figaro gem to manage environment variables in rails.
2. config/database.yml, which handles the database connection password.

Subspace uses a template file in config/provision/templates/application.yml.template that contains environment variables for all environments. If you have non-secret variables that change based on the target server, you can simply put that in plaintext in the template file. This was designed so the configuration that is not secret is visible and version controlled, while the secret values are stored in the vault files for their environments.

NOTE: application.yml should be in the .gitignore, since subspace creates a new version on the server and symlinks it on top of whatever is checked in. You should make changes to the template file instead, which should be checked in to version control.

The default template created by subspace init looks like this:

# These environment variables are applied to all environments, and can be secret or not

# This is secret and can be changed on all three environment easily by using subspace vars <env> --edit
SECRET_KEY_BASE: {{secret_key_base}}

# This is not secret, and is the same value for all environments
ENABLE_SOME_FEATURE: false

development:
  INSECURE_VARIABLE: "this isn't secret"

staging:
  INSECURE_VARIABLE: "but it changes"

production:
  INSECURE_VARIABLE: "on different servers"

You can also use this command to automatically create a local version of config/application.yml based on the template and encrypted secrets for a specific environment.

# Create a local copy of config/application.yml with the secrets encrypted in secrets/development.yml
$ subspace vars development --create

This can get you up and running quickly in development securely.

Procedure for updating on projects

When subspace is updated, you should also update it for projects that use it. If subspace is locked to a specific version in the a project's gemfile, you'll need to update that to the new release number, first. Make sure this won't introduce any breaking changes by looking at the changelog.

• bundle update subspace

Then,

• subspace provision production

If you get an error saying you need a vault password file, you need to get it from somoene on the team (see above). You might also need to update ansible.

You'll want to do this for each environment (ie: subspace provision qa, etc.). Best to start with staging and work your way up.

Host configuration

We need to know some info about hosts, but not much. See the files for details, it's mostly the hostname and the user that can administer the system, eg ubuntu on AWS/ubuntu, ec2-user, or even root (not recommended, but used on linode/Digital Ocean)

Role Configuration

This is a description of all the roles that are included by installing subspace, along with their configuration.

common

This role should almost always be there. It ties a bunch of stuff together, runs apt-get update or yum upgrade, sets hostnames, and generally makes the server sane.

project_name: my_project
swap_space: 512M
deploy_user: deploy

Note: we grant the deploy user limited sudo access to run service xyz restart and also add it to the adm group so it can view logs in /var/log.

Roles

This is a description of all the roles that are included by installing subspace, along with their configuration.

alienvault

Fixes CVEs on remote servers by installing updated versions of packages depending on the operating system version.

Each CVE fix is tagged by the CVE name so updates can be targeted to a specific vulnerability.

Variables: None

apache

The most important file for an apache install is the "project.conf" file that gets created in sites-available and symlinked to sites-enabled. This is generated in a sensible way, but if you want to customize it you can do so by setting this variable to anything other than "project.conf":

apache_project_conf: my_custom_configuration.conf

Then place my_custom_configuration.conf in config/provision/templates/my_custom_configuration.conf. This will still get copied to the server as sites-available/{project_name}.conf

Apache also support canonicalizing the domain now, so if you alwyas want to redirect to WWW for example, simply add a variable:

canonical_domain: "www.example.com"

collectd

Collectd is a super useful daemon that grabs and reports statistics about a server's health. Adding this role will make your server start reporting to a graphite server that you specify, and you can make cool graphs and data feeds after that using something like Grafana

graphite_host: graphite.example.com
graphite_port: "2003"

Aside from basic statistics like free memory, disk, load averages, etc, we have some custom things:

1. If Postgres and delayed job are installed, it will collect stats on number of outstanding delayed jobs. a. If you have pg on a different server or in RDS, you can set this manually:

   collectd_pgdj: true

2. If apache is installed, it will collect stats from the /server-status page

3. If nginx is installed, it will collect stats from the "status port"

4. (TODO) add something for pumas

5. (TODO) add something for sidekiq

6. If you're using our standard lograge format, you can enable lograge collection which will provide stats on request count and timers (db/view/total)

   rails_lograge: true

common

delayed_job

Install monitoring and automatic startup for delayed job workers via monit. You MUST set the job_queues variable as follows:

job_queues:
  - default
  - mailers
  - exports

If you want to have multiple workers for a single queue, just add the queue name multiple times:

job_queues:
  - default
  - mailers
  - exports
  - exports
  - exports

Please note that by default, delayed job does not set a queue (eg it uses the "null" queue). You MUST also add an initializer to your rails app where you set the default queue name to "default" (or some other queue). Otherwise, the named queue workers managed by this role will not process the "null" queue.

# config/initializers/delayed_job.rb
Delayed::Worker.default_queue_name = 'default'

Defaults:

delayed_job_command: bin/delayed_job

letsencrypt

This creates a single certificate for every server alias/server name in the configuration file.

letsencrypt_email: "[email protected]"
server_name: app.example.com

If you'd like more control over the cert, you can customize the variable le_ssl_cert as follows:

le_ssl_cert:
  cert_name: "{{server_name}}"
  preferred_challenges: "http"
  plugin: standalone
  domains: "{{ [server_name] + server_aliases }}"

For example, to force a manual DNS challenge you can do the following:

le_ssl_cert:
  cert_name: star_example
  preferred_challenges: dns
  plugin: manual
  domains:
    - example.com
    - "*.example.com"

(you will need to futz around the first time and manually install the DNS record, but it should work on renewals)

Note that this role needs to be included before the webserver (apache or nginx) role

logrotate

Installs logrotate and lets you configure logs for automatic rotation. Example config for rails:

logrotate_scripts:
  - name: rails
    path: "/u/apps/{{project_name}}/shared/log/{{rails_env}}.log"
    options:
      - weekly
      - size 100M
      - missingok
      - compress
      - delaycompress
      - copytruncate

memcache

Installs memcache on the server. By default, memcache will only listen on localhost which needs to be changed if other servers needs to connect.

# Default Value
memcache_bind: "127.0.0.1"

# bind to all interfaces
memcache_bind: "0.0.0.0"

newrelic-infra

This role will install the next-gen "Newrelic One" infrastructure agent which can perform a few different functions for newrelic. The previous "newrelic" role is deprecated.

Variables:

# Required, the newrelic license key you get after signing up.
newrelic_license: "longhashthingyougetfromnewrelichere"
# Optional - send logs to newrelic one's log aggregator.
newrelic_logs:
  - name: rails-production
    path: /u/apps/blah/shared/log/production.log
  - name: nginx-error
    path: /var/log/nginx/error.log

nginx-rails

Configures nginx to look at localhost:9292 for the socket/backend connection. If you need to do fancy stuff you should simply override this role

subspace override nginx-rails

defaults are here, we'll probably add more:

client_max_body_size: 4G
ssl_force_redirect: true
default_server: true
keepalive_timeout: 10
extra_nginx_config: ""

Optional variables:

asset_cors_allow_origin: Set this to set the Access-Control-Allow-Origin for
everything in /assets.

nginx_proxy_read_timeout: Set [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout). This is in seconds. You probably only want to change this if using rack-timeout (although I may be wrong). If using rack-timeout, it should be slightly higher than the rack-timeout timeout. I'm doing 5 seconds higher, but that was arbitrarily chosen.

ssl_force_redirect: redirect all HTTP traffic to HTTPS on the same host.  Defaults to true and only applies if ssl_enabled is also true.
extra_nginx_config: anything else you want to configure in the main nginx config block, formatted like:
extra_nginx_config: |
  proxy_http_version 1.1;
  chunked_transfer_encoding off;
  proxy_buffering off;
  proxy_cache off;

nodejs

Used to install different versions of NodeJS. This uses NodeSource's apt repositories. You must define a variable called nodejs_version and choose a major version supported by NodeSource:

nodejs_version: 14.x
nodejs_version: 17.x
nodejs_version: lts
nodejs_version: current

The full list of distributions is here: https://github.com/nodesource/distributions#installation-instructions

papertrail

passenger

postgresql

Sets up a postgres server - only use this on the database machine.

backups_enabled: true
s3_db_backup_bucket: disabled
s3_db_backup_prefix: "{{project_name}}/{{rails_env}}"
database_user: "{{project_name}}"

puma

Use the puma app server for your rails app. Usually combined with nginx to server as a static file server and reverse proxy.

Prerequesites:

• add gem puma to your gemfile
• add config/puma/ to the linked_dirs config in capistrano's deploy.rb

This role will generate a reasonable puma.rb and configure it to be controlled by systemd.

Variables:

puma_workers: 1        # Puma process count (usually == vCPU count)
puma_min_threads: 4    # Min threads/process
puma_max_threads: 16   # Max threads/process

rails

Provisions for a rails app. This one is probably pretty important.

We no longer provider default values, so make sure to define all the following variables:

rails_env: production
database_pool: 5
database_name: "{{project_name}}_{{rails_env}}"
database_user: "{{project_name}}"
database_host: localhost
database_adapter: postgresql
database_password: # usually defined in the encrypted vault
job_queues:
  - default
  - mailers

redis

Installs redis on the server.

# Change to * if you want this available everywhere instead of localhost
redis_bind: 127.0.0.1

As of Subspace 3.0, this uses the official redis apt repo instead of the debian/ubuntu ones. If you previously had installed redis from the distro, you will need to manually uninstall, purge, and reinstall. This should not delete any data but back it up just in case.

sudo apt-get purge redis-server
sudo apt-get install redis-server

resque

Install monitoring and automatic startup for resque workers via monit. You MUST set the job_queues variable as follows:

job_queues:
  - default
  - mailers
  - exports

redis_bind: "*"

ruby-common

Installs ruby on the machine. YOu can set a version by picking off the download url and sha hash from ruby-lang.org

ruby_version: ruby-2.4.1
ruby_checksum: a330e10d5cb5e53b3a0078326c5731888bb55e32c4abfeb27d9e7f8e5d000250
ruby_download_location: 'https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.1.tar.gz'
bundler_version: 2.0.1

sidekiq

This will install a monit script that keeps sidekiq running. We spawn one sidekiq instance that manages as many queues as you need. Varaibles of note:

# Process these queues on this server
job_queues:
  - default
  - mailers

# Number of sidekiq *processes* to run
sidekiq_workers: 1

• Note that as of v0.4.13, we now also add a unique job queue for each host with its hostname. This is handy if you need to assign a job to a specific host. In general you should use named queues, but occasionally this is useful and there's no harm in having it there unused.

Sidekiq uses redis by default, and rails connects to a redis running on localhost by default. However, this role does not depend on redis since in production it's likely redis will be running elsewhere. If you're provisioning a standalone server, make sure to include the redis role.

Other Internal Roles

Since ansible doesn't support versioning of roles, we cloned the role here so that it doesn't change unexpectedly. We expect to update from upstream occasionally, please let us know if we're missing something we should have.

You should not include these roles directly in your subspace config files. For example, instead of including zenoamaro.postgresql, simply include our postgresql role which depens on zenoamaro's role.

Thanks to the following repositories for making their roles available:

• https://github.com/zenoamaro/ansible-postgresql
• https://github.com/mtpereira/ansible-passenger

Mitogen

In order to dramatically speed up ansible, you can install Mitogen: https://github.com/mitogen-hq/mitogen/blob/master/docs/ansible_detailed.rst

pip install -g mitogen

Subspace will automatically detect this and update your ansible.cfg file so it is blazing fast.

Directory Structure

ansible/roles

Contains all of our custom roles. When the gem is installed and subspace init is ran, the newly created ansible.cfg will be configured to look for these roles.

template

Contains the template files that get copied over when subspace init is ran.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version:

1. update the version number in version.rb
2. gem build subspace.gemspec
3. gem push subspace-x.y.z.gem

This will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/tenforwardconsulting/subspace. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Roles and Variables

subspace's Issues

Certbot dir shouldn't be /ubuntu/home

should be /opt/certbot

ruby-common is broken on new servers

In ruby common main:

- name: Update rubygems
  command: gem update --system
  become: true

.... ... ...

- name: Make Ruby symlinks
  file: path=/usr/local/bin/{{ item }}
        src={{ ruby_location }}/bin/{{ item }}
        state=link
  become: true
  with_items: "{{ ruby_symlinks }}"

The task to run the gem command is before the command that actually makes gem available. This means it will only work on servers that have previously been ran with older versions.

Short term fix is probably check for existence of gem and not run that task if it's not available.

The long-term solution is that there needs to be a way to test subspace against new servers, not just existing ones. I have no idea how to do this. It sounds difficult.

better access for deploy

We should let the deploy user have view permissions to the webserver logs. Maybe add a group or something?

Deploy should also be able to restart apache if it can't already (but I think it can). Anything else? Reboot the machine?

Use --diff by default and save output to directory somewhere.

Also have a way to specify --check (still using --diff) from subspace so you can test changes and see the diff before running for real.

postgresql role doesn't allow configuration of listen addresses

Add `subspace list`

I want subspace list to show all the inventory and groups so its easier to ssh or whatever to them:

$ subspace list
Listing all hosts: 
  [production]
  prod-web1
  prod-web1
  prod-db

or something. It could just cat the hosts/inventory file as a basic impl

I want to pass an argument to subspace to get ansible's verbose logging

You can run ansible-playbook dev.yml -vvvvv and it will output verbose logging.

It should probably pass all extra params straight to ansible or something.

Make `subspace provision` command

The idea is that subspace will work more like git or cap, having multiple commands to use with the subspace executable. One of these should be provision. e.g. subspace dev provision, subspace production provision (same order as cap).

Part of this should be getting rid of the separate playbooks for the environments (since they should be the same anyways). We should have one provision.yml playbook that takes the hosts to run on from the subspace command.

We could even have subspace dev,production provision be a valid command. There's no technical reason that couldn't work.

Fix letsencrypt / nginx install sequence

Get rid of all defaults for variables.

Instead, generate a file with all of them (group_vars/all probably?).

A more unixy thing might be to generate that file and have actual defaults but the file have all the commented ones?

What prompted this was Brett found out subspace defaults to 16 puma threads and a db connection of 5. Being more clear about our defaults I believe is the goal.

Rename current apache role to apache-passenger

since the project.conf template is definitely passenger specific.

The apache role should have a variable to set the ssl paths it reads from.
The letsencrypt role should have a variable to set the path to write to.

Get rid of project-le-ssl.conf template

Instead, copy the non-ssl one and use ansible's lineinfile to add the ssl lines as needed.

subspace init shouldn't use absolute path for roles directory.

In ansible.cfg

Subspace should diff application.yml before and after to make sure you didn't accidentally change/delete values

Because this is totally a thing that happens sometimes. We're usually good about not snowflaking appyml...but this seems like a solid sanity check.

Make `subspace update` command

e.g. subspace dev update, subspace production update, subspace dev,production update.

Should run apt-get update and apt-get upgrade. Probably will have a playbook called update.yml or somesuch that does this. The existing common role already does this, so copy it out of there (don't remove it. common should still update & upgrade, we just want another way to do it without running all of common).

Add `ansible.cfg` to gitignore

Add ansible.cfg to the gitignore template.
https://github.com/tenforwardconsulting/subspace/blob/master/template/provision/.gitignore

Certbot-nginx updated their path for a required file

This file got moved: https://github.com/tenforwardconsulting/subspace/blob/master/ansible/roles/letsencrypt/tasks/main.yml#L63

Per this commit: certbot/certbot@e4af1f3

subspace should write a file or something with the last time it was run

Set batch size with serial so ansible runs individually instead of in parallel against all hosts

---
  - hosts: production
    serial: 1
    become: yes

higher priority for ssh

https://serverfault.com/questions/510611/how-to-prioritize-ssh-to-work-in-the-event-of-high-cpu-load

https://unix.stackexchange.com/questions/209199/how-to-ensure-ssh-via-cgroups-on-centos

Make `subspace status` command.

e.g. subspace production status. Return information such as (but not limited to): memory usage, cpu usage, disk space, number of packages that need updating, ...

Add `*.retry` to git ignore

Add *.retry to the .gitignore template https://github.com/tenforwardconsulting/subspace/blob/master/template/provision/.gitignore

Deprecation Warning on provision

[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is
 'sudo' (default).
This feature will be removed in a future release. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.

http://docs.ansible.com/ansible/become.html

File for automatically using a private key

Put a file somewhere (next to .vault_pass and gitignore it) that each dev can put the path to their copy of a poem file in.

Then it just uses that for the --private-key option

Run `monit reload` on provision

I ran into a problem on BrandScope when I changed the hostname from brand-scope-dev to just brandscope. When I tried to cap dev deploy:restart_delayed_job I got:

$ cap dev deploy:restart_delayed_job
00:00 deploy:restart_delayed_job
      01 sudo monit restart all
      01 monit: action failed -- There is no service by that name
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as [email protected]: sudo monit restart all exit status: 1
sudo monit restart all stdout: Nothing written
sudo monit restart all stderr: monit: action failed -- There is no service by that name

SSHKit::Command::Failed: sudo monit restart all exit status: 1
sudo monit restart all stdout: Nothing written
sudo monit restart all stderr: monit: action failed -- There is no service by that name

Tasks: TOP => deploy:restart_delayed_job
(See full trace by running task with --trace)

I fixed it by sshing in and running sudo monit reload

If delayed_job_queues is empty, it should run the delayed_job binary with no identifier or queue option so that everything is run with the single delayed job process.

See the monitrc file in ansible/roles/delayed_job/templates/

Remove 10fw from host_vars/template.erb

I want a command that creates all the files needed for a new environment

Create group_vars/<env>, host_vars/<env>, vars/<env>.yml, <env>.yml

Maybe a flag to copy from another environment.

Add a line to ~/.profile in production hosts - turn terminal background color bright red

So you know, whenever you SSH in, that you're on a prod host

Subspace Init should get ruby version from .ruby-version file if it exists

It will also need to get the checksum and download locations.

Find download locations here. Use the tar.gz archive.
Find checksums here.
Make sure to get the one for the tar.gz archive.

Support puma >= 5

The daemonize part of puma was removed in puma 5 and Subspace's config doesn't work with it now.

There's a gem here to add it back in: https://github.com/kigster/puma-daemon
It didn't just work as a drop-in replacement when I tried, and I did no further investigating, I just downgraded 😬.

Here's some discussion about the removal: puma/puma#1983

Not sure what the fix is, it could be really simple, but I haven't looked.

AWS Ubuntu 16.04 Instances don't have `python` or `aptitude` installed

I ran into a problem with provisioning stimmi and had to ssh in and manually install python and aptitude

https://gist.github.com/gwillem/4ba393dceb55e5ae276a87300f6b8e6f

This is for python, but can probably easily work for aptitude as well

Checking for "true" (string match) is non-intuitive for ssl_force_redirect

subspace/ansible/roles/nginx-rails/templates/nginx-project-ssl

Line 9 in e1b26e0

{% if ssl_force_redirect == "true" %}

^^ With this check, you have to set ssl_force_redirect to "true" with string quotes.

The variable defaults to true, the boolean value:

subspace/ansible/roles/nginx-rails/defaults/main.yml

Line 3 in e1b26e0

ssl_force_redirect: true

Make `subspace <env> env (set)` command

e.g. subspace production env and subspace production env set to both see the current application.yml and to set application.yml.

Subspace should warn me that ansible isn't installed

Currently if you don't have ansible installed, the output is this:

$ subspace vars dev --edit
>> Running ansible-vault edit vars/dev.yml
<< Done

It should output a message stating that ansible isn't installed.

`init` adds extra block to playbook

  - hosts: localhost
    connection: local
    gather_facts: False

    vars_files:
      - vars/aws.yml

    roles:
      - { role: aws, hostname: stimmi-production }

subspace init adds this to provision/env.yml above the real block that we want.

After `subspace init`, there is an out of date message about how to provision

$ subspace init

    ...
    4. Then provision your server:                                                                    
                                                                                                           
      cd config/provision && ansible-playbook dev.yml 

Enforce running from master only?

Or as a config option at least. The idea being it can be annoying to have to commit subspace changes to prod and merge back in case someone provisions from not-master.

But if we enforce master, then we can commit directly to it and not worry about other branches having old subspace configuration.

Delete all of the passenger SSL stuff.

I think because we're replacing it all with lets encrypt somehow?

CRS Integration

curl --request POST 'http://43c87247.ngrok.io/api/client_stats' \
--header 'X-Api-Version:1' \
--header 'X-Client-Api-Key:d7d4ccdfb0b130952674198c18d51f6a' \
--header 'Content-Type:application/json' \
--data "{\"client_stat\":{ \"key\":\"ruby_version\", \"value\":\"$(ruby -v)\" }}"

I think that should be in a cronjob that subspace adds. Do this when provision runs. Do same for rails version. Get the hostname and also send that up.

Get bundler-audit stats

We should run bundler-audit and report on insecure gems during provision!

Add log rotating configuration

We already rotate backups, we should also have easy configuration for logfiles that should get rotated. Default value should be /u/apps/whatever/production.log

Unattended security updates

https://help.ubuntu.com/community/AutomaticSecurityUpdates

Install Aptitude task fails on brand new hosts

When provisioning a new host for the first time, the install aptitude step fails. Sshing into the server and running sudo apt update fixes the problem, so we may just need to swap the order of some stuff here.

Brett thinks this is related to the Linux version, so maybe we can put some conditional stuff in here?

TASK [common : install aptitude] **************************
fatal: [prod-worker1]: FAILED! => {"changed": false, "msg": "No package matching 'aptitude' is available"}

$ sudo apt install aptitude
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package aptitude is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'aptitude' has no installation candidate

https://github.com/tenforwardconsulting/subspace/blob/master/ansible/roles/common/tasks/main.yml#L67-L73

Provisioning does not always reload monit

When changing the hostname of a host in host_vars/, subspace provision <env> does not always run monit reload to update monit with the new name. This causes errors on deploy when trying to run monit restart all.

Update MOTD template to pull the Subspace version dynamically

Every release, we have to update lib/subspace/version.rb to the new version. We also have to update the message of the day template: ansible/roles/common/templates/motd.

The template should instead dynamically pull the current subspace version at runtime. This is probably possible to do in the same way that the "Last subspace run" time is injected into the motd.

Something like this:

This server brought to you by:
 ____        _    ____
/ ___| _   _| |__/ ___| _ __   __ _  ___ ___
\___ \| | | | '_ \___ \| '_ \ / _` |/ __/ _ \
 ___) | |_| | |_) |__) | |_) | (_| | (_|  __/
|____/ \__,_|_.__/____/| .__/ \__,_|\___\___|
                       |_|             {{subspace_version}}
~~~ https://github.com/tenforwardconsulting/subspace ~~~

If you need to make configuration changes to the server, please modify the
config/provision directory in the app or risk the changes disappearing.
Last subspace run: {{ansible_date_time.iso8601}}

Letsencrypt cron job should run at 4 am

instead of midnight and noon.

authorized_keys should append instead of replace

and arguably it should only be for the deploy user. I think the latter, but at worst it should not overwrite the main ubuntu user's authorized key since it's possible to lock out the user who initially runs provision if their key isn't in the authorized_keys file

Subspace edit vaults command

There should be some command to edit the vaults so I don't have to know how to do that.

So like:

subspace edit dev

instead of:

cd config/provision/
ansible-vault edit vars/dev.yml

zenoamaro role doesn't work for Postgres 11

If you set postgresql_version: 11, postgres installs just fine but does not run. This is due to the postgresql.conf file having a parameter that's since been removed from postgresql, and that causes the service not to start:

-- Logs begin at Wed 2019-09-11 21:12:19 CDT. --
Sep 11 22:51:31 mach-ehr-dev postgresql@11-main[16201]: 2019-09-12 03:51:31.500 GMT [16206] LOG:  unrecognized configuration parameter "sql_inheritance" in file "/etc/postgresql/11/main/postgresql.conf" line 490
Sep 11 22:51:31 mach-ehr-dev postgresql@11-main[16201]: 2019-09-12 03:51:31.500 GMT [16206] FATAL:  configuration file "/etc/postgresql/11/main/postgresql.conf" contains errors
Sep 11 22:51:31 mach-ehr-dev postgresql@11-main[16201]: pg_ctl: could not start server
Sep 11 22:51:31 mach-ehr-dev postgresql@11-main[16201]: Examine the log output.
Sep 11 22:51:31 mach-ehr-dev systemd[1]: [email protected]: Can't open PID file /run/postgresql/11-main.pid (yet?) after start: No such file or directory
Sep 11 22:51:31 mach-ehr-dev systemd[1]: [email protected]: Failed with result 'protocol'.
Sep 11 22:51:31 mach-ehr-dev systemd[1]: Failed to start PostgreSQL Cluster 11-main.
Sep 11 22:51:31 mach-ehr-dev systemd[1]: Starting PostgreSQL RDBMS...
Sep 11 22:51:31 mach-ehr-dev systemd[1]: Started PostgreSQL RDBMS.

See commit : postgres/postgres@e13486e for discussion in postgres.

FIle reference in subspace: https://github.com/tenforwardconsulting/subspace/blob/master/ansible/roles/zenoamaro.postgresql/templates/postgresql.conf#L500