freedomofpress / ansible-role-beats

Ansible role for installing and configuring elastic beats

Beats client Ansible role

Ansible role for installing and configuring elastic beats clients. Primarily used for shipping logs and metrics to an ELK stack. By default, this role will ship to logstash on the same box using filebeat and metricbeat.

Requirements

• someplace to ship data to - as of the beats 5.x series this includes shipping to file, kafka, redis, console, elasticsearch, and/or logstash

Role Variables

You'll need an SSL cert to encrypt logs in transit to the logstash/elasticsearch. This logic is not provided by this role and encryption is not enabled by default. You'll need to reference the official beats documentation output guides to add those options via variables.

Example Playbook

- name: Configure beats clients.
  hosts: clients
  roles:
    - role: freedomofpress.beats
  tags: clients

Running the tests

This role uses Molecule and Testinfra for testing. To use it:

pip install -r requirements.txt
molecule test

You can also run selective commands:

molecule idempotence
molecule verify

To fire up an elasticsearch UI for debugging, run:

make elastic-ui

See the Molecule docs for more info.

Contributions

The following resources were invaluable in creating this role.

• geerlingguy.logstash
• DigitalOcean's ELK guide
• filebeat configuration guide

License

MIT

ansible-role-beats's Issues

More resilient metrics logging

Right now, the topbeat service outputs directly to logstash on the logserver, and from there makes it into Elasticsearch. The filebeat services does the same, but with log files rather than metrics.

The downside to the topbeat config is that metrics are discarded if not immediately received by logstash. A better solution is the approach taken by filebeat, which will remember the offset of the last successfully shipped log line and resume there once the connection to logstash is restored.

It should be possible to set the topbeat config to log to local files, then use filebeat to monitor those files and ship them into logstash. Then we get the best of both worlds. So:

• disable logstash output in topbeat config
• enable local logfiles in topbeat config
• enable logging of logfiles for topbeat in filebeat config
• write a JSON logstash parser for ingesting the topbeat metrics

Then, even in the event of service disruption, we'll still be able to collect and analyze metrics data, rather than having gaps in the metrics config.

Fix validation of configs

I need to shake this out in production - I don't have a lot of data points at the moment but had to temporarily disable in CIEnable branch.

It passes locally/CI with flying colors... probably because I'm using root for everything in the docker container. The error had something to do with trying to run validation to a spot where the root user did not own the temporary validation config files. This error was coming from beats validation script rather than an ansible error.

To shake this out in CI, probably have to make default docker user non-root and utilize become in the molecule playbook. This would be beneficial for a number of other reasons.

Replace topbeat with metricbeat

This is the new hotness.

Check default vars for SSL info

The changes in abdb5d2 may have introduced a side-effect in the SSL config that breaks writing the filebeat template. The >- should omit any trailing newlines, and therefore shouldn't break anything, but ¯_(ツ)_/¯.

Refactor CircleCI tests

The original CI tests are now defunct just from lack of time working on this repo and from lack of testing on the 6.x/7.x series. Would love to circle back, fully update the tests and re-commit.

Topbeat event frequency

Right now the input period for topbeat is 10 seconds (./templates/topbeat.yml.j2). With multiple hosts over a length of time, this results in a lot of data filling up the ElasticSearch indices, and the granularity is more than necessary. We should update it to report the system state every minute instead. The period should also become a var so it can be dynamically adjusted.

Stop publishing docker container ports to static ports

Example beats_role_elasticsearch publishing ES port at 9200. Let's make this variable (whatever docker decides) and then utilize that value further down the chain and print it out to the local user.

Reasoning to do this is because in the process of writing this role alongside the rest of the ELK stack... I've been hitting a bunch of port conflicts locally between multiple instances of ELK components running in different containers 😖

Update documentation

Docs are a bit stale.

• copy current default vars to README
• rename repo -> ansible-role-logstash-client
• link to related roles
• add to galaxy

Load the index templates / mappings

This is a crucial step that should come immediately after installing Metricbeat+Filebeat... However, it should really be done on the server which is running ELK and ingesting the inputs, as you need to be authenticated to the ElasticSearch API.

It can be done either via cURL:

curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/metricbeat-6.1.2 [email protected]
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/filebeat-6.1.2 [email protected]

Or the beats program(s) provide a facility for doing so:

metricbeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

Publish to Ansible Galaxy

This role is great - thanks for creating it!

But I can't find this role published on Ansible Galaxy. It would be really nice if it were published there, so requirements.yml would look like this:

- src: freedomofpress.beats

Instead of what it currently looks like:

- name: freedomofpress.beats
  src: https://github.com/freedomofpress/ansible-role-beats.git

Add ufw logs by default

Track entries to /var/log/ufw.log and ship them to the logstash.

Mixin in CI to this repo

Stretch goal of adding in a docker logstash instance to perform an end-to-end test. As part of this effort, strip out the serverspec tests, convert those to testinfra, and add in molecule v2 (still in rc) to replace v1.x