freedomofpress / ansible-role-beats Goto Github PK
View Code? Open in Web Editor NEWAnsible role for installing and configuring elastic beats
Ansible role for installing and configuring elastic beats
Example beats_role_elasticsearch
publishing ES port at 9200
. Let's make this variable (whatever docker decides) and then utilize that value further down the chain and print it out to the local user.
Reasoning to do this is because in the process of writing this role alongside the rest of the ELK stack... I've been hitting a bunch of port conflicts locally between multiple instances of ELK components running in different containers ๐
This role is great - thanks for creating it!
But I can't find this role published on Ansible Galaxy. It would be really nice if it were published there, so requirements.yml
would look like this:
- src: freedomofpress.beats
Instead of what it currently looks like:
- name: freedomofpress.beats
src: https://github.com/freedomofpress/ansible-role-beats.git
This is a crucial step that should come immediately after installing Metricbeat+Filebeat... However, it should really be done on the server which is running ELK and ingesting the inputs, as you need to be authenticated to the ElasticSearch API.
It can be done either via cURL:
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/metricbeat-6.1.2 [email protected]
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/filebeat-6.1.2 [email protected]
Or the beats program(s) provide a facility for doing so:
metricbeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
The changes in abdb5d2 may have introduced a side-effect in the SSL config that breaks writing the filebeat template. The >-
should omit any trailing newlines, and therefore shouldn't break anything, but ยฏ_(ใ)_/ยฏ.
The original CI tests are now defunct just from lack of time working on this repo and from lack of testing on the 6.x/7.x series. Would love to circle back, fully update the tests and re-commit.
I need to shake this out in production - I don't have a lot of data points at the moment but had to temporarily disable in CIEnable
branch.
It passes locally/CI with flying colors... probably because I'm using root for everything in the docker container. The error had something to do with trying to run validation to a spot where the root user did not own the temporary validation config files. This error was coming from beats validation script rather than an ansible error.
To shake this out in CI, probably have to make default docker user non-root and utilize become in the molecule playbook. This would be beneficial for a number of other reasons.
Docs are a bit stale.
ansible-role-logstash-client
Right now the input period
for topbeat is 10 seconds (./templates/topbeat.yml.j2
). With multiple hosts over a length of time, this results in a lot of data filling up the ElasticSearch indices, and the granularity is more than necessary. We should update it to report the system state every minute instead. The period should also become a var so it can be dynamically adjusted.
Right now, the topbeat
service outputs directly to logstash on the logserver, and from there makes it into Elasticsearch. The filebeat
services does the same, but with log files rather than metrics.
The downside to the topbeat
config is that metrics are discarded if not immediately received by logstash. A better solution is the approach taken by filebeat
, which will remember the offset of the last successfully shipped log line and resume there once the connection to logstash is restored.
It should be possible to set the topbeat
config to log to local files, then use filebeat
to monitor those files and ship them into logstash. Then we get the best of both worlds. So:
Then, even in the event of service disruption, we'll still be able to collect and analyze metrics data, rather than having gaps in the metrics config.
This is the new hotness.
Stretch goal of adding in a docker logstash instance to perform an end-to-end test. As part of this effort, strip out the serverspec tests, convert those to testinfra, and add in molecule v2 (still in rc) to replace v1.x
Track entries to /var/log/ufw.log
and ship them to the logstash.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.