geerlingguy / ansible-role-filebeat

Ansible Role - Filebeat for ELK stack

Ansible Role: Filebeat for ELK Stack

An Ansible Role that installs Filebeat on RedHat/CentOS or Debian/Ubuntu.

Requirements

None.

Role Variables

Available variables are listed below, along with default values (see defaults/main.yml):

filebeat_version: 7.x

Controls the major version of Filebeat which is installed.

filebeat_package: filebeat
filebeat_package_state: present

The specific package to be installed. You can specify a version of the package using the correct syntax for your platform and package manager by changing the package name. You can also control the package state (e.g. present, absent, or latest).

filebeat_create_config: true

Whether to create the Filebeat configuration file and handle the copying of SSL key and cert for filebeat. If you prefer to create a configuration file yourself you can set this to false.

filebeat_inputs:
  - type: log
    paths:
      - "/var/log/*.log"

Inputs that will be listed in the inputs section of the Filebeat configuration. Read through the Filebeat Inputs configuration guide for more options.

filebeat_output_elasticsearch_enabled: false
filebeat_output_elasticsearch_hosts:
  - "localhost:9200"

Whether to enable Elasticsearch output, and which hosts to send output to.

filebeat_output_elasticsearch_auth:
    username: "admin"
    password: "S3CR3eeet"

# or, only for version 8+
filebeat_output_elasticsearch_auth:
    api_key: "xa-123a-f3ea012d-aaae1"

Configures the authentication for the elasticsearch output. Note that api_key and user/pass are mutually exclusive, and api_key is only available from version 8.

filebeat_output_logstash_enabled: true
filebeat_output_logstash_hosts:
  - "localhost:5000"

Whether to enable Logstash output, and which hosts to send output to.

filebeat_enable_logging: false
filebeat_log_level: warning
filebeat_log_dir: /var/log/filebeat
filebeat_log_filename: filebeat.log

Filebeat logging.

filebeat_ssl_certs_dir: /etc/pki/logstash
filebeat_ssl_private_dir: "{{ filebeat_ssl_certs_dir }}"

The path where certificates and keyfiles will be stored.

filebeat_ssl_ca_file: ""
filebeat_ssl_certificate_file: ""
filebeat_ssl_key_file: ""

Local paths to the SSL certificate and key files.

filebeat_ssl_copy_file: true

Wether to copy certificate and key into the filebeat_ssl_dir, or use existing ones.

For utmost security, you should use your own valid certificate and keyfile, and update the filebeat_ssl_* variables in your playbook to use your certificate.

To generate a self-signed certificate/key pair, you can use use the command:

$ sudo openssl req -x509 -batch -nodes -days 3650 -newkey rsa:2048 -keyout filebeat.key -out filebeat.crt

Note that filebeat and logstash may not work correctly with self-signed certificates unless you also have the full chain of trust (including the Certificate Authority for your self-signed cert) added on your server. See: elastic/logstash#4926 (comment)

filebeat_ssl_insecure: "false"

Set this to "true" to allow the use of self-signed certificates (when a CA isn't available).

Overriding the filebeat template

If you can't customize via variables because an option isn't exposed, you can override the template used to generate the filebeat configuration.

filebeat_template: "filebeat.yml.j2"

You can either copy and modify the provided template, or you can, for example, point to a template file in your playbook directory that will be used instead of the managed template.

filebeat_template: "{{ playbook_dir }}/templates/filebeat.yml.j2"

Dependencies

None.

Example Playbook

- hosts: logs

- name: Set the java_packages variable (Debian/Ubuntu).
  set_fact:
    java_packages:
      - openjdk-8-jdk
  when: ansible_os_family == 'Debian'

  roles:
    - geerlingguy.java
    - geerlingguy.elasticsearch
    - geerlingguy.logstash
    - geerlingguy.filebeat

License

MIT / BSD

Author Information

This role was created in 2016 by Jeff Geerling, author of Ansible for DevOps.

ansible-role-filebeat's Issues

Error including SSL certificate

Ansible 2.2.1.0 (Ubuntu 16.04)

When running ansible-role-filebeat with an filebeat_ssl_dir set - I get the following error:

TASK [geerlingguy.filebeat : Ensure Filebeat SSL key pair directory exists.] ***
fatal: [localhost]: FAILED! => {"failed": true, "msg": "The conditional check 'filebeat_ssl_key_file' failed. The
error was: error while evaluating conditional (filebeat_ssl_key_file): 'filebeat' is undefined\n\nThe error
appears to have been in '/etc/ansible/roles/geerlingguy.filebeat/tasks/config.yml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to
be:\n\n\n- name: Ensure Filebeat SSL key pair directory exists.\n ^ here\n"}

everything is pretty much standard - with my vars/ssl.yml configured like this:

filebeat_ssl_dir: '/etc/pki/graylog2'
filebeat_ssl_certificate_file: 'filebeat.crt'
filebeat_ssl_key_file: 'filebeat.key'

If I edit config.yml manually and alter the "when" on the Ensure Filebeat SSL key pair directory exists. step from when: filebeat_ssl_key_file to when: filebeat_ssl_key_file is defined then the script completes correctly.

Have I hit a regression bug - or am I doing something fundamentally silly? ;-)

broken configurations

Host machine os: CentOS Linux release 7.6.1810 (Core)
Filebeat package version: filebeat-6.8.0-1.x86_64

Example configuration:

---
- hosts: demohost

  roles:
    - role: geerlingguy.filebeat
      vars:
        filebeat_output_elasticsearch_enabled: true
        filebeat_output_elasticsearch_hosts:
          - "10.1.1.199:9200"
        filebeat_prospectors:
          - input_type: log
            paths:
              - "/var/log/*.log"

Playbook runs without issue. However resulting configuration fails:

[root@elkclient ~]# filebeat test config
Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'/etc/filebeat/filebeat.yml')

Format does not seem to be matching the source configuration at-all.

I'd suggest adding the "filebeat test config" command as a step when filebeat_create_config is set to true. This allows the playbook to fail when the configuration is managed and it fails.

Adapt new pipeline configuration for filebeats 6.3

Prospectors are deprecated and renamed to inputs in 6.3.0 (Release note for 6.3.0 or more specifically the PR), and further removed in 7.0.
So some version checking might be required in the filebeat.yml.j2.

Allow configuration of filebeat major version

Would like to add a variable like filebeat_version: 6.x.

repos are old

The filebeat installed by this role is rather ancient. I've updated the repos on my local copy of this role.

repo: 'deb https://artifacts.elastic.co/packages/5.x/apt stable main'

baseurl=https://artifacts.elastic.co/packages/5.x/yum

ubuntu container seems not to be run on centos7

cleanup=false container_id=$(date +%s) distro=centos7 ./tests/geerlingguy.test/ansible-role-test.sh

but when

cleanup=false container_id=$(date +%s) distro=ubuntu1604 ./tests/geerlingguy.test/ansible-role-test.sh

then a ubuntu login screen appears and I have to login again in my centos7 gnome machine

Allow to skip config.yml

I prefer to generate the config file myself.
Will you accept a PR for skipping the config tasks, e.g. with filebeat_skip_config variables whose default value is no?

Copy CA file during SSL configuration as well

Shouldn't the CA file be copied as well?

https://github.com/geerlingguy/ansible-role-filebeat/blob/master/tasks/config.yml#L21

I specified the following:

filebeat_ssl_certs_dir: /etc/filebeat
filebeat_ssl_private_dir: /etc/filebeat
filebeat_ssl_ca_cert_file: ca-cert.pem
filebeat_ssl_certificate_file: mycert.pem
filebeat_ssl_key_file: mykey.pem
filebeat_ssl_copy_file: true

And the resulting config part is:

ssl:
  certificate_authorities: ["/etc/filebeat/ca-cert.pem"]
  certificate: "/etc/filebeat/mycert.pem"
  key: "/etc/filebeat/mykey.pem"

But ca-cert.pem does not exist in /etc/filebeat

Role fails to install Filebeat 8.x

This role throws errors for me on the Add Filebeat repository step under Ubuntu:

fatal: [seed-1]: FAILED! => changed=false 
  invocation:
    module_args:
      codename: null
      filename: null
      install_python_apt: true
      mode: null
      repo: deb https://artifacts.elastic.co/packages/8.3.3/apt stable main
      state: present
      update_cache: true
      update_cache_retries: 5
      update_cache_retry_max_delay: 12
      validate_certs: true
  msg: 'Failed to update apt cache: E:The repository ''https://artifacts.elastic.co/packages/8.3.3/apt stable Release'' does not have a Release file.'

According to the Filebeat docs, the correct string to use for this is probably:

deb https://artifacts.elastic.co/packages/8.x/apt stable main

How to pass "validate_certs=false"?

Getting following error messages. modifying "setup-RedHat.yml" with additional parameter named "validate_certs: false" solves the issue but how to pass this same variable using ansible playbook?

fatal: [deployTarget]: FAILED! => { "changed": false, "invocation": { "module_args": { "fingerprint": null, "key": "https://packages.elastic.co/GPG-KEY-elasticsearch", "state": "present", "validate_certs": true } }, "msg": "failed to fetch key at https://packages.elastic.co/GPG-KEY-elasticsearch , error was: Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>" }

Support Elastic Cloud

Would you be interested in a PR that added support for shipping logs to Elastic Cloud?

I think the changes would be fairly minimal per the docs. It looks like there would be cloud.id and cloud.auth in the template config and then a couple of conditions:

When specified, the cloud.auth overwrites the output.elasticsearch.username and output.elasticsearch.password settings.

This setting [cloud.id] overwrites the output.elasticsearch.hosts and setup.kibana.host settings.

3.4.1 release CI failed

Hi, your recent release of 3.4.1 triggered CI correctly, but it timed out and terminated with a green status despite not having released the tag on Galaxy. Can you check please? #62 (comment)

Saw you on Youtube recently btw, hope you are doing well and wish you all the best from Australia! Thanks for all your Ansible, RPi and Docker tools.

Allow people to pass their own filebeat config file

Hello,

You could just allow passing the filebeat.yml file via the playbook. Matching all possible variables or trying to externalize all the filebeat config to allow parametrizing from the playbook as "vars" is going to be very hard.

A very easy way could be doing something like:

- name: get file stat to be able to perform a check in the following tasks
  local_action: stat path={{ playbook_dir }}/filebeat/filebeat.yml.j2
  register: file

- name: Copy Filebeat configuration.
  template:
    src: "filebeat.yml.j2"
    dest: "/etc/filebeat/filebeat.yml"
    owner: root
    group: root
    mode: 0644
  notify: restart filebeat
  when: not file.stat.exists

- name: Copy Filebeat configuration (provided).
  template:
    src: "{{ playbook_dir }}/filebeat/filebeat.yml.j2"
    dest: "/etc/filebeat/filebeat.yml"
    owner: root
    group: root
    mode: 0644
  notify: restart filebeat
  when: file.stat.exists

I know this means an extra file, but this will make things easier. Less maintenance trying to make everything configurable and less people forking to make their own filebeat.yml.j2.

This can even be optional. If you don't want to pass your own file, then work with the default .j2 and its externalized variables.

Add support for Ubuntu 18.04 Bionic, and test Debian 9

As the title says.

Make filebeat.yml configuration configurable

See:

• https://github.com/elastic/filebeat/blob/master/etc/filebeat.yml
• https://www.elastic.co/guide/en/beats/filebeat/1.3/filebeat-configuration.html
• https://www.elastic.co/guide/en/beats/filebeat/1.3/config-filebeat-logstash.html

I'll have to use Ansible/Jinja to write some YAML to the file.

How to enable filebeat modules?

Is it possible to enable filebeat modules via this role?

How do we use the keystore?

How can we stick this into the conf and copy the keystore file over?

cloud.id: "${I'd}"
cloud.auth: "${auth}"

Specify ES index scope

I propose to add ability to define filebeat_output_index for ES scope.
Generated /etc/filebeat/filebeat.yml
Now for filebeat_output_elasticsearch_enabled:

# Optional index name. The default is "filebeat" and generates
# [filebeat-]YYYY.MM.DD keys.
#index: "filebeat"

Now for filebeat_output_logstash_enabled

# Optional index name. The default index name depends on the each beat.
# For Packetbeat, the default is set to packetbeat, for Topbeat
# top topbeat and for Filebeat to filebeat.
#index: filebeat

Could be something like

{% if filebeat_output_index %}
index: "{{ filebeat_output_index }}"
{% else %}
# Optional index name. The default is "filebeat" and generates
# [filebeat-]YYYY.MM.DD keys.
#index: "filebeat"
{% endif %}
.......
{% if filebeat_output_index %}
index: "{{ filebeat_output_index }}"
{% else %}
# Optional index name. The default index name depends on the each beat.
# For Packetbeat, the default is set to packetbeat, for Topbeat
# top topbeat and for Filebeat to filebeat.
#index: filebeat
{% endif %}

Sorry don't have access to make a PR :)

Invalid jinja2 template syntax

#76 appears to have issues, when providing a username and password the username gets put at the end on the commented line.

    # Optional auth via API Key or username/password.
    # The options are mutually exclusive and api_key takes the precedence.username: "<my username>"
    password: "<my password>"
    # Number of workers per Elasticsearch host.
    #worker: 1

Configuration:

    filebeat_output_elasticsearch_auth:
        username: "<my username>"
        password: "<my password>"

apt-key for elastic repository is fetched even if it's already imported

We encounter an issue fetching the key for the elastic repository for Debian. Our servers need a proxy setting, but the apt_key module does not support providing proxy configuration. apt-key itself has no configuration file to configure a proxy, either.
Until this is fixed, we would like to import the key manually (or by a script that gets called by ansible).
Further we should provide the elastic apt-key ID (46095ACC8548582C1A2699A9D27D666CD88E42B4) in setup-Debian.yml, so it does not get fetched again when its already there. This also will reduce web calls for all other installations.

Enable logging dependent on Logstash output

Inside the filebeat.yml.j2 it appears the logging config is nested inside the boolean logic for whether or not logstash output is enabled. Can the first {% endif %} on line 148 be moved to line 107?

{% if filebeat_output_logstash_enabled %}
  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: {{ filebeat_output_logstash_hosts | to_json }}

    # Number of workers per Logstash host.
    #worker: 1

    # Optional load balance the events between the Logstash hosts
    #loadbalance: true

    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: filebeat

{% if filebeat_ssl_certificate_file and filebeat_ssl_key_file %}
    # ssl configuration. By default is off.
    ssl:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["{{ filebeat_ssl_dir }}/{{ filebeat_ssl_certificate_file | basename }}"]

      # Certificate for TLS client authentication
      certificate: "{{ filebeat_ssl_dir }}/{{ filebeat_ssl_certificate_file | basename }}"

      # Client Certificate Key
      key: "{{ filebeat_ssl_dir }}/{{ filebeat_ssl_key_file | basename}}"

      # Controls whether the client verifies server certificates and host name.
      # If insecure is set to true, all server host names and certificates will be
      # accepted. In this mode TLS based connections are susceptible to
      # man-in-the-middle attacks. Use only for testing.
      insecure: {{ filebeat_ssl_insecure }}

      # Configure cipher suites to be used for TLS connections
      #cipher_suites: []

      # Configure curve types for ECDHE based cipher suites
      #curve_types: []
{% endif %}

{% if filebeat_enable_logging %}
logging:
  ### Filebeat log
  level: {{ filebeat_log_level }}

  # Enable file rotation with default configuration
  to_files: true

  # Do not log to syslog
  to_syslog: false

  files:
    path: {{ filebeat_log_dir }}
    name: {{ filebeat_log_filename }}
    keepfiles: 7
{% endif %}
{% endif %}

If variable is defined

You use:
{% if filebeat_output_elasticsearch_enabled %}

In current version of ansible this not works fine:
FAILED! => {
"changed": false,
"failed": true,
"msg": "AnsibleUndefinedVariable: 'filebeat_output_elasticsearch_enabled' is undefined"
}

Should use:
{% if filebeat_output_elasticsearch_enabled is defined %}

This works for me.
2.4.0.0-1ppa~trusty

RH9 fails to import elastic gpg key due to crypto policy.

Ran across an issue in Alma 9 (assuming the other variants of 9 as well) where the crypto policy does not allow the import of the elastic key due to its SHA1 signature.

TASK [geerlingguy.filebeat : Add Elasticsearch GPG key.] ********************************************************************************************************************************************
fatal: [xxx.xxx.xxx.xxx]: FAILED! => {"changed": false, "msg": "error: /tmp/tmptfcvb_bz: key 1 import failed.\n"}

Workaround
I'm not quite skilled enough with ansible to suggest a fix, but I've found a workaround that should work for RH9 variants.
Temporarily reduces security but enables functionality.

https://almalinux.discourse.group/t/unable-to-import-a-gpg-key-on-almalinux-9/1235

pre_tasks:
    - name: Set legacy crypto policy for filebeat install
      ansible.builtin.command: update-crypto-policies --set DEFAULT:SHA1

roles:
    - geerlingguy.filebeat

post_tasks:
    - name: Reset crypto policy to default.
      ansible.builtin.command: update-crypto-policies --set DEFAULT