elastic / ansible-beats Goto Github PK

View Code? Open in Web Editor NEW

131.0 131.0 89.0 257 KB

Ansible Beats Role

License: Other

Ruby 77.43% Makefile 6.69% Jinja 15.87%

ansible-beats's People

Contributors

Stargazers

Watchers

Forkers

talend levonet star2one awlodarkiewicz meecr0b sdarwin bikoumba freemanpd evilezh shunneng humio jmlrt samaws1 gh94uk cvali blockdaemon lanzafame fabiofarina spitfire55 xysforever xabim ktibi nkiraly aakashofficial lean-delivery hardestworkingmaninshowbusiness rheak includerandom totosh beand naoh87 sg-yourserveradmin-com pluggi oulibraries vladgb isweluiz nkammah hakamine entiis opuscapita drewbarrett raykishev v1v isabella232 lifeplus-io trongtan124 metabsd jmhossain vielfarbig leandromoreirati strophy stoffus k2death devops-collection ekmixon elk-ansible liza-mae slconley pipastudios-archive sysbot1843 julienorain jpiron ajdelgado seadog007 amanoncloud nicolasberens budman17r tejayageti tribunadigital igor-p-imprivata jledoze sysolyakin bitzlato dtourde-dd terkea suryaakana117 tavex

ansible-beats's Issues

`beat_conf.setup.ilm.policy_file` must be specified

Beats product: metricbeat/filebeat

Beats version: 7.11.1

Role version: 7.11.1, 7.11.2, 7.12.0

OS version: Linux xxxx 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behaviour:

Following the doc, I wanted to add a custom ILM policy to my playbook. Up until now I was using the beat_conf.setup.ilm.policy_name field and it was working fine.

When adding default_ilm_policy and beat_conf.setup.ilm.policy_file, Ansible returns this error:

fatal: [es-1]: FAILED! => {"changed": false, "msg": "beat_conf.setup.ilm.policy_file must be specified if default_ilm_policy is used"}

The issue is that it is indeed in the playbook.

Expected behavior is to have a new default ilm policy created used by the beat.

Playbook:

---
- name: setup
  hosts: es-1
  tasks:
  - name: setup metricbeat kibana dashboards and templates
    include_role:
      name: elastic.beats
    vars:
      beats_version: 7.11.1
      beat: metricbeat
      version_lock: true
      default_ilm_policy: "../files/conf/new_policy.json"
      become: yes
      output_conf:
        elasticsearch:
          hosts: ["https://es-1:9200"]
          username: "redacted"
          password: "redacted"
          ssl.certificate: "/etc/ssl/localcerts/redacted.crt"
          ssl.key: "/etc/ssl/localcerts/redacted.key"
          ssl.certificate_authorities: ["/usr/local/share/ca-certificates/redacted.crt"]
          ssl.verification_mode: certificate
          index: "metricbeat"
      beat_conf:
        monitoring.enabled: true
        setup.kibana:
          host: "https://kbn-1:5601"
          username: "redacted"
          password: "redacted"
          ssl.certificate: "/etc/ssl/localcerts/redacted.crt"
          ssl.key: "/etc/ssl/localcerts/redacted.key"
          ssl.certificate_authorities: ["/usr/local/share/ca-certificates/redacted.crt"]
          ssl.verification_mode: certificate
        setup.dashboards.enabled: true
        setup.dashboards.index: "metricbeat-*"
        setup.ilm:
          enabled: true
          rollover_alias: "metricbeat"
          pattern: "000001"
          policy_file: "/etc/metricbeat/policies/new_policy.json
          overwrite: true
        setup.template:
          name: "metricbeat_template"
          overwrite: true
          pattern: "metricbeat-*"
          settings:
            index.number_of_shards: 1
            index.number_of_replicas: 0
        metricbeat.modules:
        - module: system
          metricsets:
            - cpu             # CPU usage
            - load            # CPU load averages
            - memory          # Memory usage
            - network         # Network IO
            - process         # Per process metrics
            - process_summary # Process summary
            - uptime          # System Uptime
            - socket_summary  # Socket summary
            #- core           # Per CPU core usage
            - diskio         # Disk IO
            #- filesystem     # File system usage for each mountpoint
            - fsstat         # File system summary metrics
            #- raid           # Raid
            - socket         # Sockets and connection info (linux only)
            - service        # systemd service information
          enabled: true
          period: 10s
          processes: ['.*']
        # Configure the metric types that are included by these metricsets.
          cpu.metrics:  ["percentages","normalized_percentages"]  # The other available option is ticks.
          core.metrics: ["percentages"]  # The other available option is ticks.
        - module: linux
          period: 10s
          metricsets:
            - "pageinfo"
            - "memory"
            # - ksm
            # - conntrack
            # - iostat
          enabled: true
          # hostfs: /hostfs

Provide logs from Ansible:

fatal: [es-1]: FAILED! => {"changed": false, "msg": "beat_conf.setup.ilm.policy_file must be specified if default_ilm_policy is used"}

I also spent some time using the Ansible debugger. I managed to see that indeed the variable was existing when I displayed p task_vars['beatf_conf'] but when I added nested path setup.ilm I had the following error: ***KeyError:KeyError('beat_conf.setup.ilm') so I don't really understand if it's a limitation of the debugger or if the issue is around those nested json paths or something else.

Thanks in advance for your help!

When does it support windows server?

since the beats part will run on windows server so do we have plan to support it?

Filebeat can't start, with default /etc/default/filebeat file on OS without systemd

Beats product: Filebeat

Beats version 7.8.0

Role version: 12ef312

OS version Linux xxx 3.16.7-ckt9-drt #4 SMP Sun Apr 19 20:59:40 CEST 2015 x86_64 GNU/Linux
Debian GNU/Linux 8.0 (jessie)

Description of the problem including expected versus actual behaviour:
When I install filebeat on this OS (this version OS don't have systemd), filebeat can't start, bacause have wrong paths:

# /etc/init.d/filebeat start
[....] Starting Filebeat sends log files to Logstash or directly to Elasticsearch.: filebeat2020-06-26T14:21:30.225+0200 INFO instance/beat.go:647 Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]

When I remove /etc/default/filebeat, dilebeat starting properly.
In this file is variable DAEMON_ARGS="-c /etc/filebeat/filebeat.yml ".
In file /etc/init.d/filebeat is also the same variable in property values:
DAEMON_ARGS="-c /etc/${NAME}/${NAME}.yml -path.home /usr/share/${NAME} -path.config /etc/${NAME} -path.data /var/lib/${NAME} -path.logs /var/log/${NAME}", but after it file /etc/default/filebeat is loading, so variable DAEMON_ARGS is from this file.

I think, will be good, if can be possible disable copy file to /etc/default/, or cat line with DAEMON_ARGS in templates/beat.j2.
There is no problem on Debian 9 or 10, because have systemd.

Playbook:

name: Filebeat with custom configuration
hosts: all
roles:

role: elastic.beats
vars:
version_lock: true
beat: filebeat
beat_conf:
filebeat.modules:

module: system
output_conf: {"logstash":{"hosts":["logstash1.domain:5044"]}}
logging_conf: {"to_files": true, "metrics.enabled": false}

Provide logs from Ansible:


PLAY [Filebeat with custom configuration] ***************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : check-parameters] *****************************************************************************************************************************************************************************************************
included: /home/kmoscick/.ansible/roles/elastic.beats/tasks/beats-param-check.yml for drt-7.domain

TASK [elastic.beats : Check beat variable] **************************************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Check beat_conf variable] *********************************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Check ILM variables] **************************************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Set beats_major_version] **********************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : define beat product] **************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : os-specific vars] *****************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : include_tasks] ********************************************************************************************************************************************************************************************************
included: /home/kmoscick/.ansible/roles/elastic.beats/tasks/beats.yml for drt-7.domain

TASK [elastic.beats : Include specific Beats] ***********************************************************************************************************************************************************************************************
included: /home/kmoscick/.ansible/roles/elastic.beats/tasks/beats-debian.yml for drt-7.domain

TASK [elastic.beats : Debian - Ensure apt-transport-https is installed] *********************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Debian - Ensure python-urllib3, python-openssl, python-pyasn1 & python-pip are installed] *****************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Debian - ensure ndg-httpsclient pip is installed] *********************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Debian - Add Beats repository key] ************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Debian - add beats repository] ****************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Debian - unhold filebeat version for install] *************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Debian - Ensure filebeat is installed] ********************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Debian - hold filebeat version] ***************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Set os_arch] **********************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Debian - Download filebeat from url] **********************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Debian - Ensure filebeat is installed from downloaded package] ********************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Include specific Beats] ***********************************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Beats configuration] **************************************************************************************************************************************************************************************************
included: /home/kmoscick/.ansible/roles/elastic.beats/tasks/beats-config.yml for drt-7.domain

TASK [elastic.beats : Set default facts] ****************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Set beat_shipper_conf] ************************************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Set beat_logging_conf] ************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Check pid_dir status] *************************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Create PID Directory] *************************************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Create Config Directory] **********************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Copy Default File for Instance] ***************************************************************************************************************************************************************************************
ok: [drt-7.domain]

TASK [elastic.beats : Copy Configuration File for filebeat] *********************************************************************************************************************************************************************************
changed: [drt-7.domain]

TASK [elastic.beats : Create default policies config directory] *****************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Copy default ILM policy file for filebeat] ****************************************************************************************************************************************************************************
skipping: [drt-7.domain]

TASK [elastic.beats : Start filebeat service] ***********************************************************************************************************************************************************************************************
fatal: [drt-7.domain]: FAILED! => {"changed": false, "msg": "Failed to start service: filebeat", "rc": 1, "stderr": "2020-06-26T16:05:35.653+0200\tINFO\tinstance/beat.go:647\tHome path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]\n2020-06-26T16:05:35.654+0200\tINFO\tinstance/beat.go:655\tBeat ID: 75deba96-a4c9-47a6-a04d-1b03b2f078a7\n2020-06-26T16:05:35.654+0200\tINFO\t[beat]\tinstance/beat.go:983\tBeat info\t{\"system_info\": {\"beat\": {\"path\": {\"config\": \"/usr/share/filebeat/bin\", \"data\": \"/usr/share/filebeat/bin/data\", \"home\": \"/usr/share/filebeat/bin\", \"logs\": \"/usr/share/filebeat/bin/logs\"}, \"type\": \"filebeat\", \"uuid\": \"75deba96-a4c9-47a6-a04d-1b03b2f078a7\"}}}\n2020-06-26T16:05:35.654+0200\tINFO\t[beat]\tinstance/beat.go:992\tBuild info\t{\"system_info\": {\"build\": {\"commit\": \"f79387d32717d79f689d94fda1ec80b2cf285d30\", \"libbeat\": \"7.8.0\", \"time\": \"2020-06-14T18:15:37.000Z\", \"version\": \"7.8.0\"}}}\n2020-06-26T16:05:35.654+0200\tINFO\t[beat]\tinstance/beat.go:995\tGo runtime info\t{\"system_info\": {\"go\": {\"os\":\"linux\",\"arch\":\"amd64\",\"max_procs\":32,\"version\":\"go1.13.10\"}}}\n2020-06-26T16:05:35.713+0200\tINFO\t[beat]\tinstance/beat.go:999\tHost info\t{\"system_info\": {\"host\": {\"architecture\":\"x86_64\",\"boot_time\":\"2020-06-25T12:15:27+02:00\",\"containerized\":false,\"name\":\"drt-7\",\"ip\":[\"127.0.0.1/8\",\"x.x.x.x/25\"],\"kernel_version\":\"3.16.7-ckt9-drt\",\"mac\":[\"xx:xx:xx:xx:xx:xx\",\"xx:xx:xx:xx:xx:xx\"],\"os\":{\"family\":\"debian\",\"platform\":\"debian\",\"name\":\"Debian GNU/Linux\",\"version\":\"8 (jessie)\",\"major\":8,\"minor\":0,\"patch\":0,\"codename\":\"jessie\"},\"timezone\":\"CEST\",\"timezone_offset_sec\":7200,\"id\":\"7986a9307f6943ab9602c1248a18cfc2\"}}}\n2020-06-26T16:05:35.713+0200\tINFO\t[beat]\tinstance/beat.go:1028\tProcess info\t{\"system_info\": {\"process\": {\"capabilities\": {\"inheritable\":null,\"permitted\":[\"chown\",\"dac_override\",\"dac_read_search\",\"fowner\",\"fsetid\",\"kill\",\"setgid\",\"setuid\",\"setpcap\",\"linux_immutable\",\"net_bind_service\",\"net_broadcast\",\"net_admin\",\"net_raw\",\"ipc_lock\",\"ipc_owner\",\"sys_module\",\"sys_rawio\",\"sys_chroot\",\"sys_ptrace\",\"sys_pacct\",\"sys_admin\",\"sys_boot\",\"sys_nice\",\"sys_resource\",\"sys_time\",\"sys_tty_config\",\"mknod\",\"lease\",\"audit_write\",\"audit_control\",\"setfcap\",\"mac_override\",\"mac_admin\",\"syslog\",\"wake_alarm\",\"block_suspend\",\"audit_read\"],\"effective\":[\"chown\",\"dac_override\",\"dac_read_search\",\"fowner\",\"fsetid\",\"kill\",\"setgid\",\"setuid\",\"setpcap\",\"linux_immutable\",\"net_bind_service\",\"net_broadcast\",\"net_admin\",\"net_raw\",\"ipc_lock\",\"ipc_owner\",\"sys_module\",\"sys_rawio\",\"sys_chroot\",\"sys_ptrace\",\"sys_pacct\",\"sys_admin\",\"sys_boot\",\"sys_nice\",\"sys_resource\",\"sys_time\",\"sys_tty_config\",\"mknod\",\"lease\",\"audit_write\",\"audit_control\",\"setfcap\",\"mac_override\",\"mac_admin\",\"syslog\",\"wake_alarm\",\"block_suspend\",\"audit_read\"],\"bounding\":[\"chown\",\"dac_override\",\"dac_read_search\",\"fowner\",\"fsetid\",\"kill\",\"setgid\",\"setuid\",\"setpcap\",\"linux_immutable\",\"net_bind_service\",\"net_broadcast\",\"net_admin\",\"net_raw\",\"ipc_lock\",\"ipc_owner\",\"sys_module\",\"sys_rawio\",\"sys_chroot\",\"sys_ptrace\",\"sys_pacct\",\"sys_admin\",\"sys_boot\",\"sys_nice\",\"sys_resource\",\"sys_time\",\"sys_tty_config\",\"mknod\",\"lease\",\"audit_write\",\"audit_control\",\"setfcap\",\"mac_override\",\"mac_admin\",\"syslog\",\"wake_alarm\",\"block_suspend\",\"audit_read\"],\"ambient\":null}, \"cwd\": \"/var/devops\", \"exe\": \"/usr/share/filebeat/bin/filebeat\", \"name\": \"filebeat\", \"pid\": 22323, \"ppid\": 22322, \"seccomp\": {\"mode\":\"disabled\"}, \"start_time\": \"2020-06-26T16:05:35.430+0200\"}}}\n2020-06-26T16:05:35.714+0200\tINFO\tinstance/beat.go:310\tSetup Beat: filebeat; Version: 7.8.0\n2020-06-26T16:05:35.715+0200\tINFO\t[publisher]\tpipeline/module.go:113\tBeat name: drt-7\n2020-06-26T16:05:35.715+0200\tERROR\tfileset/modules.go:127\tNot loading modules. Module directory not found: /usr/share/filebeat/bin/module\n2020-06-26T16:05:35.715+0200\tERROR\tinstance/beat.go:958\tExiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?\nExiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?\n", "stderr_lines": ["2020-06-26T16:05:35.653+0200\tINFO\tinstance/beat.go:647\tHome path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]", "2020-06-26T16:05:35.654+0200\tINFO\tinstance/beat.go:655\tBeat ID: 75deba96-a4c9-47a6-a04d-1b03b2f078a7", "2020-06-26T16:05:35.654+0200\tINFO\t[beat]\tinstance/beat.go:983\tBeat info\t{\"system_info\": {\"beat\": {\"path\": {\"config\": \"/usr/share/filebeat/bin\", \"data\": \"/usr/share/filebeat/bin/data\", \"home\": \"/usr/share/filebeat/bin\", \"logs\": \"/usr/share/filebeat/bin/logs\"}, \"type\": \"filebeat\", \"uuid\": \"75deba96-a4c9-47a6-a04d-1b03b2f078a7\"}}}", "2020-06-26T16:05:35.654+0200\tINFO\t[beat]\tinstance/beat.go:992\tBuild info\t{\"system_info\": {\"build\": {\"commit\": \"f79387d32717d79f689d94fda1ec80b2cf285d30\", \"libbeat\": \"7.8.0\", \"time\": \"2020-06-14T18:15:37.000Z\", \"version\": \"7.8.0\"}}}", "2020-06-26T16:05:35.654+0200\tINFO\t[beat]\tinstance/beat.go:995\tGo runtime info\t{\"system_info\": {\"go\": {\"os\":\"linux\",\"arch\":\"amd64\",\"max_procs\":32,\"version\":\"go1.13.10\"}}}", "2020-06-26T16:05:35.713+0200\tINFO\t[beat]\tinstance/beat.go:999\tHost info\t{\"system_info\": {\"host\": {\"architecture\":\"x86_64\",\"boot_time\":\"2020-06-25T12:15:27+02:00\",\"containerized\":false,\"name\":\"drt-7\",\"ip\":[\"127.0.0.1/8\",\"x.x.x.x/25\"],\"kernel_version\":\"3.16.7-ckt9-drt\",\"mac\":[\"xx:xx:xx:xx:xx:xx\",\"xx:xx:xx:xx:xx:xx\"],\"os\":{\"family\":\"debian\",\"platform\":\"debian\",\"name\":\"Debian GNU/Linux\",\"version\":\"8 (jessie)\",\"major\":8,\"minor\":0,\"patch\":0,\"codename\":\"jessie\"},\"timezone\":\"CEST\",\"timezone_offset_sec\":7200,\"id\":\"7986a9307f6943ab9602c1248a18cfc2\"}}}", "2020-06-26T16:05:35.713+0200\tINFO\t[beat]\tinstance/beat.go:1028\tProcess info\t{\"system_info\": {\"process\": {\"capabilities\": {\"inheritable\":null,\"permitted\":[\"chown\",\"dac_override\",\"dac_read_search\",\"fowner\",\"fsetid\",\"kill\",\"setgid\",\"setuid\",\"setpcap\",\"linux_immutable\",\"net_bind_service\",\"net_broadcast\",\"net_admin\",\"net_raw\",\"ipc_lock\",\"ipc_owner\",\"sys_module\",\"sys_rawio\",\"sys_chroot\",\"sys_ptrace\",\"sys_pacct\",\"sys_admin\",\"sys_boot\",\"sys_nice\",\"sys_resource\",\"sys_time\",\"sys_tty_config\",\"mknod\",\"lease\",\"audit_write\",\"audit_control\",\"setfcap\",\"mac_override\",\"mac_admin\",\"syslog\",\"wake_alarm\",\"block_suspend\",\"audit_read\"],\"effective\":[\"chown\",\"dac_override\",\"dac_read_search\",\"fowner\",\"fsetid\",\"kill\",\"setgid\",\"setuid\",\"setpcap\",\"linux_immutable\",\"net_bind_service\",\"net_broadcast\",\"net_admin\",\"net_raw\",\"ipc_lock\",\"ipc_owner\",\"sys_module\",\"sys_rawio\",\"sys_chroot\",\"sys_ptrace\",\"sys_pacct\",\"sys_admin\",\"sys_boot\",\"sys_nice\",\"sys_resource\",\"sys_time\",\"sys_tty_config\",\"mknod\",\"lease\",\"audit_write\",\"audit_control\",\"setfcap\",\"mac_override\",\"mac_admin\",\"syslog\",\"wake_alarm\",\"block_suspend\",\"audit_read\"],\"bounding\":[\"chown\",\"dac_override\",\"dac_read_search\",\"fowner\",\"fsetid\",\"kill\",\"setgid\",\"setuid\",\"setpcap\",\"linux_immutable\",\"net_bind_service\",\"net_broadcast\",\"net_admin\",\"net_raw\",\"ipc_lock\",\"ipc_owner\",\"sys_module\",\"sys_rawio\",\"sys_chroot\",\"sys_ptrace\",\"sys_pacct\",\"sys_admin\",\"sys_boot\",\"sys_nice\",\"sys_resource\",\"sys_time\",\"sys_tty_config\",\"mknod\",\"lease\",\"audit_write\",\"audit_control\",\"setfcap\",\"mac_override\",\"mac_admin\",\"syslog\",\"wake_alarm\",\"block_suspend\",\"audit_read\"],\"ambient\":null}, \"cwd\": \"/var/devops\", \"exe\": \"/usr/share/filebeat/bin/filebeat\", \"name\": \"filebeat\", \"pid\": 22323, \"ppid\": 22322, \"seccomp\": {\"mode\":\"disabled\"}, \"start_time\": \"2020-06-26T16:05:35.430+0200\"}}}", "2020-06-26T16:05:35.714+0200\tINFO\tinstance/beat.go:310\tSetup Beat: filebeat; Version: 7.8.0", "2020-06-26T16:05:35.715+0200\tINFO\t[publisher]\tpipeline/module.go:113\tBeat name: drt-7", "2020-06-26T16:05:35.715+0200\tERROR\tfileset/modules.go:127\tNot loading modules. Module directory not found: /usr/share/filebeat/bin/module", "2020-06-26T16:05:35.715+0200\tERROR\tinstance/beat.go:958\tExiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?", "Exiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?"], "stdout": "Starting Filebeat sends log files to Logstash or directly to Elasticsearch.: filebeat failed!\n", "stdout_lines": ["Starting Filebeat sends log files to Logstash or directly to Elasticsearch.: filebeat failed!"]}

RUNNING HANDLER [elastic.beats : restart the service] ***************************************************************************************************************************************************************************************

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
drt-7              : ok=21   changed=1    unreachable=0    failed=1    skipped=12   rescued=0    ignored=0

Beats logs if relevant:

Enable metricbeat docker module

Beats product: MetricBeat

Beats version: 7.8

The docker module is not displayed when I do metricbeat modules list
Only system appears.
Do I have to exec a command in my role like metricbeat modules enable docker ?

beat: metricbeat
beat_conf:
    cloud:
        id: '{{ elastic.cloud.id }}'
        auth: '{{ elastic.cloud.auth }}'
    setup:
        dashboards.enabled: true
        ilm:
            overwrite: true
    metricbeat.modules:
        - module: system
          metricsets:
              - cpu             # CPU usage
              - load            # CPU load averages
              - memory          # Memory usage
              - network         # Network IO
              - process         # Per process metrics
              - process_summary # Process summary
              - uptime          # System Uptime
              - socket_summary  # Socket summary
              - core           # Per CPU core usage
              - diskio         # Disk IO
              - filesystem     # File system usage for each mountpoint
              - fsstat         # File system summary metrics
              - socket         # Sockets and connection info (linux only)
              - service        # systemd service information
          enabled: true
          period: 10s
          processes: [".*"]
          cpu_ticks: false
        - module: docker
          metricsets:
              - container
              - cpu
              - diskio
              - event
              - healthcheck
              - info
              - image
              - memory
              - network
          hosts: ["unix:///var/run/docker.sock"]
          period: 10s
          enabled: true
    metricbeat:
        config:
            modules:
                path: '${path.config}/modules.d/*.yml'

When including the role more than once for different `{{ beats }}`, notify only gets called for the last role in the task list.

#34

artifacts.elastic.co repo issues

With the release of Filebeat 7.15.0, I'm no longer able to install beats using this role. Just yesterday it worked fine.

When running playbook I get:

FAILED - RETRYING: Debian - Ensure filebeat is installed (1 retries left).
fatal: [*.*.*.*]: FAILED! => {"attempts": 2, "cache_update_time": 1632335658, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"     install 'filebeat=7.14.0'' failed: E: Version '7.14.0' for 'filebeat' was not found\n", "rc": 100, "stderr": "E: Version '7.14.0' for 'filebeat' was not found\n", "stderr_lines": ["E: Version '7.14.0' for 'filebeat' was not found"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information..."]}

When trying to apt update manually:

Reading package lists... Done
E: Failed to fetch https://artifacts.elastic.co/packages/7.x/apt/dists/stable/main/binary-amd64/Packages.bz2  File has unexpected size (78886 != 78795). Mirror sync in progress? [IP: 34.120.127.130 443]
   Hashes of expected file:
    - Filesize:78795 [weak]
    - SHA512:6a31510a68ed4ec43116053445bcae3e01e8aee5a0a91115d33106751aa07803260f9e7c8de356cc6e80ca3225f2f60280de694cb59e0bbb9654de3149a2147b
    - SHA256:33e8f2fba7c5914d6bae8741560b4f12874e7bd655dfc7eaa73c93ed80837707
    - SHA1:41ff70e24888210b61ddf373af05076563e69e82 [weak]
    - MD5Sum:d6ea4093e04c8575814315b36e7174f6 [weak]
   Release file created at: Wed, 15 Sep 2021 21:37:54 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Support Beats OSS version

This role currently only support Beats with Elastic License.
We should allow users to deploy Beats OSS versions.

Related to #28

Enable modules on filebeat

Documentation how to enable / disable modules for filebeat, load the dashboards, set variable for module etc

Debian repository configuration is lacking TLS usage

The tasks/beats-debian.yml file uses plain HTTP URLs (e.g. http://packages.elasticsearch.org/GPG-KEY-elasticsearch).

This is not entirely safe. Usage of HTTPS URLs is strongly recommended.

Thanks!

Support for Auditbeat and Winlogbeat

Please add support for Auditbeat and Winlogbeat.

output_conf is appended to default, not overwritten

Role version: https://github.com/elastic/ansible-beats/releases/tag/7.0.0

I'm not 100 sure since I'm no ansible expert, by my playbook uses these variables:

beat: filebeat
beats_version: 7.3.2
beat_conf:
  inputs:
    - type: log
      enabled: true
      paths:
        - /foo/bar/*.log
      include_lines: ['sometext']
  path.logs: /var/log/filebeat
output_conf:
  logstash:
    hosts:
      - "somehost:9004"

And the resulting filebeat.yml that gets deployed reads:

################### filebeat Configuration #########################

############################# filebeat ######################################
inputs:
-   enabled: true
    include_lines:
    - 'sometext'
    paths:
    - /foo/bar/*.log
    type: log
path.logs: /var/log/filebeat


###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

output:
    elasticsearch:
        hosts:
        - localhost:9200
    logstash:
        hosts:
        - somehost:9004


############################# Logging #########################################

logging:
    files:
        rotateeverybytes: 10485760

This makes filebeat unable to start, since 2 different output options are not supported. I seems like my customer logstash output is added to the default elasticsearch output on localhost, while it should be overwritten. If I understand it correctly, "elasticsearch" and "logstash" are different keys in the "output" dictionary, and hence they are co-existing. Probably easy to fix by removing the default, but I'm not sure what the proper fix would be.

Or am I missing something?

How to define ILM policy from Ansible ?

When adding a beat, after process start, it will load default template into kibana and configure default template and default policy for those templates (metricbeat-* for example).
As ILM is enable by default since 7.0.0, it is quite a surprise to see that index are not rollover every day as it was before.
Policy is by default size 50g and retention 30days. It would be great to be able to define those within ansible.
For now, after policy and template are created, we need to change policy through kibana (or elastic tools) (version 1 becomes version 2), remove the policy from existing index (version 1) and add back policy (version 2).

Beats keystore support

This ticket is to add keystore support as requested in #65 (comment) and #58 (comment)

Service doesn't restarted if service already ran

Ansible doesn't restart service with state=started if service already ran.

Rule in handlers name: restart beat was False because beat_install.changed = false.

  when: start_service and restart_on_change and not beats_started.changed and beat_install.changed

log:

TASK [beats : RedHat - Ensure filebeat is installed] *******************************************************************************************************************************************************
ok: [globus2.dev.*****] => {"changed": false, "msg": "", "rc": 0, "results": ["filebeat-6.2.4-1.x86_64 providing filebeat-6.2.4 is already installed"]}

...

TASK [beats : Copy Configuration File] *********************************************************************************************************************************************************************
changed: [globus2.dev.*****] => {"changed": true, "checksum": "2c2692b5147416348ef64099617cec9ff32fe69e", "dest": "/etc/filebeat/filebeat.yml", "gid": 0, "group": "root", "md5sum": "9e3c78e1796ae48dee244939eb3b64f3", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 999, "src": "/home/pasha/.ansible/tmp/ansible-tmp-1527153468.01-180088665979184/source", "state": "file", "uid": 0}

TASK [beats : Start beats service] *************************************************************************************************************************************************************************
ok: [globus2.dev.*****] => {"changed": false, "enabled": true, "name": "filebeat", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2018-05-24 04:03:24 UTC", "ActiveEnterTimestampMonotonic": "20510694", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network-online.target systemd-journald.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Thu 2018-05-24 04:03:24 UTC", "AssertTimestampMonotonic": "20510080", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2018-05-24 04:03:24 UTC", "ConditionTimestampMonotonic": "20510080", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/filebeat.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "filebeat", "DevicePolicy": "auto", "Documentation": "https://www.elastic.co/guide/en/beats/filebeat/current/index.html", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "856", "ExecMainStartTimestamp": "Thu 2018-05-24 04:03:24 UTC", "ExecMainStartTimestampMonotonic": "20510647", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/share/filebeat/bin/filebeat ; argv[]=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/filebeat.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "filebeat.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2018-05-24 04:03:24 UTC", "InactiveExitTimestampMonotonic": "20510694", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "39291", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "39291", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "856", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "filebeat.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "always", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "network-online.target system.slice", "WatchdogTimestamp": "Thu 2018-05-24 04:03:24 UTC", "WatchdogTimestampMonotonic": "20510676", "WatchdogUSec": "0"}}

RUNNING HANDLER [beats : restart beat] *********************************************************************************************************************************************************************
skipping: [globus2.dev.*****] => {"changed": false, "skip_reason": "Conditional result was False"}

"use_repository: false" - Default URL Broken

Beats product:
metricbeat (probably others too?)

Beats version
7.11.2

Role version: (If using master please specify github sha)
v7.11.2

OS version (uname -a if on a Unix-like system):
4.19.0-13-amd64 Debian (Host)
4.15.0-99-generic Ubuntu (Target)

Description of the problem including expected versus actual behaviour:
During the Download metricbeat from URL step, it fails. This happens, when using "use_repository: false" to download the packages without apt.

Playbook:
Please specify the full playbook used to reproduce this issue.


---
- hosts: all
  pre_tasks: 

    - name: includepasswords
      include_vars: passwords.yml

    - set_fact: ansible_sudo_pass={{vars[inventory_hostname + '_sudopass']}}
  roles:
    - role: elastic.beats
      become: yes
    
  vars:
    beats_version: 7.11.2
    beat: metricbeat
    beat_conf:
      metricbeat.modules:
      - module: system
        metricsets:
          - cpu             # CPU usage
          - load            # CPU load averages
          - memory          # Memory usage
          - network         # Network IO
          - process         # Per process metrics
          - process_summary # Process summary
          - uptime          # System Uptime
          #- socket_summary  # Socket summary
          #- core           # Per CPU core usage
          - diskio         # Disk IO
          - filesystem     # File system usage for each mountpoint
          - fsstat         # File system summary metrics
          #- raid           # Raid
          #- socket         # Sockets and connection info (linux only)
        enabled: true
        period: 30m
        processes: ['.*']
        # Configure the metric types that are included by these metricsets.

Provide logs from Ansible:

Typo in example

name: Example playbook for installing packetbear.

Just a typo

Windows support

Hello,
Is there any plan to introduce Windows support the the playbooks?

Add support for Auditbeat, Heartbeat and Functionbeat

This Ansible role currently support Filebeat, MetricBeat and Packetbeat.

We should also support the other official Beats: Auditbeat, Heartbeat and FunctionBeat.

Note that supporting Winlogbeat would require Windows support which we currently don't have (see #17 and #27).

Related to #36

Tasks

Add 3 new integration tests which will deploy and validate each of the new Beats to support

apt-mark unhold fails

Beats product: any

Beats version: any

Role version: v7.10.2

OS version : Linux srv01 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03) x86_64 GNU/Linux

Description of the problem including expected versus actual behaviour:

when installing from custom_package_url, unhold command fail with rc=100 if beat is not installed

Playbook:

beats_use_repository: false
beats_add_repository: false
metricbeat_package_url: 'file:///tmp/metricbeat-7.10.2-amd64.deb'
beat_output_conf:
  elasticsearch:
    enabled: true
    hosts:
      - els01
      - els02
    loadbalance: true
beat_conf_metricbeat:
  metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - load
      - memory
      - network
      - diskio
      - service
    period: 10s

- hosts: all
  roles:
    - role: elastic_beats
      vars:
        beat: metricbeat
        beat_conf: "{{ beat_conf_metricbeat }}"
        output_conf: "{{ beat_output_conf }}"
        use_repository: "{{ beats_use_repository }}"
        custom_package_url: "{{ metricbeat_package_url | default('') }}"

Provide logs from Ansible:

TASK [elastic_beats : Debian - unhold metricbeat version for install] ***************************************************************************************************************************************
fatal: [srv01]: FAILED! => changed=false 
  cmd:
  - apt-mark
  - unhold
  - metricbeat
  delta: '0:00:00.123186'
  end: '2021-05-19 15:00:20.453167'
  msg: non-zero return code
  rc: 100
  start: '2021-05-19 15:00:20.329981'
  stderr: |-
    E: Unable to locate package metricbeat
    E: No packages found
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

Beats logs if relevant:

Avoid locking beats_version and provide documentation.

The role defaults to beats_version 6.2.1. This was rather surprising and undocumented behavior, and would likely be confusing for others. For example, docs for the latest version of filebeat references "filebeat.inputs", which didn't exist in 6.2.1. Filebeat fails quietly when a key is present in config it doesn't expect, causing me to spend a few hours to find the root cause.

Instead, the version should probably be a required variable, just like beats_conf. Users can then choose which version to pin.

Add journalbeat tests/support

Journalbeat is listed as experimental
https://www.elastic.co/guide/en/beats/journalbeat/master/index.html

but has been merged into the main beats repository.

The potential for integration directly into journald (checks may have to be required in order to verify journald is present) is great, since you can then directly log from systemd units.

conditionals in output config

To have multiple output indices, like in https://discuss.elastic.co/t/filebeat-apache-module-change-index-name/176955/2

We use this syntax:

output.elasticsearch:
  hosts: ["elastic1:9200", "elastic2:9200"]
  indices:
    - index: "filebeat-%{[agent.version]}-apache-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "apache"
    - default: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

I have been struggling hard to make ansible to_nice_yaml produce such output. I wonder if it is even possible. What is hard is you need to use an array to obtain the - index: XXXX part, but also you need the when to be indented at the same level...

How do you guys make this work ?

Beats SSL Support

Can SSL Support be added to this role similar to what was recently added to the ansible-elasticsearch role?

elastic.beats was NOT installed successfully: Unable to compare role versions

ansible-galaxy install elastic.beats --force                
- changing role elastic.beats from 7.9.2 to unspecified
- downloading role 'beats', owned by elastic
[WARNING]: - elastic.beats was NOT installed successfully: Unable to compare role versions (7.0.0, 7.4.0, 7.4.1, 7.5.0, 7.5.1, 7.5.2, 7.6.0, 7.6.1, 7.6.2, 7.7.0, 7.7.1, 7.8.0, 7.8.1, 7.9.0, 7.9.1,
7.9.2, 7.9.3, 7.10.0, v7.10.1, v7.10.2, v7.11.1, v7.11.2, v7.12.0) to determine the most recent version due to incompatible version formats. Please contact the role author to resolve versioning
conflicts, or specify an explicit role version to install.

ansible-galaxy and proper versioning

Hi,

(aesthetic improvment, not a bug)

Any chance to have this project available in ansible-galaxy ? with a proper stable version ?
It is already done on a sister ansible role (https://github.com/elastic/ansible-elasticsearch) so i guess it should be doable on this one as well.
Yeah, i know i can use it in requirement anyway, but i'd like it ot be consistent =) :

- src: elastic.elasticsearch
  version: 6.4.0

- src: https://github.com/elastic/ansible-beats
  version: 1abc778

(maybe next step would be kibana =P)

Error during "unhold filebeat version for install"

Beats product: Filebeat

Beats version: 7.6.1

OS version (uname -a if on a Unix-like system):

Ubuntu 18.04.
Linux xxx 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behaviour:

I would like to install filebeat via this role and provide a custom package url.

beat: filebeat
use_repository: false
custom_package_url: "https://xxx/filebeat/filebeat-7.6.1-amd64.deb"
beat_conf:
  filebeat:
    xxx

But this fails with the following error:

TASK [elastic.beats : Debian - unhold filebeat version for install] **********************************************************************************************************************************************************************************************************
fatal: [xxx]: FAILED! => changed=false 
  cmd:
  - apt-mark
  - unhold
  - filebeat
  delta: '0:00:00.099800'
  end: '2020-04-08 16:10:52.052093'
  msg: non-zero return code
  rc: 100
  start: '2020-04-08 16:10:51.952293'
  stderr: |-
    E: Paket filebeat kann nicht gefunden werden.
    E: Keine Pakete gefunden
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

The issue seems to be that the following task is always executed:

- name: Debian - unhold {{ beat }} version for install
  command: apt-mark unhold {{ beat }}
  changed_when: false

Without this task, everything runs fine.

And Elastic username and password for all beats

Hello,

I would like to add Elasticsearch username and password like this in my playbook

output_conf:
elasticsearch:
hosts: http://localhost:9200
password: ******
username: *****

Regards.

Pin repository OpenPGP fingerprint

As mentioned in #2 (comment), additionally to use https to download the key, it should be pinned. The pinning has additional benefits because what should the OpenPGP key actually protect against? -> Compromised webservers in case we already have TLS.

Currently the key comes from: https://packages.elastic.co/GPG-KEY-elasticsearch
And the repo: https://artifacts.elastic.co/packages/

They are both hosted by fastly.net. So currently, adding the OpenPGP key or just accepting not trusted sources does not make much difference from a security point of view.

@jmlrt You just closed #2 without tracking that the actual issue still persists.

repo_key can overlap with other ansible roles, should be named to something unique

#10 introduced a regression in my environment because I have other roles that use "repo_key"

It should probably renamed to "elastic_repo_key"

Change mode of config file away from 644

Currently the Filebeat Configuration file is world readable:
https://github.com/elastic/ansible-beats/blob/main/tasks/beats-config.yml#L61

since it also can contain a username and password to an elasticsearch cluster, a mode of 0640 might be more appropriate

Using ansible-beats conflict with using ansible-elasticsearch at the same time

Here is the content of elastic.beats var/main.yml :

---
# vars file for beats

beats_package_url: "https://download.elastic.co/beats"
es_conf_dir: "/etc"

It overrides the es_conf_dir set by elastic.elasticsearch :

es_conf_dir: "/etc/elasticsearch"

That is not ok as when we are using both roles in the same playbook processsing, we got /etc folder which gets broken :

$ sudo ls -al /etc/ | grep elasticsearch
drwxr-xr-x 83 elasticsearch elasticsearch  4096 Jul 21 13:54 .
drwxr-s---  2 root          elasticsearch  4096 Jul 21 13:54 elasticsearch
-rw-rw----  1 elasticsearch elasticsearch   199 Jul 21 13:54 elasticsearch.keystore
-rw-r--r--  1 elasticsearch elasticsearch   351 Jul 21 13:54 elasticsearch.yml
-rw-r--r--  1 elasticsearch elasticsearch  2580 Jul 21 13:54 jvm.options
-rw-r--r--  1 elasticsearch elasticsearch  5090 Jul 21 13:54 log4j2.properties
drwxr-xr-x  4 elasticsearch elasticsearch  4096 Jun 18 17:36 security

More interesting, es_conf_dir is not used anywhere in elastic.beats role. Therefore, it does not seems to serve any purpose and should be removed.

Settings the variable es_conf_dir explicitly in inventory does not seem to change anything as elastic.beats one is still used.

Support Beats modules and Kibana Dashboard installation

We should be able to install Filebeat modules and Metricbeat modules using this Ansible role.
The implementation should be generic to allow management of modules for future other Beats in the same way.

In the same way we should be able to load Kibana Dashboards for all supported Beats.

Related to #13

Filebeat installs successfully but does not start/restart

Beats product: Filebeat

Beats version 7.5.1

Role version: elastic.beats 7.5.1

OS version (uname -a if on a Unix-like system):

Linux dev.sandbox.example 2.6.32-754.23.1.el6.x86_64 #1 SMP Thu Sep 26 12:05:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
OS Version: CentOS release 6.10 (Final)

Description of the problem including expected versus actual behaviour:
When i install role and run, https://github.com/elastic/ansible-beats/blob/master/tasks/beats-redhat.yml tasks are completed successfully, Filebeat is installed but fails when restart handler is called.

Playbook:

- name: Install Graylog Collector
  hosts: graylog_clients
  roles:
    - role: elastic.beats
  vars:
    beats_version: 7.5.1
    beat: filebeat
    beat_conf:
      inputs:
        - type: log
          enabled: true
          paths: []

Provide logs from Ansible:
Find attached a copy of the log
output.log

Beats logs if relevant:

2020-01-23T07:58:49.413-0500	INFO	[publisher]	pipeline/module.go:97	Beat name: dev.designintools.intel.com
2020-01-23T07:58:49.413-0500	INFO	instance/beat.go:402	filebeat stopped.
2020-01-23T07:58:49.422-0500	ERROR	instance/beat.go:916	Exiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?

Mismatch between release name v7.10.1 and version 7.10.1

Beats product: All

Beats version: 7.10.1

Role version: v7.10.1 (If using master please specify github sha)

OS version (uname -a if on a Unix-like system): any

Description of the problem including expected versus actual behaviour:
The latest beats role is published as v7.10.1. The specified version in the readme usage section states to use 7.10.1. Every previous version has been released without starting v e.g. 7.10.0.

Playbook:
No playbook. Just run ansible-galaxy install elastic.beats,7.10.1

Provide logs from Ansible:
$ ansible-galaxy install elastic.beats,7.10.1

Starting galaxy role install process
downloading role 'beats', owned by elastic
[WARNING]: - elastic.beats was NOT installed successfully: - the specified version (7.10.1) of elastic.beats was not found in the list of available versions ([{'id': 135579,
'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-09-03T15:46:53.277519Z', 'modified': '2020-09-03T15:46:53.277556Z', 'name': '7.9.1', 'version': '7.9.1',
'commit_date': '2020-09-03T11:43:18-04:00', 'commit_sha': '4ea2a005ed8dccac0623e45e586c9802ec84831c', 'download_url': 'https://github.com/elastic/ansible-
beats/archive/7.9.1.tar.gz', 'active': None}, {'id': 120806, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-03-04T18:09:06.421294Z', 'modified':
'2020-03-04T18:09:06.421322Z', 'name': '7.6.1', 'version': '7.6.1', 'commit_date': '2020-03-04T13:00:59-05:00', 'commit_sha': 'fe677ed42c7fb49149d143285ba57ab654006b4f',
'download_url': 'https://github.com/elastic/ansible-beats/archive/7.6.1.tar.gz', 'active': None}, {'id': 136809, 'url': '', 'related': {}, 'summary_fields': {}, 'created':
'2020-09-24T14:54:52.570184Z', 'modified': '2020-09-24T14:54:52.570220Z', 'name': '7.9.2', 'version': '7.9.2', 'commit_date': '2020-09-24T10:45:49-04:00', 'commit_sha':
'8d69347ddf43de2addd873a08d71c1f7db401840', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.9.2.tar.gz', 'active': None}, {'id': 129697, 'url': '', 'related':
{}, 'summary_fields': {}, 'created': '2020-06-18T15:13:21.707514Z', 'modified': '2020-06-18T15:13:21.707543Z', 'name': '7.8.0', 'version': '7.8.0', 'commit_date':
'2020-06-18T11:06:57-04:00', 'commit_sha': '12ef312357b5881aabba3c453f07a70533169e14', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.8.0.tar.gz', 'active':
None}, {'id': 126328, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-05-13T18:03:29.687904Z', 'modified': '2020-05-13T18:03:29.687933Z', 'name': '7.7.0',
'version': '7.7.0', 'commit_date': '2020-05-13T14:01:56-04:00', 'commit_sha': '978875bcd7ca5a9a95596f3cb400976134fd0d7e', 'download_url': 'https://github.com/elastic/ansible-
beats/archive/7.7.0.tar.gz', 'active': None}, {'id': 114101, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-01-21T20:05:03.316237Z', 'modified':
'2020-01-21T20:05:03.316264Z', 'name': '7.5.2', 'version': '7.5.2', 'commit_date': '2020-01-21T14:58:14-05:00', 'commit_sha': '6ea4b5780be25a74d0bd20e5cd98fe6cd2c2e75f',
'download_url': 'https://github.com/elastic/ansible-beats/archive/7.5.2.tar.gz', 'active': None}, {'id': 115673, 'url': '', 'related': {}, 'summary_fields': {}, 'created':
'2020-02-11T17:36:44.807545Z', 'modified': '2020-02-11T17:36:44.807575Z', 'name': '7.6.0', 'version': '7.6.0', 'commit_date': '2020-02-11T12:32:05-05:00', 'commit_sha':
'95994cc60a0fc278e7bae025fa0148f3e42be33c', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.6.0.tar.gz', 'active': None}, {'id': 133807, 'url': '', 'related':
{}, 'summary_fields': {}, 'created': '2020-08-18T15:31:10.811051Z', 'modified': '2020-08-18T15:31:10.811078Z', 'name': '7.9.0', 'version': '7.9.0', 'commit_date':
'2020-08-18T11:26:57-04:00', 'commit_sha': '82827d16a2d23aea46ba3ed5b864a8e4a263825e', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.9.0.tar.gz', 'active':
None}, {'id': 111698, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2019-12-18T19:27:03.483395Z', 'modified': '2019-12-18T19:27:03.483427Z', 'name': '7.5.1',
'version': '7.5.1', 'commit_date': '2019-12-18T14:22:24-05:00', 'commit_sha': 'ae3723679dd0f9f99435992d2d819bd30d6934b8', 'download_url': 'https://github.com/elastic/ansible-
beats/archive/7.5.1.tar.gz', 'active': None}, {'id': 110509, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2019-12-02T17:47:40.442061Z', 'modified':
'2019-12-02T17:47:40.442089Z', 'name': '7.5.0', 'version': '7.5.0', 'commit_date': '2019-12-02T12:43:50-05:00', 'commit_sha': 'e3ccf93598dcb97b16ef22688a4e70a006d470d7',
'download_url': 'https://github.com/elastic/ansible-beats/archive/7.5.0.tar.gz', 'active': None}, {'id': 138867, 'url': '', 'related': {}, 'summary_fields': {}, 'created':
'2020-10-22T15:24:48.972321Z', 'modified': '2020-10-22T15:24:48.972348Z', 'name': '7.9.3', 'version': '7.9.3', 'commit_date': '2020-10-22T10:32:23-04:00', 'commit_sha':
'1de7b6e8c810d51e94636c2fd7359258a5cba4c7', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.9.3.tar.gz', 'active': None}, {'id': 93439, 'url': '', 'related':
{}, 'summary_fields': {}, 'created': '2019-05-09T15:02:39.525053Z', 'modified': '2019-05-09T15:02:39.525076Z', 'name': '7.0.0', 'version': '7.0.0', 'commit_date':
'2019-05-09T10:57:40-04:00', 'commit_sha': 'd84339f790431414e42813e10413f5c6d6ca47f4', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.0.0.tar.gz', 'active':
None}, {'id': 128259, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-06-04T19:37:45.230744Z', 'modified': '2020-06-04T19:37:45.230771Z', 'name': '7.7.1',
'version': '7.7.1', 'commit_date': '2020-06-04T15:33:13-04:00', 'commit_sha': 'd4b5f6da58256179e4ad96f42154f9c7592caacf', 'download_url': 'https://github.com/elastic/ansible-
beats/archive/7.7.1.tar.gz', 'active': None}, {'id': 143086, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-12-09T15:33:49.652250Z', 'modified':
'2020-12-09T15:33:49.652276Z', 'name': 'v7.10.1', 'version': '7.10.1', 'commit_date': '2020-12-09T10:20:12-05:00', 'commit_sha': '14bd3c9321e0718deb802c099c06610d61cea888',
'download_url': 'https://github.com/elastic/ansible-beats/archive/v7.10.1.tar.gz', 'active': None}, {'id': 122753, 'url': '', 'related': {}, 'summary_fields': {}, 'created':
'2020-03-31T17:36:21.632389Z', 'modified': '2020-03-31T17:36:21.632416Z', 'name': '7.6.2', 'version': '7.6.2', 'commit_date': '2020-03-31T13:34:04-04:00', 'commit_sha':
'2a9edd1d680ce6d872d5f3a1df76f4566cd2443d', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.6.2.tar.gz', 'active': None}, {'id': 132459, 'url': '', 'related':
{}, 'summary_fields': {}, 'created': '2020-07-28T14:39:49.615763Z', 'modified': '2020-07-28T14:39:49.615791Z', 'name': '7.8.1', 'version': '7.8.1', 'commit_date':
'2020-07-28T10:32:03-04:00', 'commit_sha': '0d97824d76eb22eab86c6543ece46e9e5a59cdda', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.8.1.tar.gz', 'active':
None}, {'id': 140203, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2020-11-11T15:37:28.802654Z', 'modified': '2020-11-11T15:37:28.802677Z', 'name': '7.10.0',
'version': '7.10.0', 'commit_date': '2020-11-11T10:06:18-05:00', 'commit_sha': 'd792f2a0d2a32ccce8f9c33e6c75ac7605aa1359', 'download_url': 'https://github.com/elastic/ansible-
beats/archive/7.10.0.tar.gz', 'active': None}, {'id': 104836, 'url': '', 'related': {}, 'summary_fields': {}, 'created': '2019-10-01T16:01:47.344857Z', 'modified':
'2019-10-01T16:01:47.344885Z', 'name': '7.4.0', 'version': '7.4.0', 'commit_date': '2019-10-01T11:54:51-04:00', 'commit_sha': '092efb0dcd5358d0582c3a6c2947e8c8f41154dd',
'download_url': 'https://github.com/elastic/ansible-beats/archive/7.4.0.tar.gz', 'active': None}, {'id': 106933, 'url': '', 'related': {}, 'summary_fields': {}, 'created':
'2019-10-23T15:16:07.513213Z', 'modified': '2019-10-23T15:16:07.513238Z', 'name': '7.4.1', 'version': '7.4.1', 'commit_date': '2019-10-23T11:13:22-04:00', 'commit_sha':
'e50cb7f07b914a1a210037e5c7ab7a53913cf6c3', 'download_url': 'https://github.com/elastic/ansible-beats/archive/7.4.1.tar.gz', 'active': None}]).
ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.

Beats logs if relevant:

Remove version_lock option from the role

Do we really need version_lock parameter to be included in this role? This looks to be excessive since not directly related to role scenario. It breaks idempotency in many cases. I'd suggest to remove this option from the role. Version lock can be easily done by an additional playbook task after role if necessary. It can be included in README examples if needed.

Support For Elastic Cloud

Feature request

Provide support for Elastic Cloud, We could provide cloud.id and cloud.auth values ideally in Playbook.yml.

Add support for CentOS / RHEL 8.x

Beats supports CentOS / RHEL 8.x since 7.4.0.

This Ansible role should be updated to add centos-8 supports and test-kitchen tests.

Escape multiline.pattern

How is it possible to properly escape multiline.pattern? I did not find a way how pass single quotes to filebeat config.

multiline:
  pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'

becomes

multiline:
  pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'

And

multiline:
  pattern: "'^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'"

becomes

multiline:
  pattern: '''^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'''

According to https://www.elastic.co/guide/en/beats/filebeat/current//regexp-support.html it should remain "We recommend that you wrap regular expressions in single quotation marks to work around YAML’s string escaping rules. "

configuration per host

Hi all,

do you have a tip on how I can create specifc moduels configuration for specific hosts?
lest see that i've metric beats,

I want to monitor for all the host thsi

module: "system"
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
# - network
- process
enabled: true
period: 10s
processes: [".*"]
cpu_ticks: false

But for the postgres machine i want to add also additional modules, what can I do? how can I do that with this role?
Should I create various play with differnt confs?

'beat' host_var breaks restart handler

Beats product:

filebeat

Beats version

7.0.0

Role version: (If using master please specify github sha)

5ec7143

OS version (uname -a if on a Unix-like system):

Ubuntu 18.04

Description of the problem including expected versus actual behaviour:

Defining the beat variable as a host_var breaks the restart handler on a fresh playbook run, with: ERROR! 'beat' is undefined. It isn't, though because role tasks referencing the variable succeed. Repeated attempts to run the playbook move the error to the next task that triggers a handler notification.

Think it might due to this limitation: ansible/ansible#48466 (comment)

Expected behaviour is that the beats role can use a host_var.

Playbook:
Please specify the full playbook used to reproduce this issue.

playbook:

- hosts: appservers
  roles:
    - elastic.beats

host_vars:

beat: filebeat
beat_conf:
  inputs:
    - type: log
      enabled: true
      paths:
        - /opt/app/log/*.log

Provide logs from Ansible:

...
TASK [beats : Debian - Ensure apt-transport-https is installed] **********************************************************************************************
ERROR! 'beat' is undefined

Repeated runs move the error to the next task that triggers a handler notify.

Copy default ILM policy file: Destination /etc/filebeat not writable

Beats product:
filebeat

Beats version
7.9.3

Role version: (If using master please specify github sha)
7.9.3

OS version (uname -a if on a Unix-like system):

Linux fine-unicorn 5.4.0-53-generic #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behaviour:
User has no permission to write ILM policy file in /etc/filebeat or /var/lib/filebeat.

My current workaround is to call the role with become: true

Suggested fix:

# Copy default ILM policy file
 - name: Create default policies config directory
+  become: true
   file:
     path: '{{ beat_conf.setup.ilm.policy_file | dirname }}'
     state: directory
   when: default_ilm_policy is defined

 - name: Copy default ILM policy file for {{ beat }}
+  become: true
   copy:
     src: '{{default_ilm_policy}}'
     dest: '{{ beat_conf.setup.ilm.policy_file }}'
   when: default_ilm_policy is defined
   notify: restart the service

Playbook:
Please specify the full playbook used to reproduce this issue.

Provide logs from Ansible:

TASK [elastic.beats : Check pid_dir status] ****************************************************************************************************************
ok: [fine-unicorn]

TASK [elastic.beats : Create Config Directory] *************************************************************************************************************
ok: [fine-unicorn]

TASK [elastic.beats : Copy Default File for Instance] ******************************************************************************************************
ok: [fine-unicorn]

TASK [elastic.beats : Copy Configuration File for filebeat] ************************************************************************************************
--- before: /etc/filebeat/filebeat.yml
+++ after: /home/vos/.ansible/tmp/ansible-local-13326nob2bwxb/tmpbb1v9g5o/beat.yml.j2
@@ -41,21 +41,21 @@
 monitoring:
   elasticsearch: null
   enabled: true
 setup:
   dashboards:
     enabled: false
   ilm:
     enabled: true
     overwrite: false
     pattern: '{now/d}-000001'
-    policy_file: ilm-policy.json
+    policy_file: /etc/filebeat/ilm-policy.json
     policy_name: filebeat
     rollover_alias: filebeat-%{[agent.version]}
   kibana:
     host: https://kibana.lan.local:443
   template:
     enabled: false
     settings:
       index:
         number_of_replicas: 2
         number_of_shards: 1

changed: [fine-unicorn]

TASK [elastic.beats : Create default policies config directory] ********************************************************************************************
ok: [fine-unicorn]

TASK [elastic.beats : Copy default ILM policy file for filebeat] *******************************************************************************************
fatal: [fine-unicorn]: FAILED! => changed=false
  checksum: 3890fbca85f351d1cee8f796b7d41f2e67bfff0e
  msg: Destination /etc/filebeat not writable

PLAY RECAP *************************************************************************************************************************************************
fine-unicorn               : ok=21   changed=1    unreachable=0    failed=1    skipped=11   rescued=0    ignored=0

Beats logs if relevant:

Debian 11

on debian 11 using the ansible i've this problem:

TASK [elastic.beats : Debian - Ensure apt-transport-https is installed] ***************************************************************
FAILED - RETRYING: Debian - Ensure apt-transport-https is installed (3 retries left).
FAILED - RETRYING: Debian - Ensure apt-transport-https is installed (2 retries left).
FAILED - RETRYING: Debian - Ensure apt-transport-https is installed (1 retries left).
[WARNING]: Updating cache and auto-installing missing dependency: python-apt
fatal: [dev1.dev.]: FAILED! => {"attempts": 3, "changed": false, "cmd": "apt-get install --no-install-recommends python-apt -y -q", "msg": "E: Package 'python-apt' has no installation candidate", "rc": 100, "stderr": "E: Package 'python-apt' has no installation candidate\n", "stderr_lines": ["E: Package 'python-apt' has no installation candidate"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nPackage python-apt is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\nHowever the following packages replace it:\n  python3-apt-dbg python3-apt python-apt-doc python-apt-common\n\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package python-apt is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", "However the following packages replace it:", "  python3-apt-dbg python3-apt python-apt-doc python-apt-common", ""]}

Missing PID file

Hi,

Using debian to work with ES, Kibana and beat. Using this ansible role to deploy metricbeat works fine, still, i'm missing the PID file creation, nothing is created in /var/run, no PID file. I'm no linux expert, so i may be missing something =/

Here is the command line that starts the process :

/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs/var/log/metricbeat

The file that define service is /etc/init.d/metricbeat and contains :

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Metricbeat is a lightweight shipper for metrics."
NAME="metricbeat"
DAEMON=/usr/share/${NAME}/bin/${NAME}
DAEMON_ARGS="-c /etc/${NAME}/${NAME}.yml -path.home /usr/share/${NAME} -path.config /etc/${NAME} -path.data /var/lib/${NAME} -path.logs /var/log/${NAME}"
TEST_ARGS="-e test config"
PIDFILE=/var/run/metricbeat.pid
WRAPPER="/usr/share/${NAME}/bin/${NAME}-god"
BEAT_USER="root"
WRAPPER_ARGS="-r / -n -p $PIDFILE"
SCRIPTNAME=/etc/init.d/metricbeat

do_start()
{
	# Return
	#   0 if daemon has been started
	#   1 if daemon was already running
	#   2 if daemon could not be started
	start-stop-daemon --start \
                --pidfile $PIDFILE  \
		--exec $WRAPPER -- $WRAPPER_ARGS -- $DAEMON $DAEMON_ARGS \
		|| return 2
}

Here is my inventory :

beats:
  metricbeat:
    beat_conf:
      setup.dashboards.enabled: true
      metricbeat.modules:
        - module: system
          metricsets:
            - cpu
            - load
            - filesystem
            - fsstat
            - memory
            - network
            - process
          enabled: true
          period: 10s
          processes:
            - ".*"

and here is my role definition :

- hosts: servers
  roles:
    - role: ansible-beats
      beat: metricbeat
      beat_conf: "{{ beats.metricbeat.beat_conf }}"

Add support or example for `processors` config section

Describe the feature:

Add support or an example for the processors configuration section within this role.

There is this closed discuss question for this problem that did not have a resolution to this same challenge.

Beats product: Metricbeat

Beats version: 7.6.0

Role version: 7.6.0

OS version (uname -a if on a Unix-like system):

Beat host: Ubuntu container

root@redacted:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
root@redacted:/# uname -a
Linux 4768ab1b9aa9 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Controlling host: MacOS Catalina - 10.15.3 (19D76)

~ % uname -a
Darwin redacted.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

The playbook does produce the default configuration of Metricbeat v7 using the ansible-playbook. On Ubuntu, the default is this with the comments removed

root@redacted:/# grep -v "^\s*#" /etc/metricbeat/metricbeat.yml | grep -v "^\s*$"
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

If one uses a playbook such as the one listed below, one can get close but the processors are incorrect

root@redacted:/# cat /etc/metricbeat/metricbeat.yml
# Ansible managed

################### metricbeat Configuration #########################

############################# metricbeat ######################################
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup.template.settings:
  index.codec: best_compression
  index.number_of_shards: 1


###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

output:
  elasticsearch:
    hosts:
    - localhost:8200


############################# Logging #########################################

logging:
  files:
    rotateeverybytes: 10485760

This configuration will run Metricbeat but the metadata about the servers will be missing from the records so the Metrics UI in Kibana does not work properly.

Single quoting (') the ~ produces a config file that includes

processors:
- add_host_metadata: '~'
- add_cloud_metadata: '~'
- add_docker_metadata: '~'
- add_kubernetes_metadata: '~'

which causes Metricbeat to not run because it throws an error about string is not an object.

RUNNING HANDLER [elastic.beats : restart the service] **************************
fatal: [7a2b85391a72]: FAILED! => {"changed": false, "msg": "Failed to restart service: metricbeat", "rc": 1, "stderr": "2020-03-02T23:05:43.532Z\tINFO\tinstance/beat.go:622\tHome path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]\n2020-03-02T23:05:43.532Z\tINFO\tinstance/beat.go:630\tBeat ID: 05094e0e-7db0-4357-a1d0-f6a40f08eda8\n2020-03-02T23:05:43.532Z\tERROR\tinstance/beat.go:933\tExiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')\nExiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')\n",
 "stderr_lines": ["2020-03-02T23:05:43.532Z\tINFO\tinstance/beat.go:622\tHome path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]",
 "2020-03-02T23:05:43.532Z\tINFO\tinstance/beat.go:630\tBeat ID: 05094e0e-7db0-4357-a1d0-f6a40f08eda8",
 "2020-03-02T23:05:43.532Z\tERROR\tinstance/beat.go:933\tExiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')",
 "Exiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')"],
 "stdout": "   ...fail!\n", "stdout_lines": ["   ...fail!"]}

Double quoting (") the ~ produces a config that includes

processors:
- add_host_metadata: '~'
- add_cloud_metadata: '~'
- add_docker_metadata: '~'
- add_kubernetes_metadata: '~'

TASK [elastic.beats : Start metricbeat service] ********************************
fatal: [7a2b85391a72]: FAILED! => {"changed": false, "msg": "Failed to start service: metricbeat", "rc": 1, "stderr": "2020-03-02T23:09:51.996Z\tINFO\tinstance/beat.go:622\tHome path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]\n2020-03-02T23:09:51.996Z\tINFO\tinstance/beat.go:630\tBeat ID: 05094e0e-7db0-4357-a1d0-f6a40f08eda8\n2020-03-02T23:09:51.996Z\tERROR\tinstance/beat.go:933\tExiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')\nExiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')\n",
 "stderr_lines": ["2020-03-02T23:09:51.996Z\tINFO\tinstance/beat.go:622\tHome path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]",
 "2020-03-02T23:09:51.996Z\tINFO\tinstance/beat.go:630\tBeat ID: 05094e0e-7db0-4357-a1d0-f6a40f08eda8",
 "2020-03-02T23:09:51.996Z\tERROR\tinstance/beat.go:933\tExiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')",
 "Exiting: error initializing processors: can not convert 'string' into 'object' accessing 'processors.0.add_host_metadata' (source:'/etc/metricbeat/metricbeat.yml')"],
 "stdout": "   ...fail!\n", "stdout_lines": ["   ...fail!"]}

Playbook:

- name: Install and configure Beats
  hosts: all
  tasks:
    - name: 'Install Metricbeat'
      include_role:
        name: elastic.beats
      vars:
        beat: metricbeat
        beat_conf:
          metricbeat.config.modules:
              path: '${path.config}/modules.d/*.yml'
              reload.enabled: false
          setup.template.settings:
            index.number_of_shards: 1
            index.codec: best_compression
          processors:
            - add_host_metadata: ~
            - add_cloud_metadata: ~
            - add_docker_metadata: ~
            - add_kubernetes_metadata: ~
        output_conf:
          elasticsearch:
            hosts: ['localhost:8200']

Provide logs from Ansible:

Beats logs if relevant:

elastic / ansible-beats Goto Github PK

ansible-beats's People

Contributors

Stargazers

Watchers

Forkers

ansible-beats's Issues

Recommend Projects

Recommend Topics

Recommend Org