Add support or example for `processors` config section about ansible-beats

[a03nikki]

Here is what I've established so far.

First, I looked up in the YAML v1.3 spec that the ~ character indicates null.

From 10.3.2. Tag Resolution:

Regular expression	Resolved to tag
null | Null | NULL | ~	tag:yaml.org,2002:null

Next, I reviewed the documentation for the add_host_metadata processor. I observed that all of the parameters on the processor are optional.

Therefore, when ~ is used in the configuration file, the processor is enabled using the default settings.

[a03nikki]

So I've retried the ~ in the Ansible file again, and it does appear that the metadata is populated on the records

processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null

So I'm confused. 😕

This does appear to work as well (sending an empty map by {})

processors:
- add_host_metadata: {}
- add_cloud_metadata: {}
- add_docker_metadata: {}
- add_kubernetes_metadata: {}

[a03nikki]

😕 I am also not certain where the undesirable behavior is coming from any more (Ansible vs. Metricbeat).

Metricbeat uses gopkg.in/yaml.v2 to parse the metricbeat.yml file, if I am reading the code correctly.

There are a number of issues (both open and closed) related to the underlying Go library used by Metricbeat to handle parsing the YAML configuration. So that could be an avenue worth investigating.

[jmlrt]

Hi @a03nikki,
Indeed ~ is translated as null value by yaml which both Ansible and Beats are using, so /etc/metricbeat/metricbeat.yml will be set with add_XXX_metadata: null.
However this configuration seem working as I'm able to retrieve cloud metadata on a GCP instance:

• playbook:

- hosts: localhost
  roles:
    - role: elastic.elasticsearch
    - role: elastic.beats
      beat: metricbeat
      beat_conf:
        metricbeat.config.modules:
          path: '${path.config}/modules.d/*.yml'
          reload.enabled: false
        setup.template.settings:
          index.number_of_shards: 1
          index.codec: best_compression
        processors:
          - add_host_metadata: ~
          - add_cloud_metadata: ~
          - add_docker_metadata: ~
          - add_kubernetes_metadata: ~

• generated /etc/metricbeat/metricbeat.yml:

# Ansible managed

################### metricbeat Configuration #########################

############################# metricbeat ######################################
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup.template.settings:
  index.codec: best_compression
  index.number_of_shards: 1


###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

output:
  elasticsearch:
    hosts:
    - localhost:9200


############################# Logging #########################################

logging:
  files:
    rotateeverybytes: 10485760

• Metricbeat logs:

Mar 04 14:53:20 jmlrt-test systemd[1]: Started Metricbeat is a lightweight shipper for metrics..
...
Mar 04 14:53:21 jmlrt-test metricbeat[29405]: 2020-03-04T14:53:21.034Z        INFO        add_cloud_metadata/add_cloud_metadata.go:93        add_cloud_metadata: hosting provider type detected as gcp, metadata={"availability_zone":"europe-west1-b","instance":{"id":"1000121967836269311","name":"jmlrt-test"},"
...
Mar 04 14:53:21 jmlrt-test metricbeat[29405]: 2020-03-04T14:53:21.050Z        INFO        instance/beat.go:439        metricbeat start running.

• Example of record in Elasticsearch:

{
  "_index" : "metricbeat-7.6.0-2020.03.04-000001",
  "_type" : "_doc",
  "_id" : "39kVpnAB1zu53QkUCx9J",
  "_version" : 1,
  "_seq_no" : 9,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "@timestamp" : "2020-03-04T15:05:58.970Z",
    "system" : {
      "filesystem" : {
        "used" : {
          "pct" : 0.0343,
          "bytes" : 3756032
        },
        "device_name" : "/dev/sda15",
        "mount_point" : "/boot/efi",
        "type" : "vfat",
        "total" : 109422592,
        "free" : 105666560,
        "available" : 105666560,
        "files" : 0,
        "free_files" : 0
      }
    },
    "ecs" : {
      "version" : "1.4.0"
    },
    "host" : {
      "hostname" : "jmlrt-test",
      "architecture" : "x86_64",
      "os" : {
        "name" : "Ubuntu",
        "kernel" : "5.0.0-1031-gcp",
        "codename" : "bionic",
        "platform" : "ubuntu",
        "version" : "18.04.4 LTS (Bionic Beaver)",
        "family" : "debian"
      },
      "name" : "jmlrt-test",
      "id" : "6c148824a8b9c66a44c3da2ce1402ec1",
      "containerized" : false
    },
    "agent" : {
      "version" : "7.6.0",
      "type" : "metricbeat",
      "ephemeral_id" : "d96c180a-c174-47de-bb3e-336ff82e6dad",
      "hostname" : "jmlrt-test",
      "id" : "07b0324a-f0fc-43a6-8289-6e5f5c27dd45"
    },
    "cloud" : {
      "project" : {
        "id" : "elastic-infra"
      },
      "provider" : "gcp",
      "instance" : {
        "name" : "jmlrt-test",
        "id" : "1000121967836269311"
      },
      "machine" : {
        "type" : "n2-standard-2"
      },
      "availability_zone" : "europe-west1-b"
    },
    "event" : {
      "dataset" : "system.filesystem",
      "module" : "system",
      "duration" : 1589087
    },
    "metricset" : {
      "name" : "filesystem",
      "period" : 60000
    },
    "service" : {
      "type" : "system"
    }
  }
}

[a03nikki]

@jmlrt : I am glad it is working for you too. I think it is sufficient to say that there is not a bug in Ansible or Metricbeat.

So maybe the best option is to document a solution in the README.md so other people don't also have to struggle with this?

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue has been automatically closed because it has not had recent activity since being marked as stale.