Load the index templates / mappings about ansible-role-beats

[ageis]

Hrm, your old ansible-role-elk has mappings.yml, but more importantly to note, there's a new approach to this in version 6.0. See here and here.

Apparently, the mappings are moved to fields.yml which is a different format... No more template JSON. Normally, if the output were ES, then the "index template [is loaded] automatically after successfully connecting to [ES]". However, with a Logstash output (such is this case) then you must either load the template manually, or bypass Logstash, which is shown in the second option above.

In order to get the template JSON file, you must export it first, e.g.:

metricbeat export template > metricbeat.template.json
filebeat export template > filebeat.template.json

It's not included in the .deb package anymore! Then you can POST it to localhost:9200 as usual.

[ageis]

Thought I'd note the old elasticsearch-template-logstash-output.json from your ELK role is not compatible with ElasticSearch 6.x...

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_default_]: No field type matched on [float], possible values are [object, string, long, double, boolean, date, binary]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_default_]: No field type matched on [float], possible values are [object, string, long, double, boolean, date, binary]","caused_by":{"type":"illegal_argument_exception","reason":"No field type matched on [float], possible values are [object, string, long, double, boolean, date, binary]"}},"status":400}

[msheiny]

Heyyy @ageis thanks for bringing this up.

I've been loading templates via logstash -- our ELK role is deprecated, no longer being used, and our configs are currently being a private repo. The export part currently looks something like this:

  if [type] == "metricsets" {
      elasticsearch {
        hosts => ["{{logstash_es_proto}}://{{ logstash_es_host }}:{{logstash_es_port}}"]
        index => "{{ logstash_metrics_index | default('metrics-logstash-%{+YYYY.MM.dd}') }}"
{% if logstash_metrics_template is defined %}
        template => "{{ logstash_template_folder }}/{{ logstash_metrics_template }}"
        template_name => "metrics-logstash-custom"
{% endif %}
{% if logstash_es_proto == "https" %}
        ssl => true
        cacert => "{{ logstash_es_cacert }}"
        ssl_certificate_verification => {{logstash_es_verifyssl|default(true)|lower }}
{% endif %}
      }
  }
  # send syslog type events a different consolidated index
  #
  #
  else {
      elasticsearch {
        hosts => ["{{logstash_es_proto}}://{{ logstash_es_host }}:{{logstash_es_port}}"]
        index => "{{ logstash_syslog_index | default('syslog-logstash-%{+YYYY.MM.dd}') }}"
{% if logstash_syslog_template is defined %}
        template => "{{ logstash_template_folder }}/{{ logstash_syslog_template }}"
        template_name => "syslog-logstash-custom"
{% endif %}
{% if logstash_es_proto == "https" %}
        ssl => true
        cacert => "{{ logstash_es_cacert }}"
        ssl_certificate_verification => {{logstash_es_verifyssl|default(true)|lower }}
{% endif %}
      }
  }

I guess technically we could as you mention, use beats to import these templates. I don't plan on adding that functionality here but I'd be happy to take PRs! 👼 We should probably get around to opening our logstash role. Its pretty basic though and needs some CI love.

[msheiny]

note the above logstash template doesnt really work for 6.x since they are deprecating the type field, going to either start using tags or custom fields. Still trying to figure that bit out.

[ageis]

@msheiny Hrm, interesting. After seeing how you've created separate indices for metrics and set the template upon ingesting the events, I am actually curious to see more of what your Logstash input/output chain looks like (I mean after all that Jinja2 logic gets evaluated), so PM me on Gitter if you want or might care to share. It could be helpful to me for a fleet of machines I'm working with these days.