How to define ILM policy from Ansible ? about ansible-beats

[jmlrt]

Hi @astik,
I didn't test it, but you should be able to define your ILM policy in beat_conf variable using Configure index lifecycle management doc:

beat_conf:
  ...
  setup.ilm.enabled: auto
  setup.ilm.rollover_alias: "filebeat"
  setup.ilm.pattern: "{now/d}-000001"

[astik]

Yes i am =)
I already work on the naive solution past week-end.

Still a WIP as it needs better polishing.

[astik]

Simple as that !
I can't test it for now, but it looks aligned with what the role will do.
Thanks for the info.

[astik]

Follow up on this ticket:

• by default, with version 7.6, beat will use respective policy name:
  • filebeat https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-policy_name-option
  • metricbeat https://www.elastic.co/guide/en/beats/metricbeat/current/ilm.html#setup-ilm-policy_name-option
• ilm definition from beat installation does not allow policy definition
• ilm policy definition should be elasticsearch responsability, not beats
• still, ilm policy for beats index is initialized by beat first run with a default policy, wo it may be beat's responsabilty after all for the default one

Default ILM policy are:

filebeat export ilm-policy
{
	"policy": {
		"phases": {
			"hot": {
				"actions": {
					"rollover": {
						"max_age": "30d",
						"max_size": "50gb"
					}
				}
			}
		}
	}
}

metricbeat export ilm-policy
{
	"policy": {
		"phases": {
			"hot": {
				"actions": {
					"rollover": {
						"max_age": "30d",
						"max_size": "50gb"
					}
				}
			}
		}
	}
}

• beat allow the definition of a non existing policy through the use of settings setup.ilm.policy_file (https://www.elastic.co/guide/en/beats/metricbeat/current/ilm.html#setup-ilm-policy_file-option and https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-policy_file-option)
• then we could choose to overwrite default policy with the new one through the settings setup.ilm.overwrite (https://www.elastic.co/guide/en/beats/metricbeat/current/ilm.html#setup-ilm-overwrite-option and https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-overwrite-option)

any chance to be able, from beat ansible role, to define default ILM policy that is created if none already exist ?
As the beat configuration is expecting a file path, we would need a way to create default policy json configuration file in the ansible-beat role

@jmlrt what do you think of that approach?

[astik]

FWIW, here is the change i had done to make it work:

• my playbook conf:

    - role: elastic.beats
      beat: metricbeat
      beat_conf:
        setup:
          dashboards.enabled: true
          ilm:
            policy_file: /etc/metricbeat/policies/my-metricbeat.json
            overwrite: true
        metricbeat.modules:
          - module: system
            metricsets:
              - cpu
              - load
              - ...
            enabled: true
            period: 10s
            processes:
              - ".*"
      default_ilm_policy: conf/elasticsearch/ilm-policies/my-metricbeat.json

(notice the setup.ilm.policy_file and setup.ilm.overwrite in beat_conf (standard stuff) and the default_ilm_policy parameter (new stuff).

• additions at the end of beats-config.yml:

# Copy default ILM policy file
- name: Create default policies config directory
  file:
    path: "{{ beats_conf_dir }}/policies"
    state: directory
  when: default_ilm_policy is defined

- name: Copy default ILM policy file for {{ beat }}
  copy:
    src: "{{default_ilm_policy}}"
    dest: "{{ beats_conf_dir }}/policies/{{default_ilm_policy | basename}}"
  when: default_ilm_policy is defined
  notify: restart the service

we create a new folder to store default policy (/etc/metricbeat/policies), then we copy our policy to this folder (/etc/metricbeat/policies/my-metricbeat.json).

This is a very naïve approach. It would be better not to have to set up setup.ilm.policy_file and have it automatically set up when default_ilm_policy is defined.

[jmlrt]

Hi @astik, that make sense.

Being able to copy a file and setup its path in the config file is something that we already do for TLS certs in ansible-elasticsearch (see elastic/ansible-elasticsearch@d7efa20).

Would you be interested to make a pull request for that?

[jmlrt]

fixed by #78