salt-formulas / salt-formula-heka

Heka Formula

Heka is an open source stream processing software system developed by Mozilla. Heka is a Swiss Army Knife type tool for data processing.

Sample pillars

Log collector service

heka:
  log_collector:
    automatic_starting: true
    elasticsearch_host: 172.16.10.253
    elasticsearch_port: 9200
    enabled: true
    metric_collector_host: 127.0.0.1
    metric_collector_port: 5567
    poolsize: 100
    max_message_size: 262144

Default values:

• automatic_starting: true
• elastisearch_port: 9200
• enabled: false
• metric_collector_host: 127.0.0.1
• metric_collector_port: 5567
• poolsize: 100
• max_message_size: 262144

Local Metric collector service

heka:
  metric_collector:
    aggregator_host: 172.16.20.253
    aggregator_port: 5565
    automatic_starting: true
    enabled: true
    influxdb_database: lma
    influxdb_host: 172.16.10.101
    influxdb_password: lmapass
    influxdb_port: 8086
    influxdb_time_precision: ms
    influxdb_timeout: 500
    influxdb_username: lma
    nagios_host: 172.16.20.253
    nagios_host_dimension_key: nagios_host
    nagios_password: secret
    nagios_port: 5601
    nagios_username: nagiosadmin
    poolsize: 100
    max_message_size: 262144

Default values:

• aggregator_port: 5565
• automatic_starting: true
• enabled: false
• influxdb_port: 8086
• influxdb_time_precision: ms
• influxdb_timeout: 5000
• nagios_port: 8001
• poolsize: 100
• max_message_size: 262144

Remote Metric Collector service

heka:
  remote_collector:
    aggregator_host: 172.16.20.253
    aggregator_port: 5565
    amqp_exchange: nova
    amqp_host: 172.16.10.254
    amqp_password: workshop
    amqp_port: 5672
    amqp_user: openstack
    amqp_vhost: /openstack
    automatic_starting: false
    elasticsearch_host: 172.16.10.253
    elasticsearch_port: 9200
    enabled: true
    influxdb_database: lma
    influxdb_host: 172.16.10.101
    influxdb_password: lmapass
    influxdb_port: 8086
    influxdb_time_precision: ms
    influxdb_username: lma
    poolsize: 100
    max_message_size: 262144

Default values:

• aggregator_port: 5565
• amqp_exchange: nova
• amqp_port: 5672
• amqp_vhost: ''
• automatic_starting: true
• elastisearch_port: 9200
• enabled: false
• influxdb_port: 8086
• influxdb_time_precision: ms
• influxdb_timeout: 5000
• poolsize: 100
• max_message_size: 262144

Aggregator service

heka:
  aggregator:
    automatic_starting: false
    enabled: true
    influxdb_database: lma
    influxdb_host: 172.16.10.101
    influxdb_password: lmapass
    influxdb_port: 8086
    influxdb_time_precision: ms
    influxdb_username: lma
    nagios_default_host_alarm_clusters: 00-clusters
    nagios_host: 172.16.20.253
    nagios_host_dimension_key: nagios_host
    nagios_password: secret
    nagios_port: 5601
    nagios_username: nagiosadmin
    poolsize: 100
    max_message_size: 262144

Default values:

• automatic_starting: true
• enabled: false
• influxdb_port: 8086
• influxdb_time_precision: ms
• influxdb_timeout: 5000
• nagios_port: 8001
• nagios_default_host_alarm_clusters: 00-clusters
• poolsize: 100
• max_message_size: 262144

Ceilometer service

heka:
  ceilometer_collector:
    elasticsearch_host: 172.16.10.253
    elasticsearch_port: 9200
    enabled: true
    influxdb_database: lma
    influxdb_host: 172.16.10.101
    influxdb_password: lmapass
    influxdb_port: 8086
    influxdb_time_precision: ms
    influxdb_username: lma
    resource_decoding: false
    amqp_exchange: ceilometer
    amqp_host: 172.16.10.253
    amqp_port: 5672
    amqp_queue: metering.sample
    amqp_vhost: /openstack

Default values:

• automatic_starting: true
• elastisearch_port: 9200
• enabled: false
• influxdb_port: 8086
• influxdb_time_precision: ms
• influxdb_timeout: 5000
• poolsize: 100
• amqp_exchange: ceilometer
• amqp_port: 5672
• amqp_queue: metering.sample
• amqp_vhost: /openstack
• resource_decoding: false

Read more

• https://hekad.readthedocs.org/en/latest/index.html

Documentation and Bugs

To learn how to install and update salt-formulas, consult the documentation available online at:

http://salt-formulas.readthedocs.io/

In the unfortunate event that bugs are discovered, they should be reported to the appropriate issue tracker. Use Github issue tracker for specific salt formula:

https://github.com/salt-formulas/salt-formula-heka/issues

For feature requests, bug reports or blueprints affecting entire ecosystem, use Launchpad salt-formulas project:

https://launchpad.net/salt-formulas

You can also join salt-formulas-users team and subscribe to mailing list:

https://launchpad.net/~salt-formulas-users

Developers wishing to work on the salt-formulas projects should always base their work on master branch and submit pull request against specific formula.

https://github.com/salt-formulas/salt-formula-heka

Any questions or feedback is always welcome so feel free to join our IRC channel:

#salt-formulas @ irc.freenode.net

salt-formula-heka's Issues

Dates in syslog logs are incorrect

In the "stacklight" branch we use generic_syslog.lua from StackLight MOS as the syslog decoder instead of Heka's rsyslog.lua. The issue is that, contrary to rsyslog.lua, the generic_syslog.lua decoder does not accept a tz config option and actually does not work with non-UTC timezones. The consequence of this issue is logs displayed at incorrect dates (in the future) in Kibana.

This issue only applies to the "stacklight" branch.

missing hekad.toml

Missing a hekad.toml

➜  heka git:(master) ack 00-hekad               
heka/server.sls
26:/etc/heka/conf.d/00-hekad.toml:
28:  - source: salt://heka/files/00-hekad.toml
42:    - file: /etc/heka/conf.d/00-hekad.toml
61:    - file: /etc/heka/conf.d/00-hekad.toml
83:    - file: /etc/heka/conf.d/00-hekad.toml
101:    - file: /etc/heka/conf.d/00-hekad.toml
121:    - file: /etc/heka/conf.d/00-hekad.toml
139:    - file: /etc/heka/conf.d/00-hekad.toml
157:    - file: /etc/heka/conf.d/00-hekad.toml
175:    - file: /etc/heka/conf.d/00-hekad.toml
194:    - file: /etc/heka/conf.d/00-hekad.toml
➜  heka git:(master) ls heka/files/00-hekad.toml
ls: cannot access 'heka/files/00-hekad.toml': No such file or directory
➜  heka git:(master) find . -name "*hekad.toml"

missing files in generall.

---

      ID: /etc/heka/conf.d/40-encoder-es_json-es-json.toml            
Function: file.managed                                                
  Result: False                                                       
 Comment: Source file salt://heka/files/encoder/es-json.toml not found
 Started: 18:52:29.107150                                             
Duration: 6.213 ms                                                    
 Changes:                                                             

Incorrect version of the Heka package is installed

0.10.0 should be installed while it's 0.10.0b2.

# apt-cache policy heka
heka:
  Installed: 0.10.0b2
  Candidate: 0.10.0b2
  Version table:
 *** 0.10.0b2 0
        500 http://apt.tcpcloud.eu/nightly/ trusty/tcp amd64 Packages
        100 /var/lib/dpkg/status
     0.10.0b1 0
        500 http://apt.tcpcloud.eu/nightly/ trusty/tcp amd64 Packages
     0.10.0 0
        500 http://apt.tcpcloud.eu/nightly/ trusty/tcp amd64 Packages

See mozilla-services/heka#1892 for the details. Either 0.10.0b1/0.10.0b2 versions should be removed from the apt repositories or the formula should pin the correct version.