fluent-plugin-out-http-buffered's Introduction

Introduction¶ ↑

Here’s a fork of ablagoev’s plugin to meet our project specified needs.

fluent-out-http-buffered¶ ↑

This is an output plugin for Fluentd which deliveres buffered log messages to an http endpoint.

It has configurable read and open timeouts.

Clients can also configure which http response statuses should be retried (in most cases clients would want to retry on status 500).

Maximum number of records in a HTTP request can be configured. A chunk will be sent by multiple threads.

Additional headers can be configured with “additional_headers” property.

The plugin was influenced by following projects: standard http_output_plugin http-out-ext ogibayashi’s_fork niku4i’s_fork

Installation:¶ ↑

‘gem install fluent-out-http-buffered`

Usage:¶ ↑

# Configuration file fluent.conf
<match fluentd.test.*>
  type http_buffered
  flush_interval 2s
  #Endpoint for messages
  endpoint_url    http://localhost.local/api/<data.id> # <data.id> refres to data.id in the record like {"data"=> {"id"=> 1, "name"=> "foo"}}
  #HTTP method (default: post)
  http_method put
  #Max number of events per HTTP request
  http_event_limit 100
  #Comma separated list of http statuses which need to be retried
  http_retry_statuses 500, 403
  #Read timeout in seconds, supports floats
  http_read_timeout 2.2
  #Open timeout in seconds, supports floats
  http_open_timeout 2.34
  buffer_chunk_limit 1m
  buffer_path /tmp/buffer
  # Additional header (default: nil). It might be useful to use this with out_forest to include tag/hostname as header.
  additional_headers X-custom-header1=foo,X-custom-header2=bar
</match>