fluent-plugins-nursery / fluent-plugin-map

fluent-plugin-map is the non-buffered plugin that can convert an event log to different event log(s)

fluent-plugin-map

fluent-plugin-map(out_map) is the non-buffered plugin that can convert an event log to different event log(s)

Requirements

fluent-plugin-map	fluentd	ruby
>= 0.2.0	>= v0.14.0	>= 2.1
< 0.2.0	>= v0.12.0	>= 1.9

MapFilter

Configuration

parameter	description	default
map	Specify rule to convert an event log	nil
time	Specify rule to convert a time. The format of time is an integer number of seconds since the Epoch	nil
record	Specify rule to convert a record.The format of record is hash.	nil
multi	Enable to output multi logs	false
timeout	Specify timeout	1
format	Specify format. If map is specified, the format is map. If time and record is specified, the format is record. (map, record, multimap)	nil

Example

This sample config filter code file and time file.

<source>
  @type tail
  format apache
  path /var/log/httpd-access.log
  tag tag
  @label @raw
</source>
<label @raw>
  <match **>
    @type copy
    <store>
      @type relabel
      @label @code
    </store>
    <store>
      @type relabel
      @label @time
    </store>
  </match>
</label>
<label @code>
  <filter **>
    @type map
    map ([time, {"code" => record["code"].to_i}])
  </filter>
  <match **>
    @type file
    path code.log
  </match>
</label>
<label @time>
  <filter **>
    @type map
    map ([time, {"time" => record["time"].to_i}])
  </filter>
  <match **>
    @type file
    path time.log
  </match>
</label>

The parameter "map" can use 2 variables in event log; time, record. The format of time is an integer number of seconds since the Epoch. The format of record is hash. The config file parses # as the begin of comment. So the "map" value cannot use #{tag} operation. This plugin can output multi logs by seting multi to true.

If you don't use multi option, you can use time, record parameter. The 2 following filter directive is same:

<filter tag>
  @type map
  map ([time, {"code" => record["code"].to_i}])
</filter>
<filter tag>
  @type map
  time time
  record ({"code" => record["code"].to_i})
</filter>

MapOutput

Configuration

parameter	description	default
map	Specify rule to convert an event log	nil
time	Specify rule to convert a time. The format of time is an integer number of seconds since the Epoch	nil
record	Specify rule to convert a record.The format of record is hash.	nil
multi	Enable to output multi logs	false
timeout	Specify timeout	1
format	Specify format. If map is specified, the format is map. If time and record and (tag or key) are specified, the format is record. (map, record)	nil
key	Specify the key in record. This parameter is deprecated.	nil
tag	Specify the tag in record	nil

Example

This sample config output code file and time file.

<source>
  @type tail
  format apache
  path /var/log/httpd-access.log
  tag tag
</source>
<match tag>
  @type map
  map ([["code." + tag, time, {"code" => record["code"].to_i}], ["time." + tag, time, {"time" => record["time"].to_i}]])
  multi true
</match>
<match code.tag>
  @type file
  path code.log
</match>
<match time.tag>
  @type file
  path time.log
</match>

The parameter "map" can use 3 variables in event log; tag, time, record. The format of time is an integer number of seconds since the Epoch. The format of record is hash. The config file parses # as the begin of comment. So the "map" value cannot use #{tag} operation. This plugin can output multi logs by seting multi to true.

If you don't use multi option, you can use key, time, record parameter. The 2 following match directive is same:

<match tag>
  @type map
  map (["code." + tag, time, {"code" => record["code"].to_i}])
</match>
<match tag>
  @type map
  tag ("code." + tag)
  time time
  record ({"code" => record["code"].to_i})
</match>

Note

you have to wrap some configuration values with parenthesis like ("code." + tag), to avoid parsing by Fluentd itself. See also: Config File Syntax - Fluentd

Copyright

• Copyright (c) 2015- Tomita Kohei
• Apache License, Version 2.0

fluent-plugin-map's Issues

multiline map directive

Using this plugin it became immediately apparent that the map directive can easily get out of hand in just one line and be completely unreadable. Is there any way to split the directive into multiple lines?

No License Specified

This product have been released under no license. Generally we cannot use products without license.

Please specify a license and release new version under that.

Light-weight timeout

Problem

This is performance improvement proposal.

Current implementation uses Timeout::timeout:

fluent-plugin-map/lib/fluent/plugin/map_support.rb

Line 106 in b2048ce

Timeout.timeout(@plugin.timeout){

Timeout::timeout is too slow because it launches one Thread per block.
In fluentd core, we implement light-weight timeout for parser plugin.
I think map plugin also uses this approach.

Here is benchmark:

require 'benchmark'
require 'benchmark/ips'

require 'timeout'
require_relative 'lib/fluent/plugin/parser'

checker = Fluent::Plugin::Parser::TimeoutChecker.new(10)
checker.start

def f
  nil
end

Benchmark.ips do |x|
  x.report "timeout" do
    Timeout::timeout(10) {
      f
    }
  end

  x.report "checker" do
    checker.execute {
      f
    }
  end
end

# result 
Warming up --------------------------------------
             timeout     5.279k i/100ms
             checker    67.093k i/100ms
Calculating -------------------------------------
             timeout     54.950k (± 3.1%) i/s -    279.787k in   5.096866s
             checker    818.054k (± 1.9%) i/s -      4.093M in   5.004828s

Comparison:
             checker:   818053.9 i/s
             timeout:    54949.9 i/s - 14.89x  slower

Steps to replicate

Don't depend on configuration

Expected Behavior

More better performance

Your environment

• OS version: All OS
• All fluentd version
• Latest version
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

Parse error with fluentd-0.10.57

When starting td-agent with a map plugin configuration like this

type map map [["code." + tag, time, {"code" => record["code"].to_i}], ["time." + tag, time, {"time" => record["time"].to_i}]] multi true

the following error is thrown.

Starting td-agent: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/basic_parser.rb:86:in `parse_error!': got incomplete JSON array configuration at td-agent.conf line 87,9 (Fluent::ConfigParseError)
86: multi true
87:

 ---------^
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/literal_parser.rb:235:in `scan_json'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/literal_parser.rb:56:in `parse_literal'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/v1_parser.rb:120:in `parse_element'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/v1_parser.rb:91:in `parse_element'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/v1_parser.rb:40:in `parse!'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config/v1_parser.rb:30:in `parse'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/config.rb:30:in `parse'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:385:in `apply_system_config'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/supervisor.rb:110:in `initialize'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/command/fluentd.rb:168:in `new'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/lib/fluent/command/fluentd.rb:168:in `<top (required)>'
from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.57/bin/fluentd:6:in `<top (required)>'
from /opt/td-agent/embedded/bin/fluentd:23:in `load'
from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
from /usr/sbin/td-agent:7:in `load'
from /usr/sbin/td-agent:7:in `<main>'
                                                       [FAILED]

The error is present for the record setting as well. I am running td-agent on CentOS 6.5 x86_64.

My apologies for the formatting of the error messages. I didn't find anything in the markup help that would allow me to group the error output.