algas / fluent-plugin-retag

fluent-plugin-retag, a plugin for Fluentd

Overview

Output plugin only retagging.

• simple retag
• remove tag prefix
• add tag prefix

Configuration

Simple Configuration

To retag foo.bar to hoge.fuga:

<match foo.bar>
  @type retag
  tag hoge.fuga
</match>

To retag foo.bar.** to xyz.bar.**:

<match foo.bar>
  @type retag
  remove_prefix foo
  add_prefix xyz
</match>

Useful Configuration

If you want to use branch condition, it is useful to use out_copy with fluent-plugin-retag.

<match foo.bar.**>
  @type copy
  <store>
    @type retag
    add_prefix copied
  </store>
  <store>
    ...
  </store>
</match>

<match copied.foo.bar.**>
  ...
</match>

Copyright

• Copyright (c) 2013- Masahiro Yamauchi
• License
  • Apache License, Version 2.0

fluent-plugin-retag's Issues

Not pass some tests

Finished in 1.615287 seconds.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4 tests, 6 assertions, 0 failures, 3 errors, 0 pendings, 0 omissions, 0 notifications
25% passed
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2.48 tests/s, 3.71 assertions/s
rake aborted!

retag makes socket write timed-out

conf:

<source>
  @type unix
  @label @MY-LABEL
  path /var/run/td-agent/td-agent.sock
</source>

<label @MY-LABEL>
  <match **>
    @type retag
    add_prefix application
  </match>

  <match application.**>
    @type stdout
  </match>
</label>

Result: https://imgur.com/906355d4-9002-4060-9cf9-c64b1b33a26a

Plugin crashes after a short period of time

The following output is seen in logs

  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event.rb:107:in `each'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:72:in `process'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/plugin/output.rb:785:in `emit_sync'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event_router.rb:87:in `emit'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:73:in `block in process'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event.rb:107:in `each'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:72:in `process'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/plugin/output.rb:785:in `emit_sync'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event_router.rb:87:in `emit'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:73:in `block in process'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event.rb:107:in `each'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:72:in `process'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/plugin/output.rb:785:in `emit_sync'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event_router.rb:96:in `emit_stream'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event_router.rb:87:in `emit'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:73:in `block in process'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/event.rb:107:in `each'
  2019-07-25 16:39:05 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-retag-0.1.1/lib/fluent/plugin/out_retag.rb:72:in `process'

using the config

<match **-service**>
  @type retag
  add_prefix s4_
</match>
<filter com_**>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>
<match fluent.**>
  @type retag
  tag clear
</match>