winebarrel / fluent-plugin-kinesis-firehose

Fluentd output plugin for Amazon Kinesis Firehose.

fluent-plugin-kinesis-firehose

Fluentd output plugin for Amazon Kinesis Firehose.

Please use fluent-plugin-kinesis.

ChangeLog

• >= 0.1.2
  • Add retries_on_putrecordbatch option (PR#2)

Installation

Add this line to your application's Gemfile:

gem 'fluent-plugin-kinesis-firehose'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fluent-plugin-kinesis-firehose

Configuration

<match kinesis.data>
  type kinesis_firehose

  delivery_stream_name DeliveryStreamName

  #profile ...
  #credentials_path ...
  #aws_key_id ...
  #aws_sec_key ...
  region us-east-1
  #endpoint ...
  #retries_on_putrecordbatch 3

  #data_key data (default: nil)

  # Put a data_key value if data_key is set
  #   {... "data":"xxx" ...} -> xxx
  # Put a record as JSON if data_key is not set
  #   {... "data":"xxx" ...} -> {... "data":"xxx" ...}

  #append_new_line true

  #include_time_key false
  #include_tag_key false

  flush_interval 1s
</match>

fluent-plugin-kinesis-firehose's Issues

Getting more done in GitHub with ZenHub

Hola! @winebarrel has created a ZenHub account for the winebarrel organization. ZenHub is the leading team collaboration and project management solution built for GitHub.

---

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• “+1” button for GitHub issues and comments;
• Drag-and-drop file sharing;
• Time-saving shortcuts like a quick repo switcher.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @winebarrel.

10000件出力したら500件のみS3に転送されました

Apacheのログをsourceに設定しab -n 10000 http://localhost/とテストしたところ、S3には500件のみログが転送されました。
Firehose APIのPutRecordBatchの上限500レコードに引っかかっているのではないでしょうか。
http://docs.aws.amazon.com/firehose/latest/dev/limits.html

Gem dependency conflict

Hi there,

I'm running into a gem dependency conflict when I try to use this plugin alongside the fluent-plugin-s3. It appears that fluent-plugin-kinesis-firehose requires aws-sdk ~> 2.1.28, which causes a conflict with the subsequently dependent aws-sdk-resources package from the s3 plugin's aws-sdk ~> 2 dependency. Here's the exception I get during my Chef run:

---- Begin output of /etc/init.d/td-agent restart ----
           STDOUT: Restarting td-agent:  * td-agent
           STDERR: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:2112:in `raise_if_conflicts': Unable to activate aws-sdk-2.1.36, because aws-sdk-resources-2.2.14 conflicts with aws-sdk-resources (= 2.1.36) (Gem::ConflictError)
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1280:in `activate'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1314:in `block in activate_dependencies'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1300:in `each'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1300:in `activate_dependencies'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1282:in `activate'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems.rb:198:in `rescue in try_activate'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems.rb:195:in `try_activate'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:126:in `rescue in require'

            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kinesis-firehose-0.1.1/lib/fluent/plugin/out_kinesis_firehose.rb:1:in `<top (required)>'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin.rb:168:in `block in try_load_plugin'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin.rb:166:in `each'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin.rb:166:in `try_load_plugin'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin.rb:126:in `new_impl'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin.rb:57:in `new_output'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/agent.rb:127:in `add_match'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/agent.rb:60:in `block in configure'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/agent.rb:54:in `each'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/agent.rb:54:in `configure'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:82:in `configure'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:117:in `configure'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:91:in `run_configure'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:515:in `run_configure'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:186:in `dry_run'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:133:in `start'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/command/fluentd.rb:171:in `<top (required)>'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
            from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
            from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/bin/fluentd:6:in `<top (required)>'
            from /opt/td-agent/embedded/bin/fluentd:23:in `load'
            from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
            from /usr/sbin/td-agent:7:in `load'
            from /usr/sbin/td-agent:7:in `<main>'
           ---- End output of /etc/init.d/td-agent restart ----
       Ran /etc/init.d/td-agent restart returned 1

To reproduce:

1. Use base Ubuntu 12.04
2. sudo apt-get update
3. Install td-agent sudo apt-get install td-agent
4. Install both td-agent-gems: sudo td-agent-gem install fluent-plugin-s3 fluent-plugin-kinesis-firehose
5. Edit /etc/td-agent/td-agent.conf to include match blocks with both plugins

<match *>
    type s3
    aws_key_id ...
    aws_sec_key ...
    s3_bucket ...
    s3_region us-east-1
    time_slice_format %Y%m%d
    buffer_path /var/log/td-agent/s3-email
    path logs/app/email
    utc
</match>
<match *>
    type kinesis_firehose
    delivery_stream_name ...
    aws_key_id ...
    aws_sec_key ...
    region us-east-1
    append_new_line true
</match>

1. sudo service td-agent restart, which fails with the above error and stack trace

Any ideas on how to resolve this issue?