codekitchen / logstash-input-kinesis

Logstash Plugin for AWS Kinesis Input

Logstash AWS Kinesis Input Plugin

This repo has moved to the logstash-plugins GitHub organization.

logstash-input-kinesis's Issues

no such file to load -- logstash-input-kinesis_jars

tried using this gem but getting no such file to load -- logstash-input-kinesis_jars .

logstash-input-kinesis_jars is added to https://github.com/codekitchen/logstash-input-kinesis/blob/master/.gitignore and hence it is not working

NoMethodError 'reason'

We are using logstash-input-kinesis within a docker container (in Amazon ECS) built with the following Dockerfile:

FROM logstash:1.5.4-1
RUN plugin install --version 1.4.0 logstash-input-kinesis
RUN plugin install --version 0.1.0 logstash-output-amazon_es

Unfortunately in our logs we see the following exception nearly every second:

Oct 11, 2015 3:58:01 PM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask call
SEVERE: Application exception. 
org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `reason' for #<Java::ComAmazonawsServicesKinesisClientlibraryTypes::ShutdownInput:0x220cf90b>
    at LogStash::Inputs::Kinesis::Worker.shutdown(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kinesis-1.4.0-java/lib/logstash/inputs/kinesis/worker.rb:34)
    at LogStash$$Inputs$$Kinesis$$Worker_1186006010.shutdown(LogStash$$Inputs$$Kinesis$$Worker_1186006010.gen:13)

Is there a version incompatibility or what could be the reason for this issue?

Performance degradation over time

Hi,

I'm using logstash-input-kinesis-1.4.3 and running into some performance degradation over time. On a fresh EC2 instance, fresh install performance is decent (1200-1400 eps) for the size of the instance. But over time (few hours), this comes down to 90-200 eps.

Looking at Kinesis stats, I see the gradual decline, which never recovers. If I kill logstash and start again, the performance results are on the same lower range. Re-installing the plugin gives a momentary boost to 600 eps, but within few minutes it comes down to 90-200 eps.

The instance being used is t2.medium. I have enough cpu credits to warrant out throttling on aws end.

There is no heavy loading on cpu or memory pressure

top - 23:44:21 up 1 day,  1:16,  1 user,  load average: 0.10, 0.09, 0.12
Tasks: 106 total,   2 running, 104 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.2 us,  0.0 sy,  0.0 ni, 92.7 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem:   4046872 total,  1834504 used,  2212368 free,    99380 buffers
KiB Swap:        0 total,        0 used,        0 free.  1210504 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                       
 6967 ubuntu    20   0 3560200 375684  15924 S  15.3  9.3   1:58.98 java                                                                                                          
    1 root      20   0   33508   2844   1492 S   0.0  0.1   0:01.22 init                                                                                                          
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd                                                                                                      
    3 root      20   0       0      0      0 S   0.0  0.0   0:02.25 ksoftirqd/0

$ free -m
 total       used       free     shared    buffers     cached
Mem:          3952       1791       2160          0         97       1182
-/+ buffers/cache:        511       3440
Swap:            0          0          0

/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -w 2 -b 500

I'm trying to profile the plugin if its getting stuck on specific call etc but haven't made much headway. Cant get logstash to print my debug statements :(

Any pointers or tips to debug this further would be helpful!

Thanks.

Logs missing in ES when aggregating logs

At the moment we are using https://github.com/samcday/logstash-output-kinesis to ingest logs into kinesis and we discovered today that many logs are missing if we use https://github.com/samcday/logstash-output-kinesis#record-aggregation. Once we disable log aggregation ingestion works as expected but ingestion get too slow ~2 hours delay.

It is also to notice than the cloudwatch metric for user pending records is at 20k records if aggregation is on, while it starts to get emptier when aggregation is disabled.

Our question is if this is related to the output or the input which we haven't been able to debug yet in a reliable way.

c/ @samcday

Multiple logstash instances reading from same kinesis stream

Wasn't sure where or how to ask you this, but if I have multiple logstash instances reading from the same kinesis stream, would I get duplicates? I am basing this on my understanding of the redis -> logstash model. Right now I have multiple logstash instances reading from one redis cluster, and seems like there is some type of read lock to prevent duplicate reads, and I was wondering if this has the same capability.

Support for Logstash 2.0

Apparently the plugin is not supported on logstash 2.0? At least the documentation mentions only logstash 1.5.

"Error processing record: Input length = 1"

I'm processing CloudWatch logs (via Kinesis subscription) and am getting the following error for all entries, no matter the length. Thoughts?

"Error processing record: Input length = 1", :level=>:error, :file=>"logstash/inputs/kinesis/worker.rb", :line=>"54", :method=>"process_record"

Error processing record.getData: java.nio.HeapByteBuffer[pos=0 lim=611 cap=611]

Any idea why we get this error when trying to use the plugin?

:message=>"Error processing record.getData: java.nio.HeapByteBuffer[pos=0 lim=611 cap=611]", :level=>:error, :file=>"logstash/inputs/kinesis/worker.rb", :line=>"68", :method=>"process_record"

Thanks

Warnings in logstash logs on startup: "deprecated in favor of declaring the version"

This doesn't seem to affect function at all, it is more of a minor annoyance. On startup we are seeing the following warning in the logstash logs:

kinesis plugin is using the 'milestone' method to declare the version of the plugin this method is deprecated in favor of declaring the version inside the gemspec.

Thanks for a great plugin.
Chris