This repo has moved to the logstash-plugins GitHub organization.
codekitchen / logstash-input-kinesis Goto Github PK
View Code? Open in Web Editor NEWLogstash Plugin for AWS Kinesis Input
Logstash Plugin for AWS Kinesis Input
This repo has moved to the logstash-plugins GitHub organization.
tried using this gem but getting no such file to load -- logstash-input-kinesis_jars
.
logstash-input-kinesis_jars
is added to https://github.com/codekitchen/logstash-input-kinesis/blob/master/.gitignore and hence it is not working
We are using logstash-input-kinesis within a docker container (in Amazon ECS) built with the following Dockerfile:
FROM logstash:1.5.4-1
RUN plugin install --version 1.4.0 logstash-input-kinesis
RUN plugin install --version 0.1.0 logstash-output-amazon_es
Unfortunately in our logs we see the following exception nearly every second:
Oct 11, 2015 3:58:01 PM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask call
SEVERE: Application exception.
org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `reason' for #<Java::ComAmazonawsServicesKinesisClientlibraryTypes::ShutdownInput:0x220cf90b>
at LogStash::Inputs::Kinesis::Worker.shutdown(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kinesis-1.4.0-java/lib/logstash/inputs/kinesis/worker.rb:34)
at LogStash$$Inputs$$Kinesis$$Worker_1186006010.shutdown(LogStash$$Inputs$$Kinesis$$Worker_1186006010.gen:13)
Is there a version incompatibility or what could be the reason for this issue?
Hi,
I'm using logstash-input-kinesis-1.4.3 and running into some performance degradation over time. On a fresh EC2 instance, fresh install performance is decent (1200-1400 eps) for the size of the instance. But over time (few hours), this comes down to 90-200 eps.
Looking at Kinesis stats, I see the gradual decline, which never recovers. If I kill logstash and start again, the performance results are on the same lower range. Re-installing the plugin gives a momentary boost to 600 eps, but within few minutes it comes down to 90-200 eps.
The instance being used is t2.medium. I have enough cpu credits to warrant out throttling on aws end.
There is no heavy loading on cpu or memory pressure
top - 23:44:21 up 1 day, 1:16, 1 user, load average: 0.10, 0.09, 0.12
Tasks: 106 total, 2 running, 104 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.2 us, 0.0 sy, 0.0 ni, 92.7 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem: 4046872 total, 1834504 used, 2212368 free, 99380 buffers
KiB Swap: 0 total, 0 used, 0 free. 1210504 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6967 ubuntu 20 0 3560200 375684 15924 S 15.3 9.3 1:58.98 java
1 root 20 0 33508 2844 1492 S 0.0 0.1 0:01.22 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:02.25 ksoftirqd/0
$ free -m
total used free shared buffers cached
Mem: 3952 1791 2160 0 97 1182
-/+ buffers/cache: 511 3440
Swap: 0 0 0
/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -w 2 -b 500
I'm trying to profile the plugin if its getting stuck on specific call etc but haven't made much headway. Cant get logstash to print my debug statements :(
Any pointers or tips to debug this further would be helpful!
Thanks.
At the moment we are using https://github.com/samcday/logstash-output-kinesis to ingest logs into kinesis and we discovered today that many logs are missing if we use https://github.com/samcday/logstash-output-kinesis#record-aggregation. Once we disable log aggregation ingestion works as expected but ingestion get too slow ~2 hours delay.
It is also to notice than the cloudwatch metric for user pending records is at 20k records if aggregation is on, while it starts to get emptier when aggregation is disabled.
Our question is if this is related to the output or the input which we haven't been able to debug yet in a reliable way.
c/ @samcday
Wasn't sure where or how to ask you this, but if I have multiple logstash instances reading from the same kinesis stream, would I get duplicates? I am basing this on my understanding of the redis -> logstash model. Right now I have multiple logstash instances reading from one redis cluster, and seems like there is some type of read lock to prevent duplicate reads, and I was wondering if this has the same capability.
Apparently the plugin is not supported on logstash 2.0? At least the documentation mentions only logstash 1.5.
I'm processing CloudWatch logs (via Kinesis subscription) and am getting the following error for all entries, no matter the length. Thoughts?
"Error processing record: Input length = 1", :level=>:error, :file=>"logstash/inputs/kinesis/worker.rb", :line=>"54", :method=>"process_record"
Any idea why we get this error when trying to use the plugin?
:message=>"Error processing record.getData: java.nio.HeapByteBuffer[pos=0 lim=611 cap=611]", :level=>:error, :file=>"logstash/inputs/kinesis/worker.rb", :line=>"68", :method=>"process_record"
Thanks
This doesn't seem to affect function at all, it is more of a minor annoyance. On startup we are seeing the following warning in the logstash logs:
kinesis plugin is using the 'milestone' method to declare the version of the plugin this method is deprecated in favor of declaring the version inside the gemspec.
Thanks for a great plugin.
Chris
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.