monetate / koupler Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Hi there, and thanks for koupler
- it is just what we were looking for.
A question on usage:
-delimiter <arg> delimiter between fields (default: ',')
-partitionKeyField <arg> zero-based index of field containing partition key (default: 0)
Might be just terminology confusion on my part, but the docs say Kinesis message data is a just a blob of data (which, in my case, will be a JSON-encoded string). What is the fields
concept here?
Additionally, if I wanted to use asdf123
as my partition key, and a JSON blob as my data, would the call (over HTTP) look like:
curl -d "asdf123,{\"key\": \"value\"}" http://192.168.99.123:4321/event
I appreciate the clarification here.
I'm trying to build/run koupler, and following the steps from the readme, I ran
gradle clean build copyRuntimeLibs
, which succeeded, but running ./koupler.sh
fails, due to a missing lib directory:
~/g/k/sh ❯❯❯ ./koupler.sh
ls: ./lib/*.jar: No such file or directory
Error: Could not find or load main class com.monetate.koupler.Koupler
How should I go about running this? Is the script outdated, or is some setup failing to create a lib directory? Any help would be appreciated.
Hi there.
Reading through the metrics section of the README, I noticed there were no metrics around errors. Surely there are error conditions in Koupler, such as with issues writing to a stream (I suppose throttling is a separate concern), and having error metrics exposes a method for monitoring the health of what's happening within Koupler.
Since this metric is not currently instrumented, how do you (or any other users of Koupler) ensure the health of Koupler, other than a basic monit
entry?
I appreciate your time and feedback.
Where in this repository/project can we modify the Kinesis endpoint similar to this:
client = new AmazonKinesisClient();
client.setEndpoint(endpoint, serviceName, regionId);
Thanks in advance!
I am by no definition a java developer, nor to I have any experience with gradle, but I noticed an intermittent build (test?) failure, it's pretty uncommon, but by simply looping gradle clean build copyRuntimeLibs
I've been able to recreate it several times.
I'll quantify what I mean by "uncommon". I had it happen twice within about 5 builds, so I set out to forcibly recreate it; the next one didn't happen until after over an hour of continuous builds (averaging 12 seconds a piece), but it does always eventually happen.
(this is what I used, pretty simple really)
while gradle clean build copyRuntimeLibs; do echo 'AGAIN!'; done
java.lang.AssertionError: Did not queue all records! expected:<5> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at com.monetate.koupler.PipeKouplerTest.test(PipeKouplerTest.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
And that number at the top: <5> but was:<3>
, the 3 is not constant, I've gotten it with 4 or 2 as well (I guess that might be obvious, but it might not be, I'm not too familiar with the project).
I'm on OSX, 10.11.3, but I've also recreated the same issue (identical stack trace) in Linux Mint, and with the java docker container (which I think is debian jessie?).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.