Time-based buffer sample about amazon-kinesis-connectors

[knorwood]

That property is referenced in the KinesisConnectorConfiguration here and is used by the BasicMemoryBuffer here to determine the result of the "shouldFlush" method.

[ananthrk]

My bad! I was using ver 1.0.0 and I see that the same has been introduced in Ver 1.1.0. Apologies. BTW, the elapsed milliseconds property is calculated and checked in the "shouldFlush"
method. This method will be invoked only from "processRecords" right? What happens in the case where we have some records in buffer that are not yet emitted, but there are no new messages being posted to the stream (and "callProcessRecordsEvenForEmptyList" is set to false)? How will the emitter be called when the time is elapsed and more importantly what happens to the pending checkpoint? Thanks again.

[knorwood]

That's correct, "shouldFlush" will only be called from "processRecords". In the scenario you describe the records would sit buffered until more records are put into the corresponding shard. The property "callProcessRecordsEvenForEmptyList" would need to be set to true to ensure that "processRecords" is eventually called in the absence of putting records into the stream.

There is no concept of a pending checkpoint, so there should not be any issue with a record processor sitting idle for a long time between flushes.

[ananthrk]

Thanks. Regarding the checkpoint, I was concerned about the scenario where the data sits in the buffer (yet to be flushed) and the process goes down. If we bring up another process what would be the starting point in the stream for the new process? Would it re-process the previously buffered entries?

[gauravgh]

Upon fail over, the new process will start consuming data from the last checkpoint.

Sincerely,
Gaurav

[samglo]

Hi Gaurav,

In my sample.properties file for kinesis-storm-spout, what should I set zookeeperPrefix to ? I set it to zookeeperPrefix = /storm but looking at my Storm UI properties, I'm not sure which variable and respective value to.

Thanks ahead of time for any reply.

Sam

[gauravgh]

Hi Sam,

Would you be able to post your question in the kinesis-storm-spout repository (https://github.com/awslabs/kinesis-storm-spout)?
Happy to answer it there.

Thank you,
Gaurav

[samglo]

Thanks Gaurav ahead of time for your replies, also thanks for your contributions to this topic.

I just submitted my question there per your recommendation.

Sam

On Tuesday, October 21, 2014 5:14 PM, gauravgh [email protected] wrote:

Hi Sam,
Would you be able to post your question in the kinesis-storm-spout repository (https://github.com/awslabs/kinesis-storm-spout)?
Happy to answer it there.
Thank you,
Gaurav
—
Reply to this email directly or view it on GitHub.