No user-record aggregation about amazon-kinesis-agent

[chaochenq]

The data aggregation feature is not currently available but is on our roadmap. Is that the main reason that you always got throttled in #29?

[mattford63]

Yeah pretty much.

[pcurry]

+1 please get to record aggregation. It should help with the cost arguments I'm about to have.

[rgarcia]

+1 record aggregation would be big for us

[nikegp]

Same here. Any news re the roadmap plans?

Thanks!

[cohi]

+1 Same here - interested in record aggregation in order to increase throughput.

[chaochenq]

For users who stream data for a Kinesis Firehose deliveryStream, we have this option that you can add to your fileFlow

"aggregatedRecordSizeBytes"

Agent will parse until the last line of your file to which the aggregated bytes is less than or equal to the specified aggregatedRecordSizeBytes.

See

https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/AggregationSplitter.java

https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/FirehoseFileFlow.java#L42

But we still don't have a solution to Kinesis data streams (kinesisStream) similar to KPL. Having Agent enable KPL is something that we haven't been able to prioritize.

[jeisinge]

Any update here?