Exceeding 4MB limit about lambda-streams-to-firehose

[benoittgt]

Thanks for the pick. Do you think it's relevant to make that change by default in the lambda?

[jonchase]

Changing the value from 4->2MB resolved our issue. I'm not sure lowering the value is the right way to go about fixing it though - it seems like somehow conditions were allowing a >4MB payload to be submitted, which sounds like the root cause to identify and fix.

[ysamlan]

I stumbled on this in some google spelunking - the AWS docs are a bit vague on this but while the 1MB per-record limit on Firehose is on the raw record data before encoding, I think the 4MB total request size limit might be on the full (encoded) payload. The SDK base64-encodes and JSON-wraps the data, so you probably need to pad this limit out by 33% for the base64, plus whatever the JSON around it works out to - someone at Amazon would have to confirm that though.

[IanMeyers]

Looking - are you using any transformers in your installation?

[IanMeyers]

I expect the issue is that the request limit is actually 4MiB, not 4MB as configured in the system. 1.5.2 and 374158a addresses this.

[ysamlan]

Yup - I asked AWS support and they actually said the base64/JSON overhead shouldn't count. But that being said, they're a bit fuzzy on MiB vs MB depending which part of the docs you look at, so probably 4 * 1000 * 1000 is a safer bet than 4 * 1024 * 1024.

[mbaran90]

Noticed that it calculates FIREHOSE_MAX_BATCH_BYTES based on current record size instead of actual next next size.

• Current code will work only if we records are with constant size. other wise will get "Exceeding 4MB limit" issue.
  I have made merge request for code fix. #64