Lambda function can't read S3 logs from Kinesis Firehose about agentless-integrations-for-aws

[tredman]

Hey, thanks for the report. I think this breakage is due to how we've tried to support gzip-compressed files that do not properly set Content-Encoding in the past by looking at other potential indicators. In hindsight, assuming all application/octet-stream content was gzipped was definitely wrong, and now we're in a position where "correct" behavior is breaking the integration. Oops :/

I put together #23 which should fix this issue. The plan is to break some use cases while providing a backwards-compatible flag to set as a workaround, while doing things correctly going forward. Will get it reviewed, but let me know if you see any issues with this approach.

[teajaymars]

Very impressed with the time it took to turn around!

I agree with the approach and think it looks basically fine. I am on the Honeycomb free trial so in order to get started quickly I've written my own Lambda (in Node.js). I've had success so far reading various log types with the following logic:

module.exports = async function s3FetchLogFile (Bucket, Key) {
  const params = { Bucket, Key }
  const response = await s3.getObject(params).promise()
  if (response.ContentType === 'application/x-gzip' || response.ContentEncoding === 'gzip') {
    return unzipBuffer(response.Body)
  } else {
    return response.Body.toString('utf8')
  }
}

Not sure if your Go version follows the same pattern, but this has successfully been auto-unzipping files with no false positives. Hope this helps.

[MikeGoldsmith]

Closing as I believe this was addressed as part of the v2.0.1 release.