fluentd_logger ruby process using 80% memory and causing instance to restart about appengine-sidecars-docker

[Cynocracy]

Apologies for the late response.

This is a known issue caused by a memory leak in the google-cloud-logging plugin fixed by:
#86

We were made aware of the issue through standard support channels on Dec 14th, and have taken steps to move all customers off of images with the broken sidecar as of Dec 20th. If you believe this is still occurring with your application, feel free to reopen and we will investigate.

[bonesyo]

@Cynocracy Thank you for the response! What can I do to ensure we are on the new image? It does not seem to be running away anymore, but our fluentd process seems to be running around 23% (230mb). Does this sound normal to you?

[Cynocracy]

Fluentd has been seen to be fairly resource intensive in the past (especially depending on the log volume). 230mb doesn't sound unreasonable to me, so long as the usage isn't growing continuously as you saw before.

You can check that you're on the newest image (or at least an unaffected image) by checking the version of the google-cloud-logging plugin in the fluentd sidecar:

sudo su
docker exec -it fluentd_logger google-fluentd-gem list fluent-plugin-google-cloud

This will show 0.6.8 if the image is impacted by the memory leak, and another version (likely 0.6 or 0.6.12) if not.

[bonesyo]

@Cynocracy I just ran that command and we are on 0.6.8. How do I update to 0.6.12?

[Cynocracy]

Redeploying should pick up the latest image (with a fixed sidecar).

If you file a support request and reference this, I'd be happy to dig into why you ended up with the memleaked sidecar.

[Cynocracy]

Ah, I apologize, I was slightly inaccurate, it looks like the issue was in reality introduced by grpc >= 1.3

Checking the version of the grpc plugin reveals

grpc (1.2.5 x86_64-linux) on the "latest" image (actually a rollback), despite the command I pasted above showing fluent-plugin-google-cloud (0.6.8, 0.6.0).

I'm admittedly not an expert on the gem system, but it appears the memory leak isn't included so long as the grpc output is < 1.3. Which means we're good :) or well, the latest image is.

[bonesyo]

Ok, it looks like we are on 1.2.5! Thank you for the help!