After 429 rate limiting, logs appear to stop being delivered to sumo via http about fluentd-kubernetes-sumologic

[jdoconnor]

more notes... seeing this on the logs of those that are failing

2017-10-06 17:47:32 +0000 [warn]: #0 fluent/root_agent.rb:314:handle_emits_error: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" tag="containers.mnt.log.containers.traefik-4h5j6_kube-system_traefik-c78f0fea26338d5a25dbb532319c83590c85c9ac30bb9908cfc2ad233193ec4f.log"

[frankreno]

Hi Jay:

Have you tried upping the number of threads and flush interval? These options can be set via the Environment Variables.

Regarding the Buffer Overflow exception, we run the with default in_tail fluentD buffer settings. Based on the docs, fluentD stops reading until the exception is resolved.

https://docs.fluentd.org/v0.12/articles/in_tail#what-happens-when-intail-receives-bufferqueuelimiterror

Currently we do not provide a way via environment variables to change the buffer configuration. You might need to tweak the default settings and create a custom image. If you are up for adding support to change those settings via a PR, we would welcome the PR.

https://docs.fluentd.org/v0.14/articles/buffer-section#type

[frankreno]

Hi Jay:

Were you able to leverage the plugin settings to prevent the buffer overflow?

[frankreno]

@jdoconnor wanted to revisit this issue. I have recently learned of a bug in the README. The README used to indicate that the FLUSH_INTERVAL was 5 seconds. This is not true, it is actually 30 seconds.

This means that we will not flush the buffer until 30 seconds. This can cause a large build up of logs, increasing memory usage and causing those errors where the Buffer is overflowing. You might want to try reducing that flush interval by setting the FLUSH_INTERVAL in the yaml file to 5s.

[frankreno]

closing due to lack of response and fixes that have been introduced...

[avdhoot]

Facing same issue. FLUSH_INTERVAL=5s
{"log":"2019-02-14 09:24:23 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/buffer.rb:269:in write'\" tag=\"xxxxxxx.log\"\n","stream":"stdout","time":"2019-02-14T09:24:23.595906259Z"}