Missing logs when using systemd source about fluentd-kubernetes-sumologic

[darend]

Update

We traced the issue to fluentd running out of memory and constantly restarting. This was not immediately apparent as the pod was not exiting, but if you look in the logs you would see an event like:

2018-03-20 15:58:45 +0000 [info]: Worker 0 finished unexpectedly with signal SIGKILL

In the host logs you would see an event like:

Mar 20 15:58:44 ip-172-20-60-200 kernel: Memory cgroup out of memory: Kill process 22663 (ruby2.3) score 1971 or sacrifice child
Mar 20 15:58:44 ip-172-20-60-200 kernel: Killed process 22663 (ruby2.3) total-vm:5557248kB, anon-rss:350160kB, file-rss:50628kB

With help from @frankreno we traced the issue to the flush interval being defaulted to 30s when installing via the chart. This is because the chart does not override the value, its commented out: https://github.com/kubernetes/charts/blob/c0e593513184490f895b3a82375934545677c309/stable/sumologic-fluentd/values.yaml#L44. The default of 30s is then inherited from the Dockerfile:

fluentd-kubernetes-sumologic/Dockerfile

Line 26 in 6ba540d

ENV FLUSH_INTERVAL "30s"

On hosts with any volume of traffic this buffer can easily require more than the default 256MB than the chart limits fluentd to: https://github.com/kubernetes/charts/blob/c0e593513184490f895b3a82375934545677c309/stable/sumologic-fluentd/values.yaml#L154

We reduced the flush interval to 5s and increased the memory limit to 512mb and collection has been stable.

Possible Improvements

• Default the flush interval to a lower value within the Dockerfile and chart
• Allow the container to exit in the event of OOM. The current approach is nice when running fluentd outside kubernetes. However when run in k8s, its better to allow it exit and be restarted. This type of activity is far more likely to be noticed