Comments (1)
Update
We traced the issue to fluentd running out of memory and constantly restarting. This was not immediately apparent as the pod was not exiting, but if you look in the logs you would see an event like:
2018-03-20 15:58:45 +0000 [info]: Worker 0 finished unexpectedly with signal SIGKILL
In the host logs you would see an event like:
Mar 20 15:58:44 ip-172-20-60-200 kernel: Memory cgroup out of memory: Kill process 22663 (ruby2.3) score 1971 or sacrifice child
Mar 20 15:58:44 ip-172-20-60-200 kernel: Killed process 22663 (ruby2.3) total-vm:5557248kB, anon-rss:350160kB, file-rss:50628kB
With help from @frankreno we traced the issue to the flush interval being defaulted to 30s
when installing via the chart. This is because the chart does not override the value, its commented out: https://github.com/kubernetes/charts/blob/c0e593513184490f895b3a82375934545677c309/stable/sumologic-fluentd/values.yaml#L44. The default of 30s
is then inherited from the Dockerfile:
fluentd-kubernetes-sumologic/Dockerfile
Line 26 in 6ba540d
On hosts with any volume of traffic this buffer can easily require more than the default
256MB
than the chart limits fluentd to: https://github.com/kubernetes/charts/blob/c0e593513184490f895b3a82375934545677c309/stable/sumologic-fluentd/values.yaml#L154
We reduced the flush interval to 5s
and increased the memory limit to 512mb
and collection has been stable.
Possible Improvements
- Default the flush interval to a lower value within the Dockerfile and chart
- Allow the container to exit in the event of OOM. The current approach is nice when running fluentd outside kubernetes. However when run in k8s, its better to allow it exit and be restarted. This type of activity is far more likely to be noticed
from fluentd-kubernetes-sumologic.
Related Issues (20)
- Release as an independent GEM HOT 3
- Sourcecategories in Sumo appending numbers HOT 12
- Feature request - Migrating .conf files into configMap
- Kubernetes_meta and Kubernetes_meta_reduce break when condition is met.
- Add ability to control which metadata you want to include/exclude when reducing the metadata
- Memory usage growing constantly
- Duplicate Logs because of Redundant SourceCategories HOT 1
- emit transaction failed: error_class=NoMethodError error="undefined method `+' HOT 4
- fluentd daemonset unable to read log files under OCI OKE
- Sumologic sourceCategory not respecting sumologic.com/sourceCategory annotation HOT 2
- RBAC fluentd daemonset unable to read log files under OCI OKE
- ls: cannot access '/fluentd/etc/fluent.file.conf': No such file or directory HOT 4
- Allow ability to set annotations in the daemonSet in Helm Chart HOT 3
- Windows support HOT 1
- Worker 0 continuously restarts HOT 1
- Question: Can I reduce Kubernetes metadata but leave labels? HOT 1
- support for CRI-O logs HOT 1
- Docker build failing for v2.4.2 HOT 1
- POD metadata is not extracted to sourceName
- Memory consumption grows to max and never goes down HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluentd-kubernetes-sumologic.