Comments (6)
more notes... seeing this on the logs of those that are failing
2017-10-06 17:47:32 +0000 [warn]: #0 fluent/root_agent.rb:314:handle_emits_error: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" tag="containers.mnt.log.containers.traefik-4h5j6_kube-system_traefik-c78f0fea26338d5a25dbb532319c83590c85c9ac30bb9908cfc2ad233193ec4f.log"
from fluentd-kubernetes-sumologic.
Hi Jay:
Have you tried upping the number of threads and flush interval? These options can be set via the Environment Variables.
Regarding the Buffer Overflow exception, we run the with default in_tail fluentD buffer settings. Based on the docs, fluentD stops reading until the exception is resolved.
Currently we do not provide a way via environment variables to change the buffer configuration. You might need to tweak the default settings and create a custom image. If you are up for adding support to change those settings via a PR, we would welcome the PR.
https://docs.fluentd.org/v0.14/articles/buffer-section#type
from fluentd-kubernetes-sumologic.
Hi Jay:
Were you able to leverage the plugin settings to prevent the buffer overflow?
from fluentd-kubernetes-sumologic.
@jdoconnor wanted to revisit this issue. I have recently learned of a bug in the README. The README used to indicate that the FLUSH_INTERVAL was 5 seconds. This is not true, it is actually 30 seconds.
This means that we will not flush the buffer until 30 seconds. This can cause a large build up of logs, increasing memory usage and causing those errors where the Buffer is overflowing. You might want to try reducing that flush interval by setting the FLUSH_INTERVAL in the yaml file to 5s.
from fluentd-kubernetes-sumologic.
closing due to lack of response and fixes that have been introduced...
from fluentd-kubernetes-sumologic.
Facing same issue. FLUSH_INTERVAL=5s
{"log":"2019-02-14 09:24:23 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/buffer.rb:269:in write'\" tag=\"xxxxxxx.log\"\n","stream":"stdout","time":"2019-02-14T09:24:23.595906259Z"}
from fluentd-kubernetes-sumologic.
Related Issues (20)
- Release as an independent GEM HOT 3
- Sourcecategories in Sumo appending numbers HOT 12
- Feature request - Migrating .conf files into configMap
- Kubernetes_meta and Kubernetes_meta_reduce break when condition is met.
- Add ability to control which metadata you want to include/exclude when reducing the metadata
- Memory usage growing constantly
- Duplicate Logs because of Redundant SourceCategories HOT 1
- emit transaction failed: error_class=NoMethodError error="undefined method `+' HOT 4
- fluentd daemonset unable to read log files under OCI OKE
- Sumologic sourceCategory not respecting sumologic.com/sourceCategory annotation HOT 2
- RBAC fluentd daemonset unable to read log files under OCI OKE
- ls: cannot access '/fluentd/etc/fluent.file.conf': No such file or directory HOT 4
- Allow ability to set annotations in the daemonSet in Helm Chart HOT 3
- Windows support HOT 1
- Worker 0 continuously restarts HOT 1
- Question: Can I reduce Kubernetes metadata but leave labels? HOT 1
- support for CRI-O logs HOT 1
- Docker build failing for v2.4.2 HOT 1
- POD metadata is not extracted to sourceName
- Memory consumption grows to max and never goes down HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluentd-kubernetes-sumologic.