Comments (7)
Could it be that this needs only to be done on node failure so the perforamnce issue does not need to be too high? i.e., only when you start trying to see if a node has recovered, which I believe is done at a decreasing frequency, e.g. 30secs, 1 min, 2 min etc?
from fluentd.
Just wondered where this fix fits into the fluentd roadmap, as its the only thing that really stopping me switching from Flume to fluentd. :-)
from fluentd.
@tagomoris Do you think is it acceptable to have new TCP connections, everytime chunks are sent? On EC2 environment, the same hostname could be resolved in to different IP. So we want to resolve DNS every time.
from fluentd.
To reconnect for every chunks is not so bad for good network conditions, but a few times of SYN packet resending (in Linux, it needs 3sec timeout) makes great problems with reconnect per chunks and cofigurations such as 'flush_interval=1'.
I vote +1 to reconnection revery 30sec/1min/2min (or configurable & default 1min?).
from fluentd.
I vote +1 to reconnection revery 30sec/1min/2min (or configurable & default 1min?).
Sounds reasonable. I will take a look at current implementation.
from fluentd.
I implemented dns cache. out_forward newly supports "expire_dns_cache" option.
default behavior: out_forward caches resolved host name and invalidates it only when the node becomes not available.
expire_dns_cache > 0: out_forward caches resolved host name and invalidates it when the node becomes not available or the specified expired time passed.
expire_dns_cache = 0: out_forward doesn't cache resolved host name.
from fluentd.
connection keepalive is another problem.
from fluentd.
Related Issues (20)
- Out forward stuck establishing connection HOT 8
- Windows: td-agent 4.5.2 Too many open files HOT 1
- Buffer files remain after restart of fluentd HOT 5
- stdout P followed my infinite loop of "\\\\\\\\\" in cloudwatch log stream for fluentd
- Compression Support in out_http plugin
- fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) HOT 1
- DiskPressure due to 80 GB /home/fluentd/core. files
- Set log_path in system config HOT 2
- Duplicate logs getting created on Opensearch through fluentd HOT 4
- Startup error using docker startup to specify profile location
- Out_file short flush interval with timekey 1h in buffer HOT 5
- in_tail error after upgrading from 3.8.1 to td-agent-4.5.2 HOT 3
- Fluentd logs HOT 1
- Support AWS SigV4 in the http output plugin HOT 3
- Buffer: v1.16.4: Emit error by IOError HOT 8
- Match directive not working HOT 1
- What will be the impact after removal of OpenSSL c_rehash script from td-agent
- in_exec: Can't handle non-ASCII characters output HOT 1
- Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. HOT 4
- [YAML] 'log_level' is deprecated parameter name. use '@log_level' instead // parameter '$log_level' is not used HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluentd.