out_forward: resolve DNS more intelligently about fluentd

[rummble]

Could it be that this needs only to be done on node failure so the perforamnce issue does not need to be too high? i.e., only when you start trying to see if a node has recovered, which I believe is done at a decreasing frequency, e.g. 30secs, 1 min, 2 min etc?

[rummble]

Just wondered where this fix fits into the fluentd roadmap, as its the only thing that really stopping me switching from Flume to fluentd. :-)

[kzk]

@tagomoris Do you think is it acceptable to have new TCP connections, everytime chunks are sent? On EC2 environment, the same hostname could be resolved in to different IP. So we want to resolve DNS every time.

[tagomoris]

To reconnect for every chunks is not so bad for good network conditions, but a few times of SYN packet resending (in Linux, it needs 3sec timeout) makes great problems with reconnect per chunks and cofigurations such as 'flush_interval=1'.

I vote +1 to reconnection revery 30sec/1min/2min (or configurable & default 1min?).

[kzk]

I vote +1 to reconnection revery 30sec/1min/2min (or configurable & default 1min?).

Sounds reasonable. I will take a look at current implementation.

[frsyuki]

I implemented dns cache. out_forward newly supports "expire_dns_cache" option.

default behavior: out_forward caches resolved host name and invalidates it only when the node becomes not available.
expire_dns_cache > 0: out_forward caches resolved host name and invalidates it when the node becomes not available or the specified expired time passed.
expire_dns_cache = 0: out_forward doesn't cache resolved host name.

[frsyuki]

connection keepalive is another problem.