As I understood, this go forwarder became obsolete/unsupported/abandoned?

It'd be wonderful to have out_secure_forward compatibility in fluentd-forwarder.

Example use case- with Docker 1.8 there is basic Fluent support, but sending the logs off the box requires a forwarder agent, preferably a secure one.

Would be nice to be able to specify http(s) forward destinations, e.g.

• to https://some.fluentd.instance:443

, as well as multiple out destinations:

• to https://some.fluentd.instance:443, fluent://other.instance:24443

Is there an easy way to use this lightweight forwarder, yet be able to inject some additional host information into the log data that gets forwarded on to other collectors?

For example if I ran the forwarder in the following manner:

$ $GOPATH/bin/fluentd_forwarder -to fluent://some-remote-node.local:24224 -data-key hostinfo -data-value "name=host1234;type=prod;asg=autoscalegroup132"

Then when a log line such as the following came in:

{"ts":"2015-05-22T18:35:00Z","appId":"myApp","msg":"logging data"}

the additional data key and value would get injected:

{"ts":"2015-05-22T18:35:00Z","appId":"myApp","msg":"logging data","hostinfo":"name=host1234;type=prod;asg=autoscalegroup132"}

This would allow me to do minimal processing on a machine that is trying to run my application, yet would allow me to inject host specific info into the log lines for processing on a different cluster of machines.

I'm assuming there's no support for record_reformer type configs for this forwarder.

Thanks

As mentioned by a number of people, the name of the bootstrapping command "build" is too generic because the workspace directory specified by GOPATH environment variable is supposed to be shared amongst the projects as seen in the official document that even instructs the user to put it in PATH.

At least it needs to be changed to something like "build_fluentd_forwarder" or possibly amend the documentation for users to build and run it by "go run".

Following the instruction, I ran into the following issue.

$ ./bin/build fluentd_forwarder
Username for 'https://github.com': kiyoto
Password for 'https://[email protected]':
# github.com/jehiah/go-strftime
../../jehiah/go-strftime/strftime.go:46: undefined: strings.IndexByte
# github.com/treasure-data/fluentd-forwarder
./file_journal.go:170: undefined: atomic.SwapPointer
build: exit status 2

My go version is as follows on Ubuntu 12.04

$ go version
go version go1.1.1 linux/amd64

Hi,

I tested fluentd_forwarder.exe on Windows 8.1 (64-bit) with the following network:

Linux: 192.168.255.1
Windows: 192.168.255.168

On Windows:

PS C:\fluentd-test1> C:\td\bin\fluentd_forwarder.exe -buffer-path c:\fluentd-test1\buffer\td -listen-on 0.0.0.0:24224 -to 192.168.255.1:24224
[fluentd-forwarder] 20:22:09 Path c:\fluentd-test1\buffer\td is designated to Worker output
[fluentd-forwarder] 20:22:09 Spawning acceptor
[fluentd-forwarder] 20:22:09 Spawning daemon
[fluentd-forwarder] 20:22:09 Spawning spooler
[fluentd-forwarder] 20:22:09 Spawning emitter
[fluentd-forwarder] 20:22:09 Acceptor started
[fluentd-forwarder] 20:22:09 Daemon started
[fluentd-forwarder] 20:22:09 Spooler started
[fluentd-forwarder] 20:22:09 Emitter started

I sent an event from Linux:

$ echo '{"x": 1}' | fluent-cat -h 192.168.255.168  app.test1

I got the following messages:

[fluentd-forwarder] 20:22:14 Connected from 192.168.255.1:64129
[fluentd-forwarder] 20:22:14 Got conn from acceptChan
[fluentd-forwarder] 20:22:14 Started handling connection from 192.168.255.1:64129
[fluentd-forwarder] 20:22:14 Client 192.168.255.1:64129 closed the connection
[fluentd-forwarder] 20:22:14 Ended handling connection from 192.168.255.1:64129
[fluentd-forwarder] 20:22:14 Flushing...
[fluentd-forwarder] 20:22:14 Flushing chunk c:\fluentd-test1\buffer\td.output.q50121eebe213b389.log
[fluentd-forwarder] 20:22:14 Connecting to 192.168.255.1:24224...
[fluentd-forwarder] 20:22:14 Forwarded 22 bytes in 0.000000 seconds (0 bytes left)
[fluentd-forwarder] 20:22:14 Error during reading from the journal: remove c:\fluentd-test1\buffer\td.output.q50121eebe213b389.log: The process cannot access the file because it is being used by another process.
[fluentd-forwarder] 20:22:19 Flushing...
[fluentd-forwarder] 20:22:19 Flushing chunk c:\fluentd-test1\buffer\td.output.q50121eebe213b389.log
[fluentd-forwarder] 20:22:19 Forwarded 22 bytes in 0.000000 seconds (0 bytes left)
[fluentd-forwarder] 20:22:19 Error during reading from the journal: remove c:\fluentd-test1\buffer\td.output.q50121eebe213b389.log: The process cannot access the file because it is being used by another process.
[fluentd-forwarder] 20:22:24 Flushing...

It seems fluentd-forwarder failed to remove the buffer file and retry flushing again and again.

Do you have any idea?

As the below error indicates, this is not working with go 1.7.1.

github.com/fluent/fluentd-forwarder/entrypoints/fluentd_forwarder

link: warning: option -X main.progVersion 0.0.1 may not work in future releases; use -X main.progVersion=0.0.1

hi,

1st question is the purpose of this project, is it mean to forward tcp package from one server to another?

2nd, how should i send the msg to the listen port?
as send: echo "a:=1" | nc 127.0.0.1
the forwarder got ,

[fluentd-forwarder] 2016/12/28 18:30:43.145474 Started handling connection from 127.0.0.1:58987
test1, c.input::: input
[fluentd-forwarder] 2016/12/28 18:30:43.145679 [pos 1]: only encoded map or array can be decoded into a slice (0)
[fluentd-forwarder] 2016/12/28 18:30:43.145697 Ended handling connection from 127.0.0.1:58987

We were trying to figure out why we weren't getting messages sent through the fluent go proxy. The issue was intermittent and while the forwarder would say that it was sending those messages onto fluentd...they never arrived.

In the end, we discovered that the go proxy will occasionally go into a CLOSED_WAIT state when no messages are coming through for a while. The connection is 'eventually' closed, but everything that comes through this system while in a closed_wait state is lost.

The readme file doesn't mention the dependency on github.com/treasure-data/td-client-go, which needs to be installed, as can be seen in the .travis.yml file.

We've noticed that if a client suddenly terminates without cleanly closing the connection to fluentd-forwarder, then the forwarder starts to emit loads of messages like:

... temporary failure: read tcp from 127.0.0.1... An existing connection was forcibly closed by the remote host.

This happens on Windows.

Maybe golang issue #6163 and this fix could be related?

It would be nice if fluentd-forwarder was more resilient to abrupt client terminations.
Is there anything we can do on our side?

Thanks!

Some users want this plugin to tail local file and forward events to fluentd destination.

https://github.com/fujiwara/fluent-agent-hydra has a good reference implementation.