heka_tcp output plugin terminates if server is not listening anymore about lua_sandbox_extensions

[trink]

I had one hang (possibly in socket.write(), process_message never returned) under very similar conditions. I was working with Ops and we were able to reproduce it without SSL but got no further. I am assuming this failure is related and expect to find something that stepped on memory (as there is nothing that would exhaust the instruction limit in that plugin).

[bbinet]

In fact it actually enters an endless loop because client:sendreturns error "Broken pipe", so the plugin tries a new send_message(msg, i):
https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/socket/sandboxes/heka/output/heka_tcp.lua#L70

Patch to come.

[bbinet]

This is addressed in pull request #66

[trink]

Good catch, I just looked at our prod config and the instruction_limit was set to zero so that explains our hang as it was just infinitely looping.