buffer metrics when tsdb can't be reached. about metrictank

[ctdk]

For clarification, is this only an issue with kairosdb, or is carbon subject to it too?

[Dieterbe]

assuming this is about on-disk buffering.
would it be an acceptable solution to just not ack the messages on the source side when the dest is down? the source (rabbitmq) is assumed to be a highly reliable persistent store of data. so why not just leverage that? why does raintank-metric have to do its own, independent buffering?

[Dieterbe]

alternatively, could we just replace it with heka? it has in-mem and on-disk buffering built-in, but not sure if it's a great fit per se.

alternatively we could also give a good look an nsq.io which is a distributed, at-least-once-delivery, high performant queuing system with good in-mem and on-disk buffering. I heard it has some nice operational controls like draining out nodes, it might have some useful throtthling too. It sucks adding more dependencies but we knew from the beginning that rabbitmq is not a safe way to scale out across multiple machines.

[ctdk]

Re: on-disk buffering, we were looking at doing that since we had already removed them from rabbitmq at that point. I had made an in-mem buffer first, but that's still vulnerable to losing data when getting shut down, and I wasn't sure if trying to put it back into rabbitmq was the right thing to do either.

Should we just put it back into rabbitmq if the destination is unavailable?

[Dieterbe]

Aren't we able to just not acknowledge the messages to rabbit until we safely delivered them? It shouldn't even be a matter of first removing them from rabbit and re-adding them if we can just not ack them.
the only thing i'm not sure about is we have 2 destinations AFAIK: ES and kairos. what to do when 1 of them is down..

[ctdk]

We don't send the messages right away, though; we queue them up as they come in and then send them in batches.

[Dieterbe]

ok, do you know if that prevents us from delaying the ack? it would be nice if we could just ack the msg to rabbit when we sucessfully processed/sent the message, even if that was a minute after we got it from rabbit

[woodsaj]

there is no point in delaying acknowledge of messages when using exclusive non-durable queues. If the raintank-metric process dies, the queue will be deleted (non-durable) allowing the exchange to correctly distribute new metrics between available raintank-workers.

i would vote for closing this in favor of issue #11