Support for "Check from Only One Node" Monitors about signalfx-agent

[mcmiv413]

Wondering if there is any traction on this, we tried to implement a Prometheus exporter check and ran into this same issue, had to farm off the exporter to check to an unrelated stand alone server.

[keitwb]

@daviddyball I actually implemented this feature last year in a hackathon we had and we have gotten several requests similar to yours in the last few months. Unfortunately my implementation is sitting in a feature branch (https://github.com/signalfx/signalfx-agent/tree/clustering) and needs a bit of fixing up and testing/debugging to make production-ready. It is basically just an extension and abstraction of what the kubernetes-cluster monitor does to ensure that it runs on a single agent instance within a K8s cluster.

The way it is implemented (in the branch) is that the agent depends on some external service to do the election process. In K8s, it uses the K8s API server and ConfigMaps to obtain "leases". I made a simple Memcached implementation in my branch. Would you be able to make use of one of those two implementations or would some other service be more convenient for you? It is pretty simple to extend it to use anything that supports basic atomic operations and can be accessed over the network by the agent.

We are looking into "productionalizing" this soon and will post any updates here.

[daviddyball]

@keitwb Cool to see it's been thought of already. Most of our services that require leadership election just use K8s ConfigMaps for doing that, so I can't see why it wouldn't work here also. Glad it's on the radar... having 500 connections to our RDS instance just for monitoring wasn't feasible 😂

[keitwb]

@mcmiv413 unfortunately not -- it was deprioritized given how complex it is to make work (there are a lot of edge cases to work out). I had something working but it was nasty and I wasn't very confident putting it out. I now believe that this problem is really better solved at a level above the agent, as that allows the agent to remain relatively simple.

[daviddyball]

@keitwb Do you have a recommendation for how it should be done?

[keitwb]

@daviddyball By running a separate standalone agent instance with the appropriate hardcoded config. If you are in K8s you can use a Deployment with size of 1 with its own ConfigMap. It's a bit more overhead to setup but once you get it running it should be simple to maintain.

[mcmiv413]

I'd argue that doing what is suggested is a poor game plan, any time where you rely on something to only be ever running as a single instance even in kubernetes is just bad practice in my opinion. I do hope that this gets revisited and implemented in an appropriate manner.