Proposal/Discussion: SignalFx Kubernetes Controller about signalfx-agent

[daviddyball]

This SignalFxMonitor CRD you are proposing would be manually updated by a cluster admin and would basically be the equivalent of a single entry under monitors: in the agent.yaml config?

Yep that was my thinking. It would mean our dev teams could keep a SignalFxMonitor object alongside their Deployment definitions and automatically get monitoring. Granted the same is possible with annotations, it is just for neatness purposes that I wanted to do it as a separate CRD.

Also note that you don't have to do everything in the annotations on the resources, you can put some of "boilerplate" config in the agent main ConfigMap and use discovery rules and then put more specific things in annotations and the agent will merge them all together.

This goes a long way to solving my problems. Do you have an example to hand? Do I just supply a monitors entry without a discoverRule to do this? All of our services are currently inheriting a single YAML anchor in the main agent.yaml to make life easy, so if I can re-use that via annotations then it's problem solved.

Also keep in mind that you can map multiple ConfigMaps into the agent container on K8s (by adding them to the DaemonSet) and reference them in the main agent.yaml with the {"#from": "file:/etc/signalfx/other-config.yaml"} remote config syntax. The agent will dynamically reload on changes to those files and would be roughly equivalent to what you are proposing with CRDs, if I am understanding you correctly.

This sounds like it would make loading multiple SignalFxMonitor CRDs easier. Do these auto-reload correctly when the contents of the #from file is changed?

[keitwb]

This SignalFxMonitor CRD you are proposing would be manually updated by a cluster admin and would basically be the equivalent of a single entry under monitors: in the agent.yaml config?

Also note that you don't have to do everything in the annotations on the resources, you can put some of "boilerplate" config in the agent main ConfigMap and use discovery rules and then put more specific things in annotations and the agent will merge them all together.

Also keep in mind that you can map multiple ConfigMaps into the agent container on K8s (by adding them to the DaemonSet) and reference them in the main agent.yaml with the {"#from": "file:/etc/signalfx/other-config.yaml"} remote config syntax. The agent will dynamically reload on changes to those files and would be roughly equivalent to what you are proposing with CRDs, if I am understanding you correctly.

[keitwb]

Do I just supply a monitors entry without a discoverRule to do this?

No, you must supply a discoveryRule that matches the endpoint specified by the annotation config on the K8s resource. I added an integration test that shows this and proves that it works: https://github.com/signalfx/signalfx-agent/pull/1088/files. Let me know if that is not clear.

Do these auto-reload correctly when the contents of the #from file is changed?

Yes, anything specified with the #from refs gets watched and changes will cause a reload automatically in the agent. You can even specify a glob of multiple files with {"#from": "file:/etc/signalfx/monitors/*.yaml"} for example and it will watch all of them for changes. ConfigMaps support multiple files specified in a single instance as well.

Let me know if there are any pain points that these two things cannot solve. I would rather not add a CRD if it is basically just a glorified alternative to ConfigMaps that is nearly functionally equivalent.

[daviddyball]

Interesting 🤔. I do understand your point about keeping it simple... I guess I just prefer being able to kubectl get signalfxmonitor 😛 from an operator perspective.

At the moment we have discoveryRule entries like so:

- discoveryRule: container_image =~ "app-1" && container_port == "9999"
- discoveryRule: container_image =~ "app-2" && container_port == "9999"
- discoveryRule: container_image =~ "app-3" && container_port == "9999"
- discoveryRule: container_image =~ "app-4" && container_port == "9999"

Which means pre-defining our apps in the agent.yaml anyway if we want to use the merge functionality.

I guess the best way to configure it would be to set a generic discoveryRule in agent.yaml that is matches all relevant Pods.

Would something like this work:

Pod

apiVersion: v1
kind: Pod
metadata:
  labels:
    agent.signalfx.com/scrape: "true"

agent.yaml

monitors:
- type: prometheus-exporter
  discoveryRule: Get(container_labels, "agent.signalfx.com/scrape") == "true" && container_port == "9999"
  .... more config here ......

[keitwb]

Yes, you can absolutely do a "generic" discoveryRule that applies to a bunch of endpoints that should share common configuration. Each one will get its own monitor instance and the merging will be done separately for each endpoint discovered.

I get the desire for a custom resource that makes things explicit, we'll take it into consideration.

Going to close for now but if you have any more questions/concerns about this reopen and let us know!