Metrics from an outside service? about kube-metrics-adapter

[pinkavaj]

The JSON metric collector is already present, but it is only used to collect metric from pods wich are managed by HPA. Metric from those pods also can be scaped by Prometheus and then used by prometheus metrics plugin, so this is a bit similar situation (but also a bit different).

I agree using Prometheus is usualy better idea, but this might help simplify some thinks.

I have already tryed to hack (I did not event tested it, the code need definitely a lot of cleanup) something which migh do what is proposed here (https://github.com/pinkavaj/kube-metrics-adapter/tree/external-json-collector).

I would guess the syntac might look somethink like:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  annotations:
    metric-config.external.my-metric.json-query/host: service.my-namespace.svc
    metric-config.external.my-metric.json-query/json-key: "$.mymetric.value"
    metric-config.external.my-metric.json-query/path: /metrics
    metric-config.external.my-metric.json-query/port: "8080"
    metric-config.external.my-metric.json-query/scheme: "http"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: custom-metrics-consumer
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metricName: my-metric
      targetAverageValue: 10

[arjunrn]

There is now an external http json collector. Implemented in #122

[szuecs]

We had also issues with Prometheus, but recent versions and running with a vpa works great for us.
If you like to enhance this tool we are happy to review your pull requests. Tests are mandatory and documentation, too.

[mikkeloscar]

@mzupan just to be clear. What you are asking for is something like the Prometheus external metric, but instead of Prometheus format want to hit a service which returns metrics in JSON format?

Is this correctly understood or do you mean something else? Maybe an example of the sort of metrics you would like to scale on would be helpful to understand the problem you are trying to solve.

[mzupan]

@mikkeloscar it's my understanding that this only reads metrics from the same deployment it is autoscaling?

We have a deployment called sidekiq which is a bunch of workers. I want to read metrics from a collector that pulls in queue count numbers for a deployment/service called sidekiq-exporter that I can add a json endpoint to. I want to hit the sidekiq-exporter service to autoscale the sidekiq deployment

[mikkeloscar]

@mzupan thanks, I understand what you want to do now. However I'm not completely convinced it makes sense to introduce such custom collectors when we already have Prometheus which is a general solution solving this problem in a "standard" way. Seems more like a hack to work around a Prometheus which is not scaled reasonable.

I do realize that not everyone is using Prometheus so having something json based could be valuable. Can you share your ideas around how the API would look like for such a feature? I.e. how would users configure the HPA. With this in mind it's easier to discuss the feature. Thanks!

[mzupan]

@pinkavaj yes that's just what I was thinking of doing is adding a host or service attribute to the annotation

[tomaspinho]

@pinkavaj's work would also be very beneficial for us, what's missing for this to be merged?
EDIT: Available to help out in any way.

[pinkavaj]

It is proof-of concept now. It is tested to the point where metric from JSON endpoint are collected and processed, but I did not get to the point to actualy use it with any HPA. It needs a bit more testing, add proper documentation and probably write some tests.

We are now testing another metric API server, but I would gladly finish this (sinc we might revert back to this solution). It would really help if You can build and run my current implementation in the the branch mentioned above and try if it actualy works.

[mikkeloscar]

Sorry I lost track of this. The proposal seems reasonable to me. I would happily review a PR with this feature.