Hi @DirectXMan12

First off thanks for taking the time to write this library, it's exactly what kubernetes needs.

I'm not sure the best way of contacting you or someone who can help, I hope it's alright to raise an issue here. This might not be an issue and could just be me not setting up the adaptor properly. I'm pretty new to the rbac and cert setup stuff so had to digest allot of docs to understand these.

What I am seeing in the adapter logs is this:

I0417 06:59:51.313272       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (311.405µs) 404 [[kubectl/v1.6.4 (linux/amd64) kubernetes/d6f4332] 10.211.156.1:58882]
I0417 06:59:51.330025       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (273.369µs) 404 [[kubectl/v1.6.4 (linux/amd64) kubernetes/d6f4332] 10.211.156.1:58882]
I0417 06:59:51.341397       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (303.466µs) 404 [[kubectl/v1.6.4 (linux/amd64) kubernetes/d6f4332] 10.211.156.1:58882]
I0417 06:59:51.519083       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (298.069µs) 404 [[kubectl/v1.6.4 (linux/amd64) kubernetes/d6f4332] 10.211.156.1:58882]
I0417 06:59:51.594021       1 authorization.go:58] Forbidden: "/", Reason: "No policy matched.\nUnknown user \"system:anonymous\""
I0417 06:59:51.594180       1 wrap.go:42] GET /: (3.108636ms) 403 [[Go-http-client/2.0] 10.211.0.5:51826]
E0417 06:59:53.043166       1 provider.go:298] unable to update list of all available metrics: Get http://prometheus-prometheus-server:9090/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer_name%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod_name%21%3D%22%22%7D&match%5B%5D=%7Bnamespace%21%3D%22%22%2C__name__%21~%22%5Econtainer_.%2A%22%7D&start=1523948303.042: dial tcp 10.211.254.128:9090: i/o timeout

Now we do have allot of metrics and if I make that call locally it does take a while for all the data to come back. So I think that could be the i/o timeout, to check this is there anyway I can set it up to just get the metrics I'm interested in?

Also I'm gathering the 404 i'm seeing are due to the fact I cant generate the list of all available metrics?

The setup I have is this:

• prometheus is setup via the helm package

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-kubernetes-adapter 
  namespace: metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-kubernetes-adapter
  template:
    metadata:
      labels:
        app: prometheus-kubernetes-adapter
    spec:
      serviceAccountName: prometheus-kubernetes-adapter
      containers:
      - image: directxman12/k8s-prometheus-adapter:v0.1.0-centos
        name: cm-adapter
        args:
        - --secure-port=6443
        - --logtostderr=true
        # use your serving certs
        - --tls-cert-file=/var/run/serving-certs/tls.crt
        - --tls-private-key-file=/var/run/serving-certs/tls.key
        # Prometheus is running in the same pod, so you can say your Prometheus
        # is at `localhost`
        - --prometheus-url=http://prometheus-prometheus-server:9090
        # relist available Prometheus metrics every 1m
        - --metrics-relist-interval=1m
        # calculate rates for cumulative metrics over 30s periods.  This should be *at least*
        # as much as your collection interval for Prometheus.
        - --rate-interval=30s
        # re-discover new available resource names every 10m.  You probably
        # won't need to have this be particularly often, but if you add
        # additional addons to your cluster, the adapter will discover there
        # existance at this interval
        - --discovery-interval=10m
        - --v=4
        ports:
        # this port exposes the custom metrics API
        - containerPort: 6443
          name: https
          protocol: TCP
        volumeMounts:
        - mountPath: /var/run/serving-certs
          name: serving-certs
          readOnly: true
      volumes:
      - name: serving-certs
        secret:
          secretName: serving-cm-adapter

• try'ed running bash kubectl create clusterrole resource-lister --verb=list --resource="*" but ended up getting bash the server doesn't have a resource type "*" which is a bit odd. So just for now I have used a known clusterrole with enough privileges and set the rolebinding to that.

kubectl describe apiservice v1beta1.custom.metrics.k8s.io 
Name:         v1beta1.custom.metrics.k8s.io
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.custom.metrics.k8s.io","namespace":""},...
API Version:  apiregistration.k8s.io/v1beta1
Kind:         APIService
Metadata:
  Creation Timestamp:  2018-04-14T08:58:57Z
  Resource Version:    80344405
  Self Link:           /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.custom.metrics.k8s.io
  UID:                 10a8278f-3fc2-11e8-a5b7-42010af00031
Spec:
  Ca Bundle:               <shh it's a secret>
  Group:                   custom.metrics.k8s.io
  Group Priority Minimum:  1000
  Service:
    Name:            prometheus-kubernetes-adapter
    Namespace:       metrics
  Version:           v1beta1
  Version Priority:  10
Status:
  Conditions:
    Last Transition Time:  2018-04-16T19:47:23Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

Any help would be greatly appreciated, I really want to see this adaptor in action 👍

Thanks,

Mark

To accommodate people who prefer to run things in containers instead of installing them locally, we should figure out a way to run Glide from a container if it's not found on the local system.

Follow up from #10

cc @luxas

Why is not in kubernetes-incubator? 😅 ... I think it's good work. Maybe under something like http://github.com/kubernetes-incubator/custom-metrics-apiserver-prometheus-adapter. Admittedly my knowledge of kubernetes-incubator is very basic.

I'm guessing it's because maybe kubernetes don't want to manage implementations? Other than that, I can't think of anything else.

cc: @luxas @zacblazic

While attempting the example deployment I ran into a TLS issue while validating the server:

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" -v 10
I0619 06:06:51.582576 27444 loader.go:357] Config loaded from file /home/ubuntu/.kube/config
I0619 06:06:51.583860 27444 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.7.5 (linux/amd64) kubernetes/17d7182" http://localhost:8080/apis/custom.metrics.k8s.io/v1beta1/
I0619 06:06:51.585319 27444 round_trippers.go:405] GET http://localhost:8080/apis/custom.metrics.k8s.io/v1beta1/ 500 Internal Server Error in 1 milliseconds
I0619 06:06:51.585420 27444 round_trippers.go:411] Response Headers:
I0619 06:06:51.585491 27444 round_trippers.go:414] Content-Type: text/plain; charset=utf-8
I0619 06:06:51.585549 27444 round_trippers.go:414] X-Content-Type-Options: nosniff
I0619 06:06:51.585605 27444 round_trippers.go:414] Date: Tue, 19 Jun 2018 06:06:51 GMT
I0619 06:06:51.585660 27444 round_trippers.go:414] Content-Length: 74
I0619 06:06:51.585746 27444 request.go:991] Response Body: tls: found a certificate rather than a key in the PEM for the private key
I0619 06:06:51.585979 27444 helpers.go:207] server response object: [{
"metadata": {},
"status": "Failure",
"message": "an error on the server ("tls: found a certificate rather than a key in the PEM for the private key") has prevented the request from succeeding",
"reason": "InternalError",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "tls: found a certificate rather than a key in the PEM for the private key"
}
]
},
"code": 500
}]
F0619 06:06:51.588980 27444 helpers.go:120] Error from server (InternalError): an error on the server ("tls: found a certificate rather than a key in the PEM for the private key") has prevented the request from succeeding

From the error it looks like I messed up setting up the secret, but I logged into the container and see that the secret seems to have loaded fine:

bash-4.2$ pwd
/var/run/serving-cert
bash-4.2$ cat tls.crt
-----BEGIN CERTIFICATE-----
MIIC7TCCAdWgAwIBAgIJAPcm5j2xunN7MA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV
BAMMAmNhMB4XDTE4MDYxOTAxNTIyN1oXDTE5MDYxOTAxNTIyN1owDTELMAkGA1UE
...
BYR30+HWYI041L43AjY0VpYmlJTUgzhWeY0elxUGDA6d
-----END CERTIFICATE-----

bash-4.2$ cat tls.key
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDcLrixgQnofhLi
...
KwNqeYrdOFnegKtu6ARXXY1x
-----END PRIVATE KEY-----
bash-4.2$

Any pointers would really help ..

Thanks!

I noticed in my own cluster and also luxcas' workshop

kubectl ​get ​--raw ​/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests"},"items":[{"describedObject":{"kind":"Service","name":"sample-metrics-app","apiVersion":"/__internal"},"metricName":"http_requests","timestamp":"2018-01-20T01:54:13Z","value":"866m"}]}

My requests total is only 866+, not very sure why the value ends with a m? ("value":"866m")

Hi,
this image got outdated:
https://hub.docker.com/r/directxman12/k8s-prometheus-adapter/

        - --metrics-relist-interval=30s

This is shorter than a typical scrape interval for prometheus, causing metrics to become available/unavailable depending on the window.

This may be a reasonable dev setting (depending on your setup) but since sample files have a strong power-of-doc, I suggest this option and the others be set to their default values.

master branch
make build

vendor/k8s.io/apiserver/pkg/server/routes/openapi.go:35: cannot use c.RegisteredWebServices() (type []"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".WebService) as type []"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".WebService in argument to handler.BuildAndRegisterOpenAPIService

github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/custom_metrics

vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/custom_metrics/reststorage.go:49: cannot use custom_metrics.MetricValue literal (type *custom_metrics.MetricValue) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object in return argument:
*custom_metrics.MetricValue does not implement "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object (wrong type for DeepCopyObject method)
have DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/vendor/k8s.io/apimachinery/pkg/runtime".Object
want DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/custom_metrics/reststorage.go:55: cannot use custom_metrics.MetricValueList literal (type *custom_metrics.MetricValueList) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object in return argument:
*custom_metrics.MetricValueList does not implement "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object (wrong type for DeepCopyObject method)
have DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/vendor/k8s.io/apimachinery/pkg/runtime".Object
want DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/custom_metrics/reststorage.go:98: cannot use *custom_metrics.MetricValueList as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object in return argument:
*custom_metrics.MetricValueList does not implement "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object (wrong type for DeepCopyObject method)
have DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/vendor/k8s.io/apimachinery/pkg/runtime".Object
want DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/custom_metrics/reststorage.go:100: cannot use *custom_metrics.MetricValueList as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object in return argument:
*custom_metrics.MetricValueList does not implement "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object (wrong type for DeepCopyObject method)
have DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/vendor/k8s.io/apimachinery/pkg/runtime".Object
want DeepCopyObject() "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/runtime".Object

github.com/directxman12/k8s-prometheus-adapter/pkg/custom-provider

pkg/custom-provider/provider.go:94: cannot use "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".Time literal (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".Time) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".Time in field value
pkg/custom-provider/provider.go:95: cannot use *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/api/resource".NewMilliQuantity(int64(value * 1000), "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/api/resource".DecimalSI) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/api/resource".Quantity) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/vendor/k8s.io/apimachinery/pkg/api/resource".Quantity in field value

github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer

vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:76: cannot use ws (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".WebService) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".WebService in argument to versionDiscoveryHandler.AddToWebService
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:248: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:251: cannot use rootScopedHandler (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(rootScopedPath).To
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:273: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:274: cannot use namespacedHandler (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(namespacedPath).To
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:297: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:298: cannot use namespaceSpecificHandler (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(namespaceSpecificPath).To
make: *** [build] 错误 2

Hi,
I'm trying to setup autoscaling with "object" metric, but I see the message "object metrics are not yet supported" which is produced here:
https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/metrics/legacy_metrics_client.go#L176

I saw "object" metrics in many examples and I've seen that they are supported in "Rest metrics client" (https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/metrics/rest_metrics_client.go)

Why do I fall in "legacy" code?

making admins manually url-decode queries to debug them is a pain. we should just print the non-url-encoded form instead.

Background: We have a Springboot app running on K8s, which exposes metrics to be scraped by Prometheus and visualised in Grafana. We are currently retro-fitting HPA based on custom metrics. (By the way, thank you for the invaluable help your walkthrough has provided).

Our app already exposes counter-type metrics which could be used the same way as (example) http_requests_total, however the suffix is different (_count in our case). I am not a dev as such, and to my understanding, the way these metrics have been exposed, it would be rather difficult for us to change the suffix to _total (or anything other than _count).

Long story short, it would be beneficial if we could treat another suffix the same way _total is being treated. Is there a way to do that without changing the adapter code?

✗ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests | jq
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Service",
        "name": "sample-metrics-app",
        "apiVersion": "/__internal"
      },
      "metricName": "http_requests",
      "timestamp": "2017-11-29T21:31:13Z",
      "value": "539m"
    }
  ]
}

Note that http_requests has value of "539m". It should be a number and not memory. In my case I was expecting 120 (that was my transaction per second i had set via my load test tool). Due to this issue hpa does not scale the replicaset appropriately. Prometheus metrics was verified to show right value.

I1129 21:27:07.806751       1 api.go:74] GET http://sample-metrics-prom.default.svc:9090/api/v1/query?query=sum%28rate%28http_requests_total%7Bservice%3D%22sample-metrics-app%22%2Cnamespace%3D%22default%22%7D%5B
1m%5D%29%29+by+%28service%29&time=1511990827.805 200 OK
I1129 21:27:07.806847       1 api.go:93] Response Body: {"status":"success","data":{"resultType":"vector","result":[{"metric":{"service":"sample-metrics-app"},"value":[1511990827.805,"30.54"]}]}}

This is what describing the hpa shows.

➜  kubectl describe hpa sample-metrics-app-hpa
Name:                                             sample-metrics-app-hpa
Namespace:                                        default
Labels:                                           <none>
Annotations:                                      kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sample-metrics-app-hpa","namespace":"default"...
CreationTimestamp:                                Wed, 29 Nov 2017 21:29:58 +0000
Reference:                                        Deployment/sample-metrics-app
Metrics:                                          ( current / target )
  "http_requests" on Service/sample-metrics-app:  125300m / 100
Min replicas:                                     1
Max replicas:                                     10
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound  the HPA was able to succesfully calculate a replica count from Service metric http_requests
  ScalingLimited  True    TooManyReplicas   the desired replica count was more than the maximum replica count
Events:
  Type     Reason                        Age              From                       Message
  ----     ------                        ----             ----                       -------
  Warning  FailedGetObjectMetric         8m (x9 over 2h)  horizontal-pod-autoscaler  unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
  Warning  FailedComputeMetricsReplicas  8m (x9 over 2h)  horizontal-pod-autoscaler  failed to get object metric value: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)

I've been trying to get this going and I'm just missing something, I think there is a good chance its related to RBAC. I've tried to follow Walk Through , Deploy and Helm Chart but can't get any of them to work.

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
Error from server (NotFound): the server could not find the requested resource

I was able to get Luxas Walkthrough to work end to end. To eliminate additional complexities I tried to use madeden Helm chart with self-signed certs. And also edit above to use self-signed in case there were issues there. I've gone over RBAC and tried to add some more open privileges for testing, but have only limitedly dealt with RBAC.

I don't see any errors til the pod service starts:

I0504 17:41:44.059453       1 serve.go:85] Serving securely on 0.0.0.0:6443
I0504 17:41:44.943488       1 round_trippers.go:383] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 17:41:44.943505       1 round_trippers.go:390] Request Headers:
I0504 17:41:44.943510       1 round_trippers.go:393]     Accept: application/json, */*
I0504 17:41:44.943515       1 round_trippers.go:393]     Content-Type: application/json
I0504 17:41:44.943519       1 round_trippers.go:393]     Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXItdG9rZW4tcmQyOG4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmloaWxpc3QtbW9ua2V5LWN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY3ZGQwZjA4LTRmYzItMTFlOC05YTI4LTBhNGFjOGVmYTA0NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOm5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.PjNg-OeUmgIs5Pm7EPY_dBj03HnjOj9P82tqHa3C6hEexrYZll96vgw2pjaCSEQz7JV7O4NE7MJ_YhAMCYrvYYO9KYosRsXzMchqG43viYxP0Hwv0T7rsvbY9DbxA0BSGDPvB8E1htN1guJnTcpZX5hfLlWUHuc5T7Gty22peTuHkJCXIi0tWMkfYDCO6yy-gU-V6-yPjFfPEKu8AtWQ7G_aGJb0WJyQyZRnY38gg_cFk7uah94UiFrROTElYjdho0a7L5EJsAm3CE94BeW3n2W_4nxR9LOdjeqedMEzzVIO5zkcu3dzUD94HafzS7Lcb1O5UVgjgKNCND4QAvEQEg
I0504 17:41:44.943528       1 round_trippers.go:393]     User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format
I0504 17:41:44.947151       1 round_trippers.go:408] Response Status: 201 Created in 3 milliseconds
I0504 17:41:44.947409       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.947460       1 wrap.go:42] GET /: (4.463264ms) 403 [[Go-http-client/2.0] 100.99.92.0:45210]
I0504 17:41:44.953296       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.953334       1 wrap.go:42] GET /: (80.866µs) 403 [[Go-http-client/2.0] 100.99.84.0:55270]
I0504 17:41:44.958829       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.958864       1 wrap.go:42] GET /: (78.637µs) 403 [[Go-http-client/2.0] 100.99.92.0:45210]
I0504 17:41:44.966197       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.966230       1 wrap.go:42] GET /: (56.14µs) 403 [[Go-http-client/2.0] 100.99.84.0:55270]
I0504 17:41:44.971338       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.971365       1 wrap.go:42] GET /: (52.565µs) 403 [[Go-http-client/2.0] 100.99.84.0:55270]
I0504 17:41:45.018497       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.018542       1 wrap.go:42] GET /: (97.306µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:45.021977       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.022004       1 wrap.go:42] GET /: (57.143µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:45.027072       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.027169       1 wrap.go:42] GET /: (127.241µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:45.222450       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.222481       1 wrap.go:42] GET /: (59.318µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:48.260032       1 round_trippers.go:383] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 17:41:48.260049       1 round_trippers.go:390] Request Headers:
I0504 17:41:48.260055       1 round_trippers.go:393]     Accept: application/json, */*
I0504 17:41:48.260059       1 round_trippers.go:393]     Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXItdG9rZW4tcmQyOG4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmloaWxpc3QtbW9ua2V5LWN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY3ZGQwZjA4LTRmYzItMTFlOC05YTI4LTBhNGFjOGVmYTA0NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOm5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.PjNg-OeUmgIs5Pm7EPY_dBj03HnjOj9P82tqHa3C6hEexrYZll96vgw2pjaCSEQz7JV7O4NE7MJ_YhAMCYrvYYO9KYosRsXzMchqG43viYxP0Hwv0T7rsvbY9DbxA0BSGDPvB8E1htN1guJnTcpZX5hfLlWUHuc5T7Gty22peTuHkJCXIi0tWMkfYDCO6yy-gU-V6-yPjFfPEKu8AtWQ7G_aGJb0WJyQyZRnY38gg_cFk7uah94UiFrROTElYjdho0a7L5EJsAm3CE94BeW3n2W_4nxR9LOdjeqedMEzzVIO5zkcu3dzUD94HafzS7Lcb1O5UVgjgKNCND4QAvEQEg
I0504 17:41:48.260068       1 round_trippers.go:393]     Content-Type: application/json
I0504 17:41:48.260074       1 round_trippers.go:393]     User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format
I0504 17:41:48.261611       1 round_trippers.go:408] Response Status: 201 Created in 1 milliseconds
I0504 17:41:48.261866       1 wrap.go:42] GET /swagger.json: (2.089729ms) 404 [[] 100.126.0.0:17268]
I0504 17:41:48.571698       1 wrap.go:42] GET /swagger.json: (202.436µs) 404 [[] 100.99.92.0:45258]
I0504 17:41:48.604608       1 wrap.go:42] GET /swagger.json: (179.826µs) 404 [[] 100.99.84.0:55292]
I0504 17:41:50.440373       1 round_trippers.go:383] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 17:41:50.440391       1 round_trippers.go:390] Request Headers:
I0504 17:41:50.440397       1 round_trippers.go:393]     Content-Type: application/json
I0504 17:41:50.440401       1 round_trippers.go:393]     User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format

When I turn on more verbose logging I'm seeing a lot of things like:

I0504 18:29:10.853028       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (218.596µs) 404 [[kubectl/v1.10.1 (darwin/amd64) kubernetes/d4ab475] 100.99.84.0:62308]
I0504 18:29:11.002233       1 request.go:991] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0504 18:29:11.420526       1 round_trippers.go:386] curl -k -v -XPOST  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXItdG9rZW4tMnp3NnMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmloaWxpc3QtbW9ua2V5LWN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY0NTgwMTJhLTRmYzgtMTFlOC05YTI4LTBhNGFjOGVmYTA0NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOm5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.EON5xGFxVjswqu9U3h1Dt1baRBWVU4_MuACXHFr2PzmbpqCNACTU9Os55a2PY3WJGbyUDQ2LPhmurGvW3RWc6nRcLBoL3-F1L6Ri3na7LjUt67gHa56GcyhWNeHgKHTKownKqYvR7dqqsAR7T2fHAM5oSmL1w9qokl7hMCmcXVxoF8SxVNBp-xYOCcYCeG4O0-f4XRzlfdEz5f4VbJYkHty-KDyouSUxin_VTrJZyTX-3h138MJlt-H_aBtostre9D7tp_XKKgMvMp-fRP8ff6MALXE7NkWV1DU3OVJj2OXCnv9Jnq3VEtnxMdsKkrYH8FkYFqYHCVsz5OZxT3nTEw" -H "User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Content-Type: application/json" https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 18:29:11.424231       1 round_trippers.go:405] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 3 milliseconds
I0504 18:29:11.424251       1 round_trippers.go:411] Response Headers:
I0504 18:29:11.424255       1 round_trippers.go:414]     Content-Type: application/json
I0504 18:29:11.424259       1 round_trippers.go:414]     Content-Length: 395
I0504 18:29:11.424263       1 round_trippers.go:414]     Date: Fri, 04 May 2018 18:29:11 GMT
I0504 18:29:11.424292       1 request.go:991] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true}}
I0504 18:29:11.424432       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (422.406836ms) 404 [[kube-controller-manager/v1.9.7 (linux/amd64) kubernetes/dd5e1a2/system:serviceaccount:kube-system:resourcequota-controller] 100.99.84.0:62308]
I0504 18:29:11.592583       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (225.387µs) 404 [[kubectl/v1.10.1 (darwin/amd64) kubernetes/d4ab475] 100.99.84.0:62308]
I0504 18:29:11.648679       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (210.463µs) 404 [[kubectl/v1.10.1 (darwin/amd64) kubernetes/d4ab475] 100.99.84.0:62308]
I0504 18:29:12.802287       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (214.386µs) 404 [[kube-controller-manager/v1.9.7 (linux/amd64) kubernetes/dd5e1a2/system:serviceaccount:kube-system:resourcequota-controller] 100.99.84.0:62308]
I0504 18:29:15.228040       1 request.go:991] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}

I've been trying to look at the difference b/t the 2, but I feel like I'm spinning my wheels a little, hoping with the details and fact that I can get Luxas to work will help surface my issue. More than happy to post anything that will help.

The adapter makes some assumptions about the Prometheus labels attached to metrics. We had problems originally when it couldn't find a Pod metric because it was missing the pod label.

To scrape Pod metrics correctly we ended up with something like this in our Prometheus config:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: pod
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace

I've not looked into collecting metrics for other types of object (Ingress, Service etc.) but I'm guessing they probably also depend on certain labels. It'd be great to have a reference that says which labels must be set in Prometheus for it to succeed.

Thanks!

In the case of metrics scraped by the Prometheus Kubernetes service discovery mechanism, it looks like labels on series can get autocreated based on labels from Kubernetes. This could allow us to bypass querying the main API server, but knowing when these are available might be tricky.

Will make cert generation, templating etc. much easier

reopen issue #40

@DirectXMan12 @itskingori I encountered same issue that message "metric xxx not registered" from function QueryForMetric, which caused metric discovery functionality doesn't work by using default prometheus-kubernetes.yml, then I try add --label-prefix=kubernetes_, it still did not work, only patch prometheus-kubernetes.yml diff provide by @itskingori can workaround.

I think the root cause is that you can't use --label-prefix=kubernetes_ to map kubernetes_ namespaces and kubernetes_pod_name` in the same time. (take job_name: 'kubernetes-pods' as an example)

Hi @DirectXMan12,

I deploy k8s-prometheus-adapter to minikube via this manifest: https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/custom-metrics.yaml

And it get CrashLoopBackoff. The log is showing:

panic: cluster doesn't provide requestheader-client-ca-file

Full log: https://gist.github.com/ngtuna/9235afed0d6d14cdd0c7c6e00bb46e05

I don't know if I am doing anything wrong, and I don't really understand why it needs requestheader-client-ca-file. Please help :-)

Firstly thank you for this awesome project. I'm to the point where I've completed the sample and I'm now looking to do more. My current confusion is the custom metrics namespace, and translation. When I query the endpoint via kubectl I see lots of metrics. I notice prefixes like pods, namespace, and job.batch. I've assumed pods translate to pods in my hpa yaml. Not sure how I go about interacting with the rest. Any links to docs or guidance is appreciated!

Thanks!

I have a custom metric, which is range [0, 300]. And in prometheus, I see the metric values collected do be in this range. But I get this value from custom apis apis/custom-metrics.metrics.k8s.io/v1alpha1, and find that this value is larger than 300 (e.g. 1400), which is quite strange.

"metricName":"worker_queue_length_percentage","timestamp":"2017-11-07T10:57:18Z","value":"1600"

The version I used is: k8s 1.7 and k8s-prometheus-adapter v0.1.0. And the custom metric is a pod metric.

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects

Hi,

I deploy Prometheus Operator(kube-prometheus/hack/cluster-monitoring/deploy) and custom metrics server through adapter build. The specific metric is able to access through curl and kubectl but not directly from api call inside a program which returns empty without any errors. I try to know if I miss something.
apiPath is set to /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/memory_max_usage_bytes and code snippet is the following.

request := &http.Request{
Header: make(http.Header),
Method: http.MethodGet,
URL: &url.URL{
Host: apiHost,
Path: apiPath,
Scheme: "http",
},
}

request.Header.Set("Accept", "application/json, /")
resp, err := http.DefaultClient.Do(request)

Also, I am able to get metrics list from /apis/custom.metrics.k8s.io/v1beta1 in the same program.

Thanks.

1. make docker-build is not working
2. Even if we make it work, does it work with Kubernetes 1.9 ?

custom metrics sometime return 404

deploy 2 node k8s cluster(1.9.1) on bare metal, and setup prometheus adapter for custom metrics, and using k8s metrics server for pods metrics. pods metrics are working fine with HPA and also I am able to autoscale them, but when I am trying to use custom metrics for HPA, sometime not getting result from API server, getting 404. sometime it is working fine.
I am using prometheus JMX exporter to fetch tomcat metrics.
have deploy k8s prometheus adapter here is yaml

#- adapter yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: custom-metrics
namespace: kube-system
labels:
app: custom-metrics
spec:
replicas: 1
selector:
matchLabels:
app: custom-metrics
template:
metadata:
labels:
app: custom-metrics
spec:
containers:
- name: custom-metrics
image: registry.text.com/k8s-prometheus-adapter-amd64
args:
- --prometheus-url=http://prometheus.monitoring.svc:9090
- --metrics-relist-interval=30s
- --rate-interval=60s
- --v=4
- --logtostderr=true
ports:
- containerPort: 443
securityContext:
runAsUser: 0

apiVersion: v1
kind: Service
metadata:
name: api
namespace: kube-system
spec:
ports:

• port: 443
  targetPort: 443
  selector:
  app: custom-metrics

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
insecureSkipTLSVerify: true
group: custom.metrics.k8s.io
groupPriorityMinimum: 1000
versionPriority: 5
service:
name: api
namespace: kube-system
version: v1beta1

#- hpa yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: fresco
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: fresco
minReplicas: 1
maxReplicas: 2
metrics:

• type: Pods
  pods:
  metricName: tomcat_threadpool_currentthreadcount
  targetAverageValue: 50

only one pod is running
}[root@docker01 prometheus]# kubectl get pods
NAME READY STATUS RESTARTS AGE
fresco-6c96cc7b57-4977m 1/1 Running 0 43m

but when trying to fetch custom metrics sometime get 404
}[root@docker01 prometheus]# curl -k -u admin:***** https://10.60.1.1/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/jvm_memory_bytes_used
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/jvm_memory_bytes_used"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "fresco-6c96cc7b57-4977m",
"apiVersion": "/__internal"
},
"metricName": "jvm_memory_bytes_used",
"timestamp": "2018-01-29T09:59:45Z",
"value": "434659600"
}
]
}

[root@docker01 prometheus]# curl -k -u admin:****** https://10.60.1.1/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/jvm_memory_bytes_used
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "the server could not find the metric jvm_memory_bytes_used for pods",
"reason": "NotFound",
"code": 404
}

#- also it is giving 404 for prometheus adapter api aggregate server
[root@docker01 ~]# curl -k -u admin:AdminPass https://10.60.1.111/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/jvm_memory_bytes_used
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "the server could not find the metric jvm_memory_bytes_used for pods",
"reason": "NotFound",
"code": 404

hpa describe
[root@docker01 prometheus]# kubectl describe hpa fresco
Name: fresco
Namespace: default
Labels:
Annotations: autoscaling.alpha.kubernetes.io/conditions=[{"type":"AbleToScale","status":"True","lastTransitionTime":"2018-01-29T09:37:00Z","reason":"SucceededGetScale","message":"the HPA controller was able to get...
kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"fresco","namespace":"default"},"spec":{"maxRe...
CreationTimestamp: Mon, 29 Jan 2018 15:06:30 +0530
Reference: Deployment/fresco
Metrics: ( current / target )
"tomcat_threadpool_currentthreadcount" on pods: 30 / 50
Min replicas: 1
Max replicas: 2
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message

22m 11m 13 horizontal-pod-autoscaler Warning FailedComputeMetricsReplicas failed to get pods metric value: unable to get metric tomcat_threadpool_currentthreadcount: unable to fetch metrics from API: the server could not find the metric tomcat_threadpool_currentthreadcount for pods
22m 2m 22 horizontal-pod-autoscaler Warning FailedGetPodsMetric unable to get metric tomcat_threadpool_currentthreadcount: unable to fetch metrics from API: the server could not find the metric tomcat_threadpool_currentthreadcount for pods

Hi,
Im trying to follow the walkthrough to deploy the custom-metrics-adapter, but i run into a couple of issues.

• When creating the resource-lister clusterrole i get an error as mentioned the below:
  src/github.com/prometheus/prometheus# kubectl create clusterrole resource-lister --verb=list --resource="*"
the server doesn't have a resource type "*".

Also for now i use --resource=pods just to be able to create the clusterrole to proceed.

• While trying to create the deployment and expose it as a service it fails as shown below:
  src/github.com/prometheus/prometheus# kubectl -n prom create -f prom-adapter.deployment.yaml
The Deployment "prometheus" is invalid: spec.template.spec.containers[0].volumeMounts[0].name: Not found: "prom-config"

Has anyone run into something similar? Is there something missing in my cluster setup/configs?

Travis initially?
Maybe could run a kubeadm cluster for e2e integration?

We currently don't support metrics on non-namespaced resources (except namespaces themselves). We need to figure out a quick way to discover all Kubernetes-related non-namespaced metrics (since we can't just look for the namespace label when listing available metrics like we do for namespaced resources).

The log shows it can download the extension-apiserver-authentication configmap but the app crashes with Unable to get configmap/extension-apiserver-authentication. Log:

kubectl -n cop-prometheus-adapter logs -f custom-metrics-apiserver-58945686d4-6dlsv
I0629 15:51:18.618704       1 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer redacted" 'https://172.23.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication'
I0629 15:51:18.631320       1 round_trippers.go:405] GET https://172.23.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 12 milliseconds
I0629 15:51:18.631338       1 round_trippers.go:411] Response Headers:
I0629 15:51:18.631343       1 round_trippers.go:414]     Content-Type: application/json
I0629 15:51:18.631347       1 round_trippers.go:414]     Date: Fri, 29 Jun 2018 15:51:18 GMT
I0629 15:51:18.632449       1 request.go:897] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"96d21401-4609-11e7-9977-005056b22938","resourceVersion":"87015391","creationTimestamp":"2017-05-31T14:01:05Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE redacted
requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
W0629 15:51:18.634396       1 authentication.go:237] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
Error: open /tmp/client-ca-file728862183: no such file or directory
Usage:
   [flags]

Flags:
      --alsologtostderr
...
panic: open /tmp/client-ca-file728862183: no such file or directory

goroutine 1 [running]:
main.main()
        /data/build/go/src/github.com/directxman12/k8s-prometheus-adapter/cmd/adapter/adapter.go:41 +0x13b

What is the recommended way for implementing this when dealing with multiple prometheus instances? We have teams broken into their own namespaces, each with its own prometheus server. Is there a straightforward way to be able to allow users to configure the hpa in their namespace and have it just talk to their prometheus instance? It seems the prometheus adapter gets registered as the target for the apiservice which looks like it can't really point to multiple adapters. Is there a way of accomplishing this, or would a completely new service layer be needed to do that type of namespace aware proxying?

The best practice that the repository should have LICENSE file according to here so that tells others what they can and can't do with the source code, it's important to make an informed decision.

I notice that many files tag Apache License, Version 2.0 license, but we should make it more clear by LICENSE file

Hi,

I have some custom node level metrics. Can prometheus adapter expose them?
If I run the following command, I can see some node metrics.

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/ | jq . | grep nodes
"name": "nodes/kube_pod_info",
"name": "nodes/kube_node_spec_unschedulable",
"name": "pods/fs_inodes",
"name": "nodes/kube_node_status_allocatable_memory_bytes",
"name": "nodes/kube_node_status_condition",
"name": "nodes/kube_node_created",
"name": "nodes/kube_node_info",
"name": "nodes/kube_pod_container_resource_limits_cpu_cores",
"name": "nodes/kube_node_status_capacity_memory_bytes",
"name": "nodes/kube_node_status_capacity_pods",
"name": "nodes/kube_pod_container_resource_requests_cpu_cores",
"name": "nodes/kube_node_status_allocatable_pods",
"name": "nodes/kube_node_labels",
"name": "nodes/kube_pod_container_resource_requests_memory_bytes",
"name": "pods/fs_inodes_free",
"name": "nodes/kube_pod_container_resource_limits_memory_bytes",
"name": "nodes/kube_node_status_allocatable_cpu_cores",
"name": "nodes/kube_node_status_capacity_cpu_cores",

Then, how can I access them?

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/nodes/*/kube_pod_container_resource_limits_memory_bytes
Error from server (InternalError): Internal error occurred: unable to list matching resources

Thanks.

If I want to autoscale base on deployment avg memory used ,how to do ,does any tutorial?

$ go build -o adapter github.com/directxman12/k8s-prometheus-adapter/cmd
# github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/openapi
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:88: cannot use config.Info (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".Info) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Info in field value
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:137: cannot use func literal (type func(string, *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".Route) (string, []string, error)) as type func(string, *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/emicklei/go-restful".Route) (string, []string, error) in assignment
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:142: cannot use func literal (type func(string, string) (string, "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Extensions)) as type func(string, string) (string, "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".Extensions) in assignment
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:144: cannot use func literal (type func(string) "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Ref) as type openapi.ReferenceCallback in argument to o.config.GetDefinitions
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:149: cannot use map[int]"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Response literal (type map[int]"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Response) as type map[int]"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".Response in assignment
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:156: cannot use *o.config.SecurityDefinitions (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".SecurityDefinitions) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".SecurityDefinitions in assignment
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:160: cannot use o.swagger (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Swagger) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".Swagger in argument to o.config.PostProcessSpec
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:160: cannot assign *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".Swagger to o.swagger (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".Swagger) in multiple assignment
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:219: cannot use item.Schema.VendorExtensible (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".VendorExtensible) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".VendorExtensible in field value
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:220: cannot use item.Schema.SchemaProps (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/vendor/github.com/go-openapi/spec".SchemaProps) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/go-openapi/spec".SchemaProps in field value
vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:220: too many errors
# github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1
vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1/types.generated.go:530: cannot use e (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/ugorji/go/codec".Encoder) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/client-go/vendor/github.com/ugorji/go/codec".Encoder in argument to yy10.CodecEncodeSelf
vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1/types.generated.go:536: cannot use e (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/ugorji/go/codec".Encoder) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/client-go/vendor/github.com/ugorji/go/codec".Encoder in argument to yy12.CodecEncodeSelf
vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1/types.generated.go:740: cannot use d (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/ugorji/go/codec".Decoder) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/client-go/vendor/github.com/ugorji/go/codec".Decoder in argument to yyv8.CodecDecodeSelf
vendor/k8s.io/metrics/pkg/apis/custom_metrics/v1alpha1/types.generated.go:875: cannot use d (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/ugorji/go/codec".Decoder) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/client-go/vendor/github.com/ugorji/go/codec".Decoder in argument to yyv22.CodecDecodeSelf
# github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:74: cannot use ws (type *"github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".WebService) as type *"github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".WebService in argument to versionDiscoveryHandler.AddToWebService
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:246: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:249: cannot use rootScopedHandler (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(rootScopedPath).To
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:271: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:272: cannot use namespacedHandler (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(namespacedPath).To
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:295: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
vendor/github.com/directxman12/custom-metrics-boilerplate/pkg/apiserver/installer/installer.go:296: cannot use namespaceSpecificHandler (type "github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/directxman12/k8s-prometheus-adapter/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(namespaceSpecificPath).To

@DirectXMan12 Maybe this was caused by the updated dependencies -- are code changes needed to use these new packages?

From what I can tell, the "metric discovery" currently being performed only exists to allow doing rate functions with a single metric "name". One can always create recording rules, but that would mean, that one also has access to modify the Kubernetes config.

At the point where one is using this adapter that person/organization has already sold into the Prometheus ecosystem pretty heavily: dashboards, alerts, etc.

Furthermore currently a very specific set of labels are expected in order for a metric to be associated with an object. This is not necessarily the case. For example a distributed worker infrastructure should be horizontally scaled based on the queue length of a certain queue of the message broker cluster, meaning currently this requires hacks as the metric for that queue would not have the respective labels. This is the most common use of custom metrics, in fact I've already been approached multiple times regarding this issue.

I propose to drop the resource intensive metric discovery and allow passing bare Prometheus queries. Even metric names are likely specific to Prometheus naming conventions, so I don't think it makes a lot of sense to try and generalize where the generalization doesn't even give us a benefit.

@DirectXMan12

I0606 19:43:11.756324 1 round_trippers.go:436] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 3 milliseconds
I0606 19:43:11.756338 1 round_trippers.go:442] Response Headers:
I0606 19:43:11.756342 1 round_trippers.go:445] Content-Type: application/json
I0606 19:43:11.756346 1 round_trippers.go:445] Content-Length: 260
I0606 19:43:11.756349 1 round_trippers.go:445] Date: Wed, 06 Jun 2018 19:43:11 GMT
I0606 19:43:11.756370 1 request.go:836] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0606 19:43:11.756432 1 authorization.go:59] Forbidden: "/", Reason: ""
I0606 19:43:11.756644 1 wrap.go:42] GET /: (3.763752ms) 403 [[Go-http-client/2.0] 100.96.152.0:32946]

@DirectXMan12 Please add the vendor folder and upload to the repo for easier building

Hi,

there is a lot of translations involved when exposing prometheus metrics to kubernetes. These are currently hardcoded. It would be great to have a configuration file which defines this mapping explicitly, helping both maintaining it as well as understanding what is going on.

Having a config file with mapping is common among prometheus projects, e.g:

• https://github.com/prometheus/statsd_exporter#metric-mapping-and-configuration
• https://github.com/prometheus/jmx_exporter#configuration

When I follow the walkthrough, after deploying prom-adapter.deployment.yaml file.
prometheus pod shows status Error and prometheus container goes in exit state with below error :-
configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:prom:prom-cm-adapter" cannot get configmaps in the namespace "kube-system"

It looks like when we generate rolebinding using :
kubectl create rolebinding prom-ext-auth-reader --role="extension-apiserver-authentication-reader" --serviceaccount=prom:prom-cm-adapter throws above error.

I have tried above command with prom and default namespaces but the pod failed to Run.

when I tried same command with namespace kube-system, it worked fine as the Pod was seen up and running.

Can anyone please let me know why this happens? Why does it work using kube-system as namespace?

Also if this is the right approach, we would need to update the walkthrough with the same.

I checked the codes, that adapter use http.DefaultClient to access prometheus instance and it doesn't support https client to access ssl/tls enabled prometheus. It is a gap here if prometheus behind a proxy or
prometheus enable ssl/tls according to official guideline Why don't the Prometheus server components support TLS or authentication? Can I add those?

This is a common case that prometheus in production, do we have any plan for this? Thanks in advance.

When making a request to the API for a custom metric, requests are being dropped:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/*/tasks_count" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/%2A/tasks_count"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "test-ns",
        "name": "pg-service",
        "apiVersion": "/__internal"
      },
      "metricName": "tasks_count",
      "timestamp": "2018-06-22T07:24:13Z",
      "value": "31"
    }
  ]
}
 ~ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/*/tasks_count" | jq .
Error from server (NotFound): the server could not find the metric tasks_count for pods
 ~ kubectl  get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/*/tasks_count" | jq .
Error from server (NotFound): the server could not find the metric tasks_count for pods
 ~ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/*/tasks_count" | jq .
Error from server (NotFound): the server could not find the metric tasks_count for pods
 ~ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/*/tasks_count" | jq .
Error from server (NotFound): the server could not find the metric tasks_count for pods
 ~ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/*/tasks_count" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test-ns/pods/%2A/tasks_count"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "test-ns",
        "name": "pg-service",
        "apiVersion": "/__internal"
      },
      "metricName": "tasks_count",
      "timestamp": "2018-06-22T07:24:57Z",
      "value": "31"
    }
  ]
}

This seems to result in the HPA not finding the metric:

 kubectl describe hpa task-service
Name:                                task-service
Namespace:                           test-ns
Labels:                              <none>
Annotations:                         kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"task-service","names...
CreationTimestamp:                   Thu, 21 Jun 2018 07:26:48 -0700
Reference:                           Deployment/task-service
Metrics:                             ( current / target )
  "tasks_count" on pods:  <unknown> / 20
Min replicas:                        1
Max replicas:                        10
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric tasks_count: unable to fetch metrics from API: the server could not find the metric tasks_count for pods
Events:
  Type     Reason               Age                  From                       Message
  ----     ------               ----                 ----                       -------
  Warning  FailedGetPodsMetric  6m (x1149 over 17h)  horizontal-pod-autoscaler  unable to get metric tasks_count: unable to fetch metrics from API: the server could not find the metric tasks_count for pods
  Warning  FailedGetPodsMetric  1m (x960 over 17h)   horizontal-pod-autoscaler  unable to get metric tasks_count: no metrics returned from custom metrics API

In Github page, i found a way to fetch any custom metrics related to Pod. How do i fetch the same for other services such as nodes,replications etc ?

Ex:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

{
"name": "namespaces/kube_deployment_status_replicas_updated",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "replicasets.extensions/kube_replicaset_status_observed_generation",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "pods/memory_failcnt",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "daemonsets.extensions/kube_daemonset_status_number_available",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
}

What is the self -link for "daemonsets.extensions/kube_daemonset_status_number_available" and replicasets.extensions/kube_replicaset_status_observed_generation ?

1. I followed walkthrough to deploy sample app. I could get prometheus and prometheus-adapter pods up and running. When i check logs of prometheus container it shows:
   level=error ts=2018-05-28T05:47:05.224231355Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:296: Failed to watch *v1.Pod: unknown (get pods)"

   It works when i updated resource-lister.yaml to use verbs: '*' instead of 'list', as shown below:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: resource-lister
rules:
- apiGroups:
  - ""
  resources:
  - '*'
  verbs:
  - '*'

2. When i tried Collecting Application Metrics steps, i was unable to find luxas/autoscale-demo image on dockerhub.
   How do I get this image or build it to move ahead with the walkthrough?

On a namespace where the following queries returns a result (say "2"):

sum( rabbitmq_queue_messages_ready{})  by (kubernetes_namespace ) > 0

Trying it directly seems to return 0:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/staging/metrics/rabbitmq_queue_messages_ready

Result:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/staging/metrics/rabbitmq_queue_messages_ready"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Namespace",
        "name": "staging",
        "apiVersion": "/__internal"
      },
      "metricName": "rabbitmq_queue_messages_ready",
      "timestamp": "2018-02-05T22:12:21Z",
      "value": "0"
    }
  ]
}

From the logs it's seems to be calling rate() within the sum():

7471c5_0[1298]: I0205 22:12:21.725052       1 api.go:74] GET http://prometheus.kube-system-monitoring.svc:9090/prometheus/api/v1/query?query=sum%28rate%28rabbitmq_queue_messages_ready_total%7Bkubernetes_namespace%3D%22staging%22%7D%5B5m%5D%29%29+by+%28kubernetes_namespace%29&time=1517868741.722 200 OK

This seems to be a bug as per:
https://github.com/DirectXMan12/k8s-prometheus-adapter/#presentation

Cheers!

Hi I'm trying to follow the walkthrough and deploy the custom metrics adapter but it seems that the image doesn't exist anymore?

Warning  Failed                 15s                kubelet, gke-custom-metrics-default-pool-293ce8a3-1tj5  Failed to pull image "directxman12/k8s-prometheus-adapter": rpc error: code = Unknown desc = Error response from daemon: manifest for directxman12/k8s-prometheus-adapter:latest not found

We have a Prometheus server that is receiving (actually fetching) a lot of metrics. This means that the request /api/v1/series that the k8s-prometheus-adapter sends get a huge response.

We observed that from time to time our HPA object is not able get the metrics from the k8s-prometheus-adapter and I presume that the problem is that the server (k8s-prometheus-adapter) is not able to handle that many replicas being stored in prometheus.

The question here is if's possible to make this request more configurable so the result of the /api/v1/series can return less results

Hello.

Metrics must have the namespace label to be considered.

For each label on a metric, if that label name corresponds to a Kubernetes resource (like pod or service), the metric will be associated with that resource.

Is there any way to configure it? My labels are "kubernetes_namespace", "kubernetes_pod_name" and so forth. Therefore the metrics I need aren't captured by the adapter. I have an existing and long time running prometheus clusters all over the place, it would be potentially a breaking change to relabel everything.

What's the best way to go here?

Thank you!

Conjecture here, but I'm guessing most people will set up Prometheus for Kubernetes using the official out of the box integration config. Which is pretty good IMHO. I'd like to suggest that you support that default (opinionated) config so that this tool works out of the box. That way you avoid having to do this:

diff --git a/common/resources/prometheus/server/02_config_maps.yml b/common/resources/prometheus/server/02_config_maps.yml
index 354afe625..cfb734c56 100644
--- a/common/resources/prometheus/server/02_config_maps.yml
+++ b/common/resources/prometheus/server/02_config_maps.yml
@@ -81,10 +81,10 @@ data:
             regex: __meta_kubernetes_service_label_(.+)
           - source_labels: [__meta_kubernetes_namespace]
             action: replace
-            target_label: kubernetes_namespace
+            target_label: namespace
           - source_labels: [__meta_kubernetes_service_name]
             action: replace
-            target_label: kubernetes_name
+            target_label: service
       - job_name: 'kubernetes-services'
         metrics_path: /probe
         params:
@@ -104,9 +104,9 @@ data:
           - action: labelmap
             regex: __meta_kubernetes_service_label_(.+)
           - source_labels: [__meta_kubernetes_namespace]
-            target_label: kubernetes_namespace
+            target_label: namespace
           - source_labels: [__meta_kubernetes_service_name]
-            target_label: kubernetes_name
+            target_label: service
       - job_name: 'kubernetes-ingresses'
         metrics_path: /probe
         params:
@@ -128,9 +128,9 @@ data:
             - action: labelmap
               regex: __meta_kubernetes_ingress_label_(.+)
             - source_labels: [__meta_kubernetes_namespace]
-              target_label: kubernetes_namespace
+              target_label: namespace
             - source_labels: [__meta_kubernetes_ingress_name]
-              target_label: kubernetes_name
+              target_label: ingress
       - job_name: 'kubernetes-pods'
         kubernetes_sd_configs:
           - role: pod
@@ -151,7 +151,7 @@ data:
             regex: __meta_kubernetes_pod_label_(.+)
           - source_labels: [__meta_kubernetes_namespace]
             action: replace
-            target_label: kubernetes_namespace
+            target_label: namespace
           - source_labels: [__meta_kubernetes_pod_name]
             action: replace
-            target_label: kubernetes_pod_name
+            target_label: pod

Thoughts?