stefanprodan / istio-hpa Goto Github PK

Installed istio-hpa onto kubernetes 1.22.6.
kube-metrics-adapter has the following in the logs.
I0512 15:53:23.575559 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) I0512 15:53:23.901168 1 serve.go:96] Serving securely on [::]:443 time="2022-05-12T15:53:23Z" level=info msg="Looking for HPAs" provider=hpa time="2022-05-12T15:53:23Z" level=info msg="Found 3 new/updated HPA(s)" provider=hpa I0512 15:53:24.147708 1 logs.go:49] http: TLS handshake error from 100.107.114.128:50213: remote error: tls: bad certificate E0512 15:53:24.148062 1 webhook.go:192] Failed to make webhook authorizer request: the server could not find the requested resource
Cluster has cert-manager installed if that makes any difference.

sum(
rate(
istio_requests_total{
destination_workload="podinfo",
destination_workload_namespace="test"
}[1m]
)
) /
count(
count(
container_memory_usage_bytes{
namespace="test",
pod_name=~"podinfo.*"
}
) by (pod_name)
)

kubectl -n kube-system logs deployment/kube-metrics-adapter
results / Error from server (NotFound): the server could not find the requested resource ( pods/log kube-metrics-adapter-596bd9bb67-bzz2k)

deployments are successful but since i deploy the kube-metrics-adapter i cant accces to any deployments or pod logs
I am missing something???

Hello @stefanprodan thanks a lot for doing this great work. We use this on our production since more than 2 years to autoscale our rest based containers based on request per min metrics.
Can you please give an example on how we can do the same for gRPC based api's ?

Error: cluster doesn't provide client-ca-file

I'm getting better results with this query to undestand the number of request per second a pod is doing:
round(
sum(
irate(istio_requests_total{reporter="destination",destination_workload_namespace="namespace",destination_workload="pod-name"}[1m])
),
0.001)
/
count(
count(
container_memory_usage_bytes{
namespace="namespace",
pod_name=~"pod-name.*"
}
) by (pod_name)
)

The first part I've taken from the istio workload dashboard itself.

With the example query I get double the size of the requests

Changed HPA metric format from:

metrics:
  - type: Object
    object:
      metricName: istio-requests-total
      target:
        apiVersion: v1
        kind: Pod
        name: podinfo
      targetValue: 10

to:

metrics:
- type: Object
  object:
    metric:
      name: istio-requests-total
    describedObject:
      apiVersion: v1
      kind: Pod
      name: podinfo
    target:
      type: Value
      value: 10

Expected Behavior
I have a prometheus query that can be used for scaling, expected behavoir scale up using metric

Actual Behavior
Error: Warning FailedGetObjectMetric 3s (x2 over 18s) horizontal-pod-autoscaler unable to get metric jvm-memory-bytes-used: Service on dev event-service/unable to fetch metrics from custom metrics API: the server could not find the metric jvm-memory-bytes-used for services event-service

HPA yaml file:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: aggregation-hazelcast
namespace: dev
annotations:
metric-config.object.jvm-memory-bytes-used.prometheus/query: |
scalar((sum(jvm_memory_bytes_used{area="heap"}) ) / (sum(jvm_memory_bytes_max{area="heap"}))
metric-config.object.jvm-memory-bytes-used.prometheus/per-replica: "true"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: hazelcast
minReplicas: 1
maxReplicas: 10
metrics:
type: Object
object:
metricName: jvm-memory-bytes-used
target:
apiVersion: v1
kind: Service
name: event-service
targetValue: 80Mi # this will be treated as targetAverageValue
Specifications
Version: kops version 1.12.5
Platform: aws
I am I missing anything ? i provided the exact query which works in prometheus..please help !!

After deploying successfully, I get the following empty return:
get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": []
}
which is not the expected result according to the readme.md