Feature request: Add more labels to kube_pod_container_status_running about kube-state-metrics

[fabxc]

You generally want to keep the number of 1:1 labels to a minimum. Otherwise, over time, you would basically enrich every metrics with all transitive and static label dimensions. If then something changes unexpectedly after all, you break all time series, as their identity is given by the full combination of all labels.
Generally speaking, you want your metrics to normalized.

What we probably want here are metrics that express the relation from pod to deployment and pod to node. In your queries, you can then connect the time series along them.

So the node name could probably be attached to the kube_pod_info metric as a label: https://github.com/kubernetes/kube-state-metrics/blob/master/pod.go#L26-L30

We could have a metric on the created-by relation (currently living as a SerializedRef type in the kubernetes.io/created-by annotation. For example (creator_namespace would be redundant):

kube_pod_created_by{pod="my-app-aef242",namespace="prod",creator_kind="ReplicaSet", creator_name="my-app-12341"} 1

Unfortunately, relation from ReplicaSet to Deployment is not yet accessible in a similar way, but we can just use regex matching below to express it via ReplicaSets.

Your PromQL query could then look like:

count(
  count by(node_name) (
     kube_pod_info 
     and on(pod,namespace) kube_pod_status_phase{phase="Running"}
     and on(pod,namespace) kube_pod_created_by{creator_kind="ReplicaSet", creator_name=~"my-app-[0-9]+"})
  )
)
/ count(kube_node_info)

I know this is not a trivial but imagine that this is just one use case of many to incorporate relationships between components into your queries. So hard cording it into the exposed metrics does solve just a single use case of many.

Recording rules can help you here to break these parts out into separate metrics, which can be useful for other alerts and dashboards as well.

I think a pull request for metric additions/changes as described above would be welcome.

[brancz]

I am unsure if we are able to get these upward and cross linking references from a pod. I'll take a little time to investigate.

[fejta-bot]

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close