Is it based on kubectl top node CPU & memory? about descheduler

[aveshagarwal]

@mossuchida descheduler follows what kube-scheduler does. As kube-scheduler works based on pods requests, descheduler also works based on pods requests. IMO, making descheduler work on real time utilization (like kubectl top) without kube-scheduler working on real time usage might lead to mismatch how pods are evicted by descheduler and then how replacement pods are scheduled by kube-scheduler.

But as soon as kube-scheduler starts supporting real time usage, descheduler would start that too.

[InAnimaTe]

Thanks for clearing that up. Really, that's the best you can do. I kind of just assumed otherwise since I really have a usecase for something that can help when we have so many services which don't have resources defined.

Honestly, it might really be worth describing this fact out in the README so its abundantly clear.

[AndresPineros]

I don't understand why is the descheduler using the "requests". For me the idea of the descheduler was to balance the cluster when the requests where not properly assigned, for example, a service wasn't expected to consume 5Gi of RAM and a limit wasn't set for it. Balancing using the requests is already done by the Kubernetes scheduler, right?

Without real-time the descheduler only tries to fill empty nodes, it doesn't try to save loaded nodes that are about to die due to resources.

[aveshagarwal]

I don't understand why is the descheduler using the "requests". For me the idea of the descheduler was to balance the cluster when the requests where not properly assigned, for example, a service wasn't expected to consume 5Gi of RAM and a limit wasn't set for it.

Descheduler only evicts pods but kube-scheduler does the actual scheduling of replacement pods. So if Descheduler acted based on anything other than requests, it would lead to mismatch between Descheduler and kube-scheduler as kube-scheduler would still schedule based on requests.

Balancing using the requests is already done by the Kubernetes scheduler, right?

kube-scheduler tries to balance only at the admission time for incoming pods, it does not rebalance already admitted pods.

Without real-time the descheduler only tries to fill empty nodes, it doesn't try to save loaded nodes that are about to die due to resources.

No, descheduler does both as far as cluster has capacity. It tries to move pods from loaded nodes to less loaded nodes.

I don't understand why is the descheduler using the "requests". For me the idea of the descheduler was to balance the cluster when the requests where not properly assigned, for example, a service wasn't expected to consume 5Gi of RAM and a limit wasn't set for it.

Descheduler only evicts pods but kube-scheduler does the actual scheduling of replacement pods. So if Descheduler acted based on anything other than requests, it would lead to mismatch between Descheduler and kube-scheduler as kube-scheduler would still schedule based on requests.

Balancing using the requests is already done by the Kubernetes scheduler, right?

kube-scheduler tries to balance only at the admission time for incoming pods, it does not rebalance already admitted pods.

Without real-time the descheduler only tries to fill empty nodes, it doesn't try to save loaded nodes that are about to die due to resources.

No, descheduler does both as far as cluster has capacity. It tries to move pods from loaded nodes to less loaded nodes.

@aveshagarwal - I am reading the Rescheduler-Design-Implementation (https://docs.google.com/document/d/1KXw02Q0cOF1MUrdpPNiug0yGZlixvPg2SwBycrT5DkE/edit) and saw it should support also the HighNodeUtilization option.

Per your above answer i understand that the descheduler should avict pods from nodes that reached high thresholds.

This is what i am trying to achieve but cannot seem to get that to work :(

Any idea how does a policy that balances HighNodeUtilization nodes should be defined ?

Thank you

Roiy

[AndresPineros]

@tmwtp Hey, I think I found what we were looking for:
https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/
https://docs.okd.io/latest/admin_guide/out_of_resource_handling.html

Some arguments in the Kubelet actually allow to evict pods from nodes under pressure. Mixing this with the descheduler (they way it works right now, not with real time resources) seems like a great combo.

Pay attention to the best practices and caveats:
https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#best-practices
https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#known-issues

This one from Kops but has relevant information for any K8s cluster:
https://github.com/kubernetes/kops/blob/master/docs/node_resource_handling.md

At this point, the descheduler WILL NOT realize if pods are using more memory that their requests, and are about to kill the Node. But it doesn't matter because it is a Kubernetes native feature.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[InAnimaTe]

Can someone clear this up for me then? So Descheduler literally does nothing for real node metrics, only acting on container resource.requests? So if I don't have any resources defined for my deployments, Descheduler won't help me in any way?

A little disheartening to be honest.

[aveshagarwal]

Can someone clear this up for me then? So Descheduler literally does nothing for real node metrics, only acting on container resource.requests? So if I don't have any resources defined for my deployments, Descheduler won't help me in any way?

Kubernetes's core/default scheduler does scheduling based on resource.requests, and Descheduler follows it.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[sidharthsurana]

/remove-lifecycle rotten

Is there any near term plans to add this support?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.