Allow descheduling of Pods which have hostDirs about descheduler

[rmohr]

HI @aveshagarwal

but could you tell your use case/scenario why would you want that?

Here are a few scenarios, where the host directory content does not necessarily require the pod to be on that node:

• Accessing special devices, e.g. a random number generator (one has to mount /dev), which exist on all or a subset of nodes.
• Communicating across Pods (e.g. one daemon set which is on every node and it communicates with workload pods through a file socket).
• Having a pre-populated cache on every node, which Pods can access, to reduce the dependency on shared storage for static/semi-static content.

In KubeVirt we start our VMs in Pods which have HostDirs, and we use all three techniques, but I don't think that they are just unique to us.

also could you elaborate on "based on the content in kubernetes.io/created-by"?

If I am not mistaken, the descheduler deschedules all pods which have that annotation and don't meet some other excluding preconditions (daemon set, host dir, ...), so if you wrote your own controller (which means that the controller adds it's name to the annotation), you could fine-tune the eviction rules for that type of controller. However, an annotation based solution, like you mentioned would also solve it, and might even be better.

Btw., do you plan to replace/extend that with a check for a controller reference?

[aveshagarwal]

@yanirq yes, but could you tell your use case/scenario why would you want that? the reason to not evict a pod tied to hostDir is that it is possible the pod might end up on the same node making descheduling unnecessary? also could you elaborate on "based on the content in kubernetes.io/created-by"?

[yanirq]

@aveshagarwal we would like to allow descheduling of regular pods (without an association to RC or deployments) in our case scenario a pod represents a VM and we would like to run a migration process - move the pod to another node according to dscheduling assessment.

the meaning of using content in kubernetes.io/created-by is maybe using one of its attribute's value to signal that the pod can be up for descheduling although it has a hostDir.
will a pod tied to a hostDir necessarily end up on the same node ?

A bit more further - the ultimate goal of the migration process i mentioned above is to :
1.run descheduler
2.find a pod that can run on a better node
3.create a replacement pod
4.evict the current pod and get the replacement pod to run on the suggest node (forcefully by our migration process)

*A side note: this is also one of the reasons why we would desire the use of a scheduler simulator (dry-run scheduling)

[aveshagarwal]

@aveshagarwal we would like to allow descheduling of regular pods (without an association to RC or deployments) in our case scenario a pod represents a VM and we would like to run a migration process - move the pod to another node according to dscheduling assessment.

Once the pod is descheduled, who would do migration if the pod is not associated with RC or deployments or Job or RS?

the meaning of using content in kubernetes.io/created-by is maybe using one of its attribute's value to signal that the pod can be up for descheduling although it has a hostDir.

i think people would not prefer changing kubernetes.io/created-by for descheduler as it is an outside component. But having a pod annotated with something to signal it is a candidate for descheduling specifically or having a command line flag to allow descheduling pods with hostDir in general could be done.

will a pod tied to a hostDir necessarily end up on the same node ?

Depends how is that configured. But it is highly possible, as the pod was using storage on that node was the reason the pod was scheduled to that node.

A bit more further - the ultimate goal of the migration process i mentioned above is to :
1.run descheduler
2.find a pod that can run on a better node
3.create a replacement pod
4.evict the current pod and get the replacement pod to run on the suggest node (forcefully by our migration process)

what do you mean suggest node(forcefully by our migration process) ? It just sounds like you manually scheduler pod as a static pod or may be either using node affinity/node selectors.

*A side note: this is also one of the reasons why we would desire the use of a scheduler simulator (dry-run scheduling)

Having a dry run might be helpful. But as you said you want to force a pod on a node after migration, how would dry-run help here?

[ravisantoshgudimetla]

@rmohr - Though the use-cases you provided seem reasonable, in most scenarios I think evicting pods which need hostDirs may cause data consistency problems. However having a force option for descheduler which actually evicts all types of pods may solve your problem and justifies the name of descheduler(so that it is disruptive in nature).

@aveshagarwal. Any further thoughts on this?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rmohr]

/reopen

[k8s-ci-robot]

@rmohr: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rmohr]

Can this be reopened? It would still be useful.