Explore local persistent volumes feasibility about cloud-on-k8s

[sebgl]

Discussing it on zoom with @nkvoll and @pebrc:

CSI does not work that well for us since it does not really map a persistent volume to a host. It works well for the case of "volume can actually be reached from any node". Otherwise, the controller plugin needs to take each individual node capacity into consideration and make sure to adapt affinity settings.

Works seems to be ongoing (cf. the design proposal) to more closely map local volumes attached to a particular node, dynamic provisioning, and remaining capacity considerations. Probably not ready for at least another year though; but that would be the appropriate "long-term" solution.

Meanwhile, a FlexVolume implementation seems to be the easier workaround for a short-term solution. The design is not as great as CSI, but simpler to deploy and implement.

The decision here for the short-term is to implement a FlexVolume plugin that handles LVM volumes provisioning. Having it running as a daemonset can also enable garbage collection on local volumes which are not referenced anymore.

It's harder to consider storage capacity though. A workaround that was also adopted on Cloud is to only consider RAM usage for pods allocation, while considering there is a global FS multiplier that we can apply from RAM to storage space. The goal here is that the cluster operator would run out of RAM before running out of disk on a given node.
This is to avoid hacky workarounds around persistent volume definitions vs. actual volume size that could still be implemented :-)

[sebgl]

I'm closing this since we implemented our own dynamic local-volume provisioner based on a flex driver.