isindir / pvc-autoresizer

Auto-resize PersistentVolumeClaim objects based on Prometheus metrics

Welcome to the pvc-autoresizer Project!

pvc-autoresizer resizes PersistentVolumeClaims (PVCs) when the free amount of storage is below the threshold.

It queries the volume usage metrics from Prometheus that collects metrics from kubelet.

• Project Status: Testing for production

Our supported platforms are:

• Kubernetes: 1.28, 1.27, 1.26
• CSI drivers that implements the following features
  • Volume Expansion
  • NodeGetVolumeStats

Container images are available on ghcr.io.

Getting Started

Prepare

pvc-autoresizer behaves based on the metrics that prometheus collects from kubelet.

Please refer to the following pages to set up Prometheus:

• Installation | Prometheus

In addition, configure scraping as follows:

• Monitoring with Prometheus | TopoLVM

Installation

Specify the Prometheus URL to pvc-autoresizer argument as --prometheus-url.

pvc-autoresizer can be deployed to a Kubernetes cluster via helm:

helm repo add pvc-autoresizer https://topolvm.github.io/pvc-autoresizer/
helm install --create-namespace --namespace pvc-autoresizer pvc-autoresizer pvc-autoresizer/pvc-autoresizer --set "controller.args.prometheusURL=<YOUR PROMETHEUS ENDPOINT>"

See the Chart README.md for detailed documentation on the Helm Chart.

How to use

To allow auto volume expansion, the StorageClass of PVC need to allow volume expansion and have resize.topolvm.io/enabled: "true" annotation. The annotation may be omitted if you give --no-annotation-check command-line flag to pvc-autoresizer executable.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: topolvm-provisioner
  annotations:
    resize.topolvm.io/enabled: "true"
provisioner: topolvm.io
allowVolumeExpansion: true

To allow auto volume expansion, the PVC to be resized needs to specify the upper limit of volume size with the annotation resize.topolvm.io/storage_limit. The value of resize.topolvm.io/storage_limit should not be zero, or the annotation will be ignored.

The PVC must have volumeMode: Filesystem, too.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: topolvm-pvc
  namespace: default
  annotations:
    resize.topolvm.io/storage_limit: 100Gi
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 30Gi
  storageClassName: topolvm-provisioner

The PVC can optionally have resize.topolvm.io/threshold, resize.topolvm.io/inodes-threshold and resize.topolvm.io/increase annotations. (If they are not given, the default value is 10%.)

When the amount of free space of the volume is below resize.topolvm.io/threshold or the number of free inodes is below resize.topolvm.io/inodes-threshold, .spec.resources.requests.storage is increased by resize.topolvm.io/increase.

If resize.topolvm.io/increase is given as a percentage, the value is calculated as the current spec.resources.requests.storage value multiplied by the annotation value.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: topolvm-pvc
  namespace: default
  annotations:
    resize.topolvm.io/storage_limit: 100Gi
    resize.topolvm.io/threshold: 20%
    resize.topolvm.io/inodes-threshold: 20%
    resize.topolvm.io/increase: 20Gi
spec:
  <snip>

Initial resize

PVC request size can also be changed at the creation time based on the largest PVC size in the same group. PVCs are grouped by labels, and the label key for grouping is specified by resize.topolvm.io/initial-resize-group-by annotation.

For example, suppose there are following three PVCs.

### existing PVCs (excerpted)
kind: PersistentVolumeClaim
metadata:
  name: pvc-x-1
  labels:
    label-foobar: group-x
  annotations:
    resize.topolvm.io/initial-resize-group-by: label-foobar
spec:
  resources:
    requests:
      storage: 20Gi

kind: PersistentVolumeClaim
metadata:
  name: pvc-x-2
  labels:
    label-foobar: group-x
  annotations:
    resize.topolvm.io/initial-resize-group-by: label-foobar
spec:
  resources:
    requests:
      storage: 16Gi

kind: PersistentVolumeClaim
metadata:
  name: pvc-y-1
  labels:
    label-foobar: group-y
  annotations:
    resize.topolvm.io/initial-resize-group-by: label-foobar
spec:
  resources:
    requests:
      storage: 30Gi

When creating the following new PVC, pvc-x-1 and pvc-x-2 with label-foobar: group-x are considered to be in the same group, and pvc-y-1 is not. Therefore, the PVC is created with 20Gi based on pvc-x-1, which has the largest capacity in the group.

kind: PersistentVolumeClaim
metadata:
  name: pvc-x-3
  labels:
    label-foobar: group-x
  annotations:
    resize.topolvm.io/initial-resize-group-by: label-foobar
spec:
  resources:
    requests:
      storage: 10Gi

When creating the following new PVC, pvc-y-1 with label-foobar: group-y is in the same group. However, since the new PVC's size(50Gi) is larger than the existing one(30Gi), the PVC is created with 50Gi.

kind: PersistentVolumeClaim
metadata:
  name: pvc-y-2
  labels:
    label-foobar: group-y
  annotations:
    resize.topolvm.io/initial-resize-group-by: label-foobar
spec:
  resources:
    requests:
      storage: 50Gi

When the size of the largest PVC in the same group is larger than the value set to resize.topolvm.io/storage_limit annotation, the PVC is resized up to this limit.

Prometheus metrics

pvcautoresizer_kubernetes_client_fail_total

pvcautoresizer_kubernetes_client_fail_total is a counter that indicates how many API requests to kube-api server are failed.

pvcautoresizer_metrics_client_fail_total

pvcautoresizer_metrics_client_fail_total is a counter that indicates how many API requests to metrics server(e.g. prometheus) are failed.

pvcautoresizer_loop_seconds_total

pvcautoresizer_loop_seconds_total is a counter that indicates the sum of seconds spent on volume expansion processing loops.

pvcautoresizer_success_resize_total

pvcautoresizer_success_resize_total is a counter that indicates how many volume expansion processing resizes succeed.

pvcautoresizer_failed_resize_total

pvcautoresizer_failed_resize_total is a counter that indicates how many volume expansion processing resizes fail.

pvcautoresizer_limit_reached_total

pvcautoresizer_limit_reached_total is a counter that indicates how many storage limit was reached.

Contributing

pvc-autoresizer project welcomes contributions from any member of our community. To get started contributing, please see our Contributor Guide.

Communications

If you have any questions or ideas, please use discussions.

Resources

docs directory contains designs, and so on.

License

This project is licensed under Apache License 2.0.