This shows how to setup dynamic provisioning of nfs volumes with k3d allowing you to have ReadWriteMany volumes in your k3d cluster. The nfs PV's are provisioned using the nfs-ganesha-serverand-external-provisioner which is backed by an initial PV created with local-path-provisioner. The instructions below show how to configure this setup using the default settings from the nfs-ganesha-server. The README on the nfs-ganesha-server has more details if you wish to modify the config of the nfs server or provisioner.
- Create a k3d cluster
docker volume create kube-nfs-volume
k3d cluster create nfs --agents 2 --volume kube-nfs-volume:/opt/local-path-provisioner --wait
k3d kubeconfig merge nfs -d
- Install the local-path storage provisioner (will create a pvc to host the nfs share with this)
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
- Install the nfs-ganesha provisioner
# create the rbac rules needed
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/master/deploy/kubernetes/rbac.yaml
# create the base pvc
kubectl create -f pvc.yaml
# create the deployment
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/master/deploy/kubernetes/deployment.yaml
# patch the deployment to use the pvc instead of the default hostPath volume
k patch deploy nfs-provisioner --type=json -p='[{"op": "replace", "path": "/spec/template/spec/volumes", "value": [{ "name": "export-volume", "persistentVolumeClaim": {"claimName": "ganesha-pvc"}}]}]'
# create the storage class
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/master/deploy/kubernetes/class.yaml
Now the nfs-provisioner
deployment failed to start due to not being able to pull the image from quay.io
. I tried pulling it locally to no avail.
The image can be build locally but i used a mirror of it that was uploaded to Dockerhub here and pushed it into the cluster directly.
docker pull kvaps/nfs-ganesha-server-and-provisioner:latest
docker tag kvaps/nfs-ganesha-server-and-provisioner:latest quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest
k3d image import quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest -c nfs
The nfs-provisioner
should now start. You can test the pvc creation and usage with:
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/master/deploy/kubernetes/claim.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/master/deploy/kubernetes/write-pod.yaml