Operator for dynamically provisioning an NFS server on any Kubernetes Persistent Volume. Also creates an NFS volume on the dynamically provisioned server for enabling Kubernetes RWX volumes.
License: Apache License 2.0
Describe the problem/challenge you have
I need to set the exports mount option sync on, because otherwise hard reboots cause file system corruption. This is already supported by the Alpine nfs-server image, through an environment variable SYNC=true.
Existing documentation doesn't describe how it would be possible to give SYNC=true environment variable for the nfs-server deployments through dynamic-nfs-provisioner.
Describe the solution you'd like
Support for passing environment variables to nfs-servers, and documentation mention for the same.
Anything else you would like to add:
This is a really useful project and thanks for the great work!
Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the bug: Creating chart for the dynamic-nfs-provisioner to make the deployment more generic
Expected behaviour: Deployment of dynamic-nfs-provisioner with helm
Environment details:
Describe the bug: A clear and concise description of what the bug is.
ReadWriteMany nfs pvc pvc-55940b24-bd9a-4d63-911c-809f9bfc10cd seems to have damaged filesystem after restarting pod it's used by
Expected behaviour: A concise description of what you expected to happen
Steps to reproduce the bug:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: cstor
namespace: openebs
spec:
interval: 10m
chart:
spec:
chart: cstor
interval: 5m
sourceRef:
kind: HelmRepository
name: openebs-cstor
namespace: flux-system
values:
csiNode:
kubeletDir: "/var/snap/microk8s/common/var/lib/kubelet/"
openebs-ndm:
# ndm options:
# https://github.com/openebs/node-disk-manager/blob/develop/deploy/helm/charts/values.yaml#L44
ndm:
filters:
osDiskExcludePaths: "/,/etc/hosts,/boot,/boot/firmware"
featureGates:
UseOSDisk:
enabled: true
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: openebs-nfs
namespace: openebs
spec:
interval: 10m
timeout: 7m
chart:
spec:
chart: nfs-provisioner
interval: 5m
sourceRef:
kind: HelmRepository
name: openebs-nfs
namespace: flux-system
values:
nfsStorageClass:
name: openebs-nfs
backendStorageClass: cstor-csi
isDefaultClass: true
reclaimPolicy: Delete
analytics:
enabled: false
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: plex-config
namespace: media
labels:
app: plex
spec:
storageClassName: openebs-nfs
volumeMode: Filesystem
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: plex-data
namespace: media
labels:
app: plex
spec:
storageClassName: openebs-nfs
volumeMode: Filesystem
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
apiVersion: apps/v1
kind: Deployment
metadata:
name: plex
namespace: media
labels:
app: plex
spec:
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
app: plex
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
template:
metadata:
labels:
app: plex
spec:
restartPolicy: Always
volumes:
- name: plex-config
persistentVolumeClaim:
claimName: plex-config
- name: plex-data
persistentVolumeClaim:
claimName: plex-data
containers:
- name: plex
image: ghcr.io/linuxserver/plex
imagePullPolicy: Always
env:
- name: PLEX_CLAIM
value: <redacted>
- name: PGID
value: '0'
- name: PUID
value: '0'
- name: VERSION
value: docker
- name: TZ
value: America/New_York
ports:
- containerPort: 32400
name: pms-web
protocol: TCP
- containerPort: 32469
name: dlna-tcp
protocol: TCP
- containerPort: 1900
name: dlna-udp
protocol: UDP
- containerPort: 3005
name: plex-companion
protocol: TCP
- containerPort: 5353
name: discovery-udp
protocol: UDP
- containerPort: 32410
name: gdm-32410
protocol: UDP
- containerPort: 32412
name: gdm-32412
protocol: UDP
- containerPort: 32413
name: gdm-32413
protocol: UDP
- containerPort: 32414
name: gdm-32414
protocol: UDP
volumeMounts:
- mountPath: /config
name: plex-config
- mountPath: /data
name: plex-data
then:
/config dir/config directoryThe output of the following commands will help us better understand what's going on:
kc get pods -n openebs --show-labelsNAME READY STATUS RESTARTS AGE LABELS
cstor-csi-node-vtf87 2/2 Running 9 (4d8h ago) 10d chart=cstor-3.1.0,component=openebs-cstor-csi-node,controller-revision-hash=6d59fbb8f8,heritage=Helm,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.1.0,pod-template-generation=1,release=cstor
cstor-cvc-operator-6f57946444-m5dmq 1/1 Running 1 (4d8h ago) 9d chart=cstor-3.1.0,component=cvc-operator,heritage=Helm,name=cvc-operator,openebs.io/component-name=cvc-operator,openebs.io/version=3.1.0,pod-template-hash=6f57946444,release=cstor
cstor-openebs-ndm-operator-55b6c8846b-njqvc 1/1 Running 4 (4d8h ago) 10d app=openebs-ndm-operator,chart=openebs-ndm-1.8.0,component=openebs-ndm-operator,heritage=Helm,name=openebs-ndm-operator,openebs.io/component-name=openebs-ndm-operator,openebs.io/version=1.8.0,pod-template-hash=55b6c8846b,release=cstor
cstor-openebs-ndm-rcgks 1/1 Running 6 (4d8h ago) 10d app=openebs-ndm,chart=openebs-ndm-1.8.0,component=ndm,controller-revision-hash=8c8486ccd,heritage=Helm,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=1.8.0,pod-template-generation=1,release=cstor
cstor-storage-jlsq-c57dd5d9b-56c5p 3/3 Running 0 4d8h app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-jlsq,openebs.io/version=3.1.0,pod-template-hash=c57dd5d9b
nfs-pvc-9b5e65a0-535e-48b6-8d8b-8dec93a32ec4-5db44b59c4-bdj5v 1/1 Running 0 3h48m openebs.io/nfs-server=nfs-pvc-9b5e65a0-535e-48b6-8d8b-8dec93a32ec4,pod-template-hash=5db44b59c4
cstor-csi-controller-0 6/6 Running 39 (161m ago) 10d chart=cstor-3.1.0,component=openebs-cstor-csi-controller,controller-revision-hash=cstor-csi-controller-867c797bd7,heritage=Helm,name=openebs-cstor-csi-controller,openebs.io/component-name=openebs-cstor-csi-controller,openebs.io/version=3.1.0,release=cstor,statefulset.kubernetes.io/pod-name=cstor-csi-controller-0
cstor-csi-node-2pvs2 2/2 Running 4 (9d ago) 10d chart=cstor-3.1.0,component=openebs-cstor-csi-node,controller-revision-hash=6d59fbb8f8,heritage=Helm,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.1.0,pod-template-generation=1,release=cstor
cstor-storage-gv6b-5d48f7854d-hpkp4 3/3 Running 1 (3d21h ago) 4d8h app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-gv6b,openebs.io/version=3.1.0,pod-template-hash=5d48f7854d
cstor-cspc-operator-cf87fdc89-zpnll 1/1 Running 2 (9d ago) 10d chart=cstor-3.1.0,component=cspc-operator,heritage=Helm,name=cspc-operator,openebs.io/component-name=cspc-operator,openebs.io/version=3.1.0,pod-template-hash=cf87fdc89,release=cstor
cstor-admission-server-77c48bb8c-xvxzw 1/1 Running 0 9d app=cstor-admission-webhook,chart=cstor-3.1.0,component=cstor-admission-webhook,heritage=Helm,openebs.io/component-name=cstor-admission-webhook,openebs.io/version=3.1.0,pod-template-hash=77c48bb8c,release=cstor
cstor-openebs-ndm-pbb6b 1/1 Running 2 (9d ago) 10d app=openebs-ndm,chart=openebs-ndm-1.8.0,component=ndm,controller-revision-hash=8c8486ccd,heritage=Helm,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=1.8.0,pod-template-generation=1,release=cstor
openebs-nfs-nfs-provisioner-7884cc9556-4kdds 1/1 Running 4 (150m ago) 3h48m app=nfs-provisioner,chart=nfs-provisioner-0.9.0,component=nfs-provisioner,heritage=Helm,name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=0.9.0,pod-template-hash=7884cc9556,release=openebs-nfs
cstor-storage-8cqv-ffbcbb6c7-78shd 3/3 Running 0 3h48m app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-8cqv,openebs.io/version=3.1.0,pod-template-hash=ffbcbb6c7
cstor-openebs-ndm-rfgbz 1/1 Running 5 (4d8h ago) 10d app=openebs-ndm,chart=openebs-ndm-1.8.0,component=ndm,controller-revision-hash=8c8486ccd,heritage=Helm,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=1.8.0,pod-template-generation=1,release=cstor
cstor-csi-node-w5hq7 2/2 Running 10 (4d8h ago) 10d chart=cstor-3.1.0,component=openebs-cstor-csi-node,controller-revision-hash=6d59fbb8f8,heritage=Helm,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.1.0,pod-template-generation=1,release=cstor
nfs-pvc-d7e6f64d-68d6-412b-98c0-f0a294508794-5ddb76577d-8szp9 1/1 Running 0 162m openebs.io/nfs-server=nfs-pvc-d7e6f64d-68d6-412b-98c0-f0a294508794,pod-template-hash=5ddb76577d
nfs-pvc-55940b24-bd9a-4d63-911c-809f9bfc10cd-587c7fdccd-r5gkx 1/1 Running 0 162m openebs.io/nfs-server=nfs-pvc-55940b24-bd9a-4d63-911c-809f9bfc10cd,pod-template-hash=587c7fdccd
pvc-f2997e87-c209-4200-baa7-3a520c2211e1-target-b5bb994c9-x5vzq 3/3 Running 0 149m app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=nfs-pvc-d7e6f64d-68d6-412b-98c0-f0a294508794,openebs.io/persistent-volume=pvc-f2997e87-c209-4200-baa7-3a520c2211e1,openebs.io/target=cstor-target,openebs.io/version=3.1.0,pod-template-hash=b5bb994c9
pvc-24b04637-8621-48d9-beca-211b15ccc675-target-7b58f96b5ffkvrw 3/3 Running 0 149m app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=nfs-pvc-9b5e65a0-535e-48b6-8d8b-8dec93a32ec4,openebs.io/persistent-volume=pvc-24b04637-8621-48d9-beca-211b15ccc675,openebs.io/target=cstor-target,openebs.io/version=3.1.0,pod-template-hash=7b58f96b5f
pvc-4a009034-ff10-453a-a3f7-049b8a219ff0-target-58cb6d448dmg2pm 3/3 Running 0 149m app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=nfs-pvc-55940b24-bd9a-4d63-911c-809f9bfc10cd,openebs.io/persistent-volume=pvc-4a009034-ff10-453a-a3f7-049b8a219ff0,openebs.io/target=cstor-target,openebs.io/version=3.1.0,pod-template-hash=58cb6d448d
kubectl get pvc -n openebsNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc-9b5e65a0-535e-48b6-8d8b-8dec93a32ec4 Bound pvc-24b04637-8621-48d9-beca-211b15ccc675 500Mi RWO cstor-csi 24h
nfs-pvc-d7e6f64d-68d6-412b-98c0-f0a294508794 Bound pvc-f2997e87-c209-4200-baa7-3a520c2211e1 500Gi RWO cstor-csi 163m
nfs-pvc-55940b24-bd9a-4d63-911c-809f9bfc10cd Bound pvc-4a009034-ff10-453a-a3f7-049b8a219ff0 1Gi RWO cstor-csi 163m
kubectl describe pvc -n media479s 3:48:34 ~/code/personal/personalโฉ kc describe pvc -n media ยฑ master
Name: plex-data
Namespace: media
StorageClass: openebs-nfs
Status: Bound
Volume: pvc-d7e6f64d-68d6-412b-98c0-f0a294508794
Labels: app=plex
kustomize.toolkit.fluxcd.io/name=media
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: openebs.io/nfsrwx
volume.kubernetes.io/storage-provisioner: openebs.io/nfsrwx
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 500Gi
Access Modes: RWX
VolumeMode: Filesystem
Used By: plex-85556c488b-8s9lv
Events: <none>
Name: plex-config
Namespace: media
StorageClass: openebs-nfs
Status: Bound
Volume: pvc-55940b24-bd9a-4d63-911c-809f9bfc10cd
Labels: app=plex
kustomize.toolkit.fluxcd.io/name=media
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: openebs.io/nfsrwx
volume.kubernetes.io/storage-provisioner: openebs.io/nfsrwx
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWX
VolumeMode: Filesystem
Used By: plex-85556c488b-8s9lv
Events: <none>
root@plex-85556c488b-8s9lv:/# ls -l /configls: reading directory '/config': Input/output error
total 0
kubectl logs -n media plex-85556c488b-8s9lv[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing...
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 01-migrations: executing...
[migrations] started
[migrations] no migrations found
[cont-init.d] 01-migrations: exited 0.
[cont-init.d] 02-tamper-check: executing...
[cont-init.d] 02-tamper-check: exited 0.
[cont-init.d] 10-adduser: executing...
-------------------------------------
_ ()
| | ___ _ __
| | / __| | | / \
| | \__ \ | | | () |
|_| |___/ |_| \__/
Brought to you by linuxserver.io
-------------------------------------
To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------
User uid: 0
User gid: 0
-------------------------------------
chown: changing ownership of '/config': Read-only file system
[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 40-chown-files: executing...
chown: changing ownership of '/config': Read-only file system
chown: cannot access '/config/*': Input/output error
[cont-init.d] 40-chown-files: exited 1.
[cont-init.d] 45-plex-claim: executing...
Temporarily starting Plex Media Server.
Waiting for Plex to generate its config
Waiting for database creation to complete...
md5sum: '/config/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-wal': Input/output error
PMS: failure detected. Read/write access is required for path: /config/Library/Application Support/Plex Media Server
Waiting for database creation to complete...
md5sum: '/config/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-wal': Input/output error
Anything else we need to know?:
any insights would be greatly appreciated this is blocking my personal on prem plex media server journey!
Environment details:
on prem cluster:
kubectl versionClient Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:17:57Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.3-2+0d2db09fa6fbbb", GitCommit:"0d2db09fa6fbbbf8e7fd3e4c9d9e1c4b29f66481", GitTreeState:"clean", BuildDate:"2022-01-26T22:17:11Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/arm64"}
cat /etc/os-release in containerNAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
uname -aLinux plex-5cd676cb7d-jsjk2 5.4.0-1052-raspi #58-Ubuntu SMP PREEMPT Mon Feb 7 16:52:35 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Describe the bug: Shortly after installing the NFS provisioner, the openebs-nfs-provisioner pod starts failing the liveness probe and is repeatedly restarted by Kubernetes.
Expected behaviour: The openebs-nfs-provisioner pod doesn't fail the liveness probe after being created.
Steps to reproduce the bug:
kubectl apply -f deploy/kubectl/openebs-nfs-provisioner.yamlAnything else we need to know?:
$ kubectl describe pod/openebs-nfs-provisioner-597b5df87b-cvgk9 -n openebs
Name: openebs-nfs-provisioner-597b5df87b-cvgk9
Namespace: openebs
Priority: 0
Node: node-2-1623/65.21.157.8
Start Time: Wed, 26 May 2021 23:55:05 -0700
Labels: name=openebs-nfs-provisioner
openebs.io/component-name=openebs-nfs-provisioner
openebs.io/version=dev
pod-template-hash=597b5df87b
Annotations: <none>
Status: Running
IP: 10.244.3.4
IPs:
IP: 10.244.3.4
Controlled By: ReplicaSet/openebs-nfs-provisioner-597b5df87b
Containers:
openebs-provisioner-nfs:
Container ID: docker://14eec501ecf2a0a998063abf5e619e75decdd04ac1626d5a9d0bac102b99a246
Image: openebs/provisioner-nfs:ci
Image ID: docker-pullable://openebs/provisioner-nfs@sha256:0fe40f6e0d6334f28b86db5ed8b0f9fb6fd356676b5f7b65285778be28b295b8
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 26 May 2021 23:59:00 -0700
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Wed, 26 May 2021 23:55:08 -0700
Finished: Wed, 26 May 2021 23:59:00 -0700
Ready: True
Restart Count: 1
Liveness: exec [sh -c test `pgrep -c "^provisioner-nfs.*"` = 1] delay=30s timeout=1s period=60s #success=1 #failure=3
Environment:
NODE_NAME: (v1:spec.nodeName)
OPENEBS_NAMESPACE: openebs (v1:metadata.namespace)
OPENEBS_SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
OPENEBS_IO_ENABLE_ANALYTICS: true
OPENEBS_IO_NFS_SERVER_USE_CLUSTERIP: true
OPENEBS_IO_INSTALLER_TYPE: openebs-operator-nfs
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from openebs-maya-operator-token-tv92s (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
openebs-maya-operator-token-tv92s:
Type: Secret (a volume populated by a Secret)
SecretName: openebs-maya-operator-token-tv92s
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m12s default-scheduler Successfully assigned openebs/openebs-nfs-provisioner-597b5df87b-cvgk9 to node-2-1623
Normal Pulling 5m12s kubelet Pulling image "openebs/provisioner-nfs:ci"
Normal Pulled 5m9s kubelet Successfully pulled image "openebs/provisioner-nfs:ci" in 2.504356904s
Normal Killing 107s kubelet Container openebs-provisioner-nfs failed liveness probe, will be restarted
Normal Created 77s (x2 over 5m9s) kubelet Created container openebs-provisioner-nfs
Normal Started 77s (x2 over 5m9s) kubelet Started container openebs-provisioner-nfs
Normal Pulled 77s kubelet Container image "openebs/provisioner-nfs:ci" already present on machine
Warning Unhealthy 47s (x4 over 3m47s) kubelet Liveness probe failed: pgrep: unrecognized option: c
BusyBox v1.31.1 () multi-call binary.
Usage: pgrep [-flanovx] [-s SID|-P PPID|PATTERN]
Display process(es) selected by regex PATTERN
-l Show command name too
-a Show command line too
-f Match against entire command line
-n Show the newest process only
-o Show the oldest process only
-v Negate the match
-x Match whole name (not substring)
-s Match session ID (0 for current)
-P Match parent process ID
sh: 1: unknown operand
$ kubectl get pods -n openebs --show-labels
NAME READY STATUS RESTARTS AGE LABELS
openebs-nfs-provisioner-597b5df87b-cvgk9 1/1 Running 1 6m1s name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=dev,pod-template-hash=597b5df87b
$ kubectl logs pod/openebs-nfs-provisioner-597b5df87b-cvgk9 -n openebs
I0527 07:11:00.662383 1 controller.go:39] Starting Provisioner...
I0527 07:11:00.673939 1 controller.go:101] Leader election enabled for nfs-provisioner
I0527 07:11:00.675240 1 leaderelection.go:242] attempting to acquire leader lease openebs/openebs.io-nfsrwx...
I0527 07:11:18.079309 1 leaderelection.go:252] successfully acquired lease openebs/openebs.io-nfsrwx
I0527 07:11:18.079509 1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"openebs", Name:"openebs.io-nfsrwx", UID:"58433b00-8898-4911-bc30-a7272e0f3966", APIVersion:"v1", ResourceVersion:"6408", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' openebs-nfs-provisioner-597b5df87b-cvgk9_23982c90-e22b-4689-9909-1f4410aa7c48 became leader
I0527 07:11:18.079533 1 controller.go:780] Starting provisioner controller openebs.io/nfsrwx_openebs-nfs-provisioner-597b5df87b-cvgk9_23982c90-e22b-4689-9909-1f4410aa7c48!
I0527 07:11:18.179790 1 controller.go:829] Started provisioner controller openebs.io/nfsrwx_openebs-nfs-provisioner-597b5df87b-cvgk9_23982c90-e22b-4689-9909-1f4410aa7c48!
$ kubectl logs --previous pod/openebs-nfs-provisioner-597b5df87b-cvgk9 -n openebs
I0527 07:08:00.618991 1 controller.go:39] Starting Provisioner...
I0527 07:08:00.631000 1 controller.go:101] Leader election enabled for nfs-provisioner
I0527 07:08:00.631419 1 leaderelection.go:242] attempting to acquire leader lease openebs/openebs.io-nfsrwx...
I0527 07:08:18.035729 1 leaderelection.go:252] successfully acquired lease openebs/openebs.io-nfsrwx
I0527 07:08:18.035903 1 controller.go:780] Starting provisioner controller openebs.io/nfsrwx_openebs-nfs-provisioner-597b5df87b-cvgk9_d9ce697e-1e67-41da-a6c8-80d2ba762425!
I0527 07:08:18.035932 1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"openebs", Name:"openebs.io-nfsrwx", UID:"58433b00-8898-4911-bc30-a7272e0f3966", APIVersion:"v1", ResourceVersion:"5875", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' openebs-nfs-provisioner-597b5df87b-cvgk9_d9ce697e-1e67-41da-a6c8-80d2ba762425 became leader
I0527 07:08:18.136159 1 controller.go:829] Started provisioner controller openebs.io/nfsrwx_openebs-nfs-provisioner-597b5df87b-cvgk9_d9ce697e-1e67-41da-a6c8-80d2ba762425!
I0527 07:10:30.443777 1 signal_handler.go:33] Receive terminated to exit
Environment details:
OpenEBS version: dev
Kubernetes version: (Client = v1.21.0) (Server = v1.19.11)
Cloud provider or hardware configuration: Run in Hetzner Cloud with mayastor-ci
OS (e.g: cat /etc/os-release): Ubuntu 20.04.2 LTS
kernel (e.g: uname -a):
Linux node-1-1623 5.8.0-53-generic #60~20.04.1-Ubuntu SMP Thu May 6 09:52:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
nfs-subdir-external-provisioner example๏ผ
nfs:
server: 10.0.0.0
path: /data/nfs/test
Describe the problem/challenge you have
Current there is a k8s bug:
If delete PV prior to deleting PVC, then the Reclaim policy is not exercised. As a result of this behavior, the associated storage asset in the external infrastructure is not removed.
For openebs nfs provisioner, that means the backend kinds of stuff (PV, PVC, deployment and service) are all left behind.
A feature gate HonorPVReclaimPolicy was added in kubernetes 1.23 to resolve such issue, refer to the KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2644-honor-pv-reclaim-policy
sig-storage-lib has already changed to support PV Deletion protection finalizer
kubernetes-sigs/sig-storage-lib-external-provisioner#117
Hope openebs can support this KEP.
Describe the solution you'd like
I think setting AddFinalizer to true when instantiating the nfs provisioner controller and upgrading the sig-storage library from v7 to v8 should be fine.
https://github.com/openebs/dynamic-nfs-provisioner/blob/1f414468b8c53b974c5dddbe02357a6ceb97d3b9/provisioner/controller.go#L70
Environment:
kubectl get po -n openebs --show-labels): v0.9.0kubectl version): v1.23.6.Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the bug:
Installing the openebs-nfs-provisioner breaks microk8s with an error in starting the kubelite which reads as "listen to 172.30.39.42:19001: listen tcp 172.30.39.42:19001: bind: cannot assign requested address"
Steps to reproduce the bug:
microk8s.add-nodemicrok8s.enable openebs
microk8s.kubectl apply -f https://raw.githubusercontent.com/openebs/dynamic-nfs-provisioner/develop/deploy/kubectl/openebs-nfs-provisioner.yaml kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: openebs-rwx-pvc
spec:
storageClassName: openebs-rwx
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: service1
spec:
containers:
- name: service1
image: bashell/alpine-bash
command: ["/bin/bash"]
args: ["-c", "while true ;do sleep 50; done"]
volumeMounts:
- mountPath: /data
name: data-volume
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: openebs-rwx-pvc
nodeName: node1
---
apiVersion: v1
kind: Pod
metadata:
name: service2
spec:
containers:
- name: service2
image: bashell/alpine-bash
command: ["/bin/bash"]
args: ["-c", "while true ;do sleep 50; done"]
volumeMounts:
- mountPath: /data
name: data-volume
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: openebs-rwx-pvc
nodeName: node2
---
apiVersion: v1
kind: Pod
metadata:
name: service3
spec:
containers:
- name: service3
image: bashell/alpine-bash
command: ["/bin/bash"]
args: ["-c", "while true ;do sleep 50; done"]
volumeMounts:
- mountPath: /data
name: data-volume
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: openebs-rwx-pvc
nodeName: node3I'm using multipass with hyper-v on Windows 11.
The log from microk8s.inspect:
inspection-report-20210809_162756.tar.gz
Describe the bug:
Go Report Card found 7 issues
Expected behaviour:
Result of the Go Report Card should be 100%
Steps to reproduce the bug:
https://goreportcard.com/report/github.com/openebs/dynamic-nfs-provisioner
The output of the following Go Report Card:
https://pastebin.com/rSdqJt5g
Environment details:
Go Report Card
When the nfs pod went down, the recovery time for NFS to be active depends on the timeout of the K8s to determine that it is down and reschedule taking ~2 minutes.
One option is to reduce by using custom tolerations like below:
tolerations:
- effect: NoExecute
key: node.alpha.kubernetes.io/notReady
operator: Exists
tolerationSeconds: 0
- effect: NoExecute
key: node.alpha.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 0
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 0
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 0
Also, check for pod termination or restart strategies to make it faster.
Describe the bug:
I created many RWX nfs shares with nfs-provisioner. As backend storageclass i use openebs-jiva.
Sometimes every nfs share mount get the permission 000
Then, the mount folder looks like
root@nfs-pvc-3da88edc-2b97-4165-93ba-49bc54056cc6-fb8f5fd66-jfz6b:/ # ls -la
d--------- 17 xfs xfs 4096 Oct 29 14:15 nfsshare
in the nfs-pvc pod.
Cause of that, the nginx container where the nfs-pvc is mounted also have 000 on the folder and cannot read the files within
The files in the mount folder have the correct permissions.
Expected behaviour:
Default mount permission 755 or something similar
Steps to reproduce the bug:
Just create a new nfs rwx pvc and wait. After some time the running nginx container cannot read the folder anymore.
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelskubectl get pvc -n <openebs_namespace>kubectl get pvc -n <application_namespace>Anything else we need to know?:
jiva and nfs installed via helm charts
https://github.com/openebs/jiva-operator/tree/develop/deploy/helm/charts
https://github.com/openebs/dynamic-nfs-provisioner/tree/develop/deploy/helm/charts
helm config:
nfs-provisioner:
rbac:
pspEnabled: false
podSecurityContext:
fsGroup: 120
storage class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-rwx
annotations:
openebs.io/cas-type: nfsrwx
cas.openebs.io/config: |
- name: NFSServerType
value: "kernel"
- name: BackendStorageClass
value: "openebs-jiva-csi-default"
# LeaseTime defines the renewal period(in seconds) for client state
- name: LeaseTime
value: 30
# GraceTime defines the recovery period(in seconds) to reclaim locks
- name: GraceTime
value: 30
# FSGID defines the group permissions of NFS Volume. If it is set
# then non-root applications should add FSGID value under pod
# Supplemental groups
- name: FSGID
value: "120"
- name: NFSServerResourceRequests
value: |-
cpu: 50m
memory: 50Mi
- name: NFSServerResourceLimits
value: |-
cpu: 100m
memory: 100Mi
provisioner: openebs.io/nfsrwx
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
Environment details:
kubectl get po -n openebs --show-labels):openebs/jiva-operator:3.0.0
openebs/provisioner-nfs:0.7.1
kubectl version):v1.21.5+k3scat /etc/os-release):AlmaLinux 8.4 (Electric Cheetah)uname -a):4.18.0-305.19.1.el8_4.x86_64Do i have a misconfigured setup or is this a bug?
Thanks for help!
As of now dynamic-nfs-provisioner doen't have support to expose metrics about nfs stats. This issue is to add the metrics for better visibility and monitoring of nfs volumes.
We should add a step to the installation instructions about installing the nfs-common package (for Ubuntu and Debian) (alternatively the nfs-utils package on CentOS and Fedora) on the worker nodes.
Help: How to delete these PVC and PV๏ผ
PV and PVC is dynamic-nfs-provisioner create
Describe the problem/challenge you have
While provisioning nfs volume, nfs-provisioner starts nfs-server container with default nfs configuration, i.e lease time, grace period. This parameter can be useful for users to speed up recovery, in case of pod restart, and can be set accordingly.
Describe the solution you'd like
Provisioner can use sc to fetch this parameter to configure nfs-server.
Anything else you would like to add:
Environment:
kubectl get po -n openebs --show-labels):kubectl version):cat /etc/os-release):uname -a):Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the bug: Mounting NFS volume fails with failed to resolve server
MountVolume.SetUp failed for volume "pvc-a819b066-8c44-4bbe-b5fc-f4574002c112" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112.openebs.svc.cluster.local:/ /var/lib/kubelet/pods/3e5b9e15-051d-4d3c-a2cc-9725f343c292/volumes/kubernetes.io~nfs/pvc-a819b066-8c44-4bbe-b5fc-f4574002c112 Output: mount.nfs: Failed to resolve server nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112.openebs.svc.cluster.local: Name or service not known
Expected behaviour: Mounting NFS volumes to work
Steps to reproduce the bug:
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelsNAME READY STATUS RESTARTS AGE LABELS
nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112-59bcd8bcbc-t8vxj 1/1 Running 0 115s openebs.io/nfs-server=nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112,pod-template-hash=59bcd8bcbc
openebs-admission-server-7f75c6b458-lklzw 1/1 Running 0 2m12s app=admission-webhook,name=admission-webhook,openebs.io/component-name=admission-webhook,openebs.io/version=2.5.0,pod-template-hash=7f75c6b458,release=openebs
openebs-apiserver-8d6fc5ff6-q5zsn 1/1 Running 0 2m12s app=openebs,component=apiserver,name=maya-apiserver,openebs.io/component-name=maya-apiserver,openebs.io/version=2.5.0,pod-template-hash=8d6fc5ff6,release=openebs
openebs-localpv-provisioner-566db58b6d-rhc5r 1/1 Running 0 2m12s app=openebs,component=localpv-provisioner,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=2.5.0,pod-template-hash=566db58b6d,release=openebs
openebs-ndm-gw97p 1/1 Running 0 2m12s app=openebs,component=ndm,controller-revision-hash=c9dd4d6d6,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.5.0,pod-template-generation=1,release=openebs
openebs-ndm-hg4jp 1/1 Running 0 2m12s app=openebs,component=ndm,controller-revision-hash=c9dd4d6d6,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.5.0,pod-template-generation=1,release=openebs
openebs-ndm-operator-7686d7b98-h995s 1/1 Running 0 2m12s app=openebs,component=ndm-operator,name=ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=2.5.0,pod-template-hash=7686d7b98,release=openebs
openebs-ndm-wv5j5 1/1 Running 0 2m12s app=openebs,component=ndm,controller-revision-hash=c9dd4d6d6,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.5.0,pod-template-generation=1,release=openebs
openebs-nfs-provisioner-64464568bf-9sfvl 1/1 Running 0 2m13s name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=dev,pod-template-hash=64464568bf
openebs-provisioner-5fdf9c6f78-c4fhc 1/1 Running 0 2m12s app=openebs,component=provisioner,name=openebs-provisioner,openebs.io/component-name=openebs-provisioner,openebs.io/version=2.5.0,pod-template-hash=5fdf9c6f78,release=openebs
openebs-snapshot-operator-6c465c75c8-fmlhp 2/2 Running 0 2m12s app=openebs,component=snapshot-operator,name=openebs-snapshot-operator,openebs.io/component-name=openebs-snapshot-operator,openebs.io/version=2.5.0,pod-template-hash=6c465c75c8,release=openebs
pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac-ctrl-5684fc4b84-t8kbs 2/2 Running 0 2m5s monitoring=volume_exporter_prometheus,openebs.io/controller=jiva-controller,openebs.io/persistent-volume-claim=data-wwwdev-mariadb-0,openebs.io/persistent-volume=pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac,openebs.io/version=2.5.0,pod-template-hash=5684fc4b84
pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac-rep-1-77698dccc9-8znd4 1/1 Running 0 117s openebs.io/persistent-volume-claim=data-wwwdev-mariadb-0,openebs.io/persistent-volume=pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac,openebs.io/replica=jiva-replica,openebs.io/version=2.5.0,pod-template-hash=77698dccc9
pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac-rep-2-74dc67b4f7-42g2x 1/1 Running 0 2m2s openebs.io/persistent-volume-claim=data-wwwdev-mariadb-0,openebs.io/persistent-volume=pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac,openebs.io/replica=jiva-replica,openebs.io/version=2.5.0,pod-template-hash=74dc67b4f7
pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac-rep-3-6cdfb5bc6b-r8vgs 1/1 Running 0 2m2s openebs.io/persistent-volume-claim=data-wwwdev-mariadb-0,openebs.io/persistent-volume=pvc-89f43333-50b6-423d-8f23-19a4b9f9f7ac,openebs.io/replica=jiva-replica,openebs.io/version=2.5.0,pod-template-hash=6cdfb5bc6b
pvc-ae705306-f540-4a92-b77b-a09288106768-ctrl-7697cc8d8c-r66t7 2/2 Running 0 115s monitoring=volume_exporter_prometheus,openebs.io/controller=jiva-controller,openebs.io/persistent-volume-claim=nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112,openebs.io/persistent-volume=pvc-ae705306-f540-4a92-b77b-a09288106768,openebs.io/version=2.5.0,pod-template-hash=7697cc8d8c
pvc-ae705306-f540-4a92-b77b-a09288106768-rep-1-69ccf8766d-5hdqd 1/1 Running 1 112s openebs.io/persistent-volume-claim=nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112,openebs.io/persistent-volume=pvc-ae705306-f540-4a92-b77b-a09288106768,openebs.io/replica=jiva-replica,openebs.io/version=2.5.0,pod-template-hash=69ccf8766d
pvc-ae705306-f540-4a92-b77b-a09288106768-rep-2-db79dc6d7-m5rcw 1/1 Running 0 113s openebs.io/persistent-volume-claim=nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112,openebs.io/persistent-volume=pvc-ae705306-f540-4a92-b77b-a09288106768,openebs.io/replica=jiva-replica,openebs.io/version=2.5.0,pod-template-hash=db79dc6d7
pvc-ae705306-f540-4a92-b77b-a09288106768-rep-3-5549b59f4-xhjbj 1/1 Running 0 107s openebs.io/persistent-volume-claim=nfs-pvc-a819b066-8c44-4bbe-b5fc-f4574002c112,openebs.io/persistent-volume=pvc-ae705306-f540-4a92-b77b-a09288106768,openebs.io/replica=jiva-replica,openebs.io/version=2.5.0,pod-template-hash=5549b59f4
kubectl logs <upgrade_job_pod> -n <openebs_namespace>Anything else we need to know?:
This is a kubernetes cluster running rancher installed by RKE.
Environment details:
kubectl get po -n openebs --show-labels): 2.5.0 installed by the 2.5.1 helm chart.Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:22:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
cat /etc/os-release): CentOS Linux release 7.9.2009 (Core)uname -a): 3.10.0-1160.6.1.el7.x86_64Describe the bug: A clear and concise description of what the bug is.
When the frontend PV and PVC are deleted, openebs nfs provisioner will do gc to remove the stale backend NFS resources every 5 mins.
Refer to PR: #80
After adding debug log, found that when doing gc, the p.hook is undefined, so it did not enter the below brace to remove the finalizer, and the service, deployment and PVC are not deleted.
https://github.com/openebs/dynamic-nfs-provisioner/blob/dbe24dc5607f1c187ff80af3236034061ee99e00/provisioner/helper_kernel_nfs_server.go#L526
From openebs-nfs-provisioner log, we can see the gc was triggered, and describe the service, deployment and backend PVC, there were deletingTimestamp in the manifest.
I0829 12:11:23.270146 1 garbage_collector.go:119] Deleting stale resources for PV=pvc-71f5a361-47fa-442d-baf1-00865d7567d6 I0829 12:11:23.272453 1 helper_kernel_nfs_server.go:522] Volume pvc-71f5a361-47fa-442d-baf1-00865d7567d6 has been initialized with Service {kube-system/nfs-pvc-71f5a361-47fa-442d-baf1-00865d7567d6}. Initiating delete... I0829 12:11:23.301297 1 helper_kernel_nfs_server.go:403] Volume pvc-71f5a361-47fa-442d-baf1-00865d7567d6 has been initialized with Deployment:nfs-pvc-71f5a361-47fa-442d-baf1-00865d7567d6. Initiating delete... I0829 12:11:23.313493 1 helper_kernel_nfs_server.go:211] Volume pvc-71f5a361-47fa-442d-baf1-00865d7567d6 has been initialized with PVC {kube-system/nfs-pvc-71f5a361-47fa-442d-baf1-00865d7567d6} Initiating delete...
From this code in garbage_collector.go, no hook initialized
https://github.com/openebs/dynamic-nfs-provisioner/blob/1f414468b8c53b974c5dddbe02357a6ceb97d3b9/provisioner/garbage_collector.go#L120
I added a fix like the below and built a new image. Verified that the service, deployment, and PVC can be deleted in GC.
Environment details:
kubectl get po -n openebs --show-labels): v0.9.0kubectl version): 1.22.4Describe the problem/challenge you have
I've deployed the provisioner into my cluster and created an NFS share. I've edited my deployment to provide 2 replicas of the provisioner for maximum resilience. When I then provision an NFS share, the replica set (created by the deployment) to manage the share only has 1 instance which causes two problems for me:
Describe the solution you'd like
I appreciate there may be really good technical reasons (kernel NFS) as to why it isn't possible, but I may as well ask, would it be possible to create a replica set that matches the number of provisioned replicas? e.g.: If I have configured 2 provisioners, each replica set created by the provisions would have 2 replicas.
Anything else you would like to add:
N/A
Environment:
kubectl get po -n openebs --show-labels):kubectl version):cat /etc/os-release):uname -a):Describe the bug: A clear and concise description of what the bug is.
Expected behaviour: A concise description of what you expected to happen
Steps to reproduce the bug:
Steps to reproduce the bug should be clear and easily reproducible to help people gain an understanding of the problem
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelskubectl get pvc -n <openebs_namespace>kubectl get pvc -n <application_namespace>Anything else we need to know?:
Add any other context about the problem here.
Environment details:
kubectl get po -n openebs --show-labels):kubectl version):cat /etc/os-release):uname -a):Hello Team,
Getting following error in the logs when using openebs velero plugin for taking volume snapshots.
--------------------------
time="2023-05-16T07:49:13Z" level=info msg="Initializing velero plugin for CStor" backup=velero/backuptwo-16052023 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/snapshot/snap.go:36" pluginName=velero-blockstore-openebs
time="2023-05-16T07:49:13Z" level=error msg="Error getting volume snapshotter for volume snapshot location" backup=velero/backuptwo-16052023 error="rpc error: code = Unknown desc = failed to get address for maya-apiserver/cvc-server service" error.file="/go/src/github.com/openebs/velero-plugin/pkg/cstor/cstor.go:259" error.function="github.com/openebs/velero-plugin/pkg/cstor.(*Plugin).Init" logSource="pkg/backup/item_backupper.go:524" name=pvc-xxxx-xxxx-xxxx namespace= persistentVolume=pvc-xxxx-xxxx-xxxx resource=persistentvolumes volumeSnapshotLocation=cstor-sp-velero
time="2023-05-16T07:49:13Z" level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/backuptwo-16052023 logSource="pkg/backup/item_backupper.go:544" name=pvc-xxxx-xxxx-xxxx namespace= persistentVolume=pvc-xxxx-xxxx-xxxx resource=persistentvolumes
------------------------
Any idea how to fix this is much appreciated.
Describe the bug: A clear and concise description of what the bug is.
Sometimes when removing a Pod, which is mounted with a NFS PV, with the corresponding NFS PVC/PV simultaneously, both the Pod/PVC/PV and the backend NFS Deployment/Service/PVC/PV are cleaned so fast that the kubelet on the worker node where the pod used to run can not unmount the NFS volume in time. This makes the remaining NFS volume on the worker node stale and won't be unmounted unless manually doing so. But the IO process will be blocked there forever until rebooting the node.
It's weird though that the Pod object is successfully removed from the cluster even without kubelet completing cleaning mount on the node.
Expected behaviour: A concise description of what you expected to happen
The NFS volume mounted on the worker node is cleaned up.
Steps to reproduce the bug:
Steps to reproduce the bug should be clear and easily reproducible to help people gain an understanding of the problem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: openebs-nfs
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
storageClassName: network-file # This is the SC name related to the openebs-nfs-provisioner
---
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: sleep
name: sleep
spec:
containers:
- image: nginx
name: sleep
resources: {}
volumeMounts:
- name: openebs-nfs
mountPath: /mnt
dnsPolicy: ClusterFirst
terminationGracePeriodSeconds: 0 # intentionally set this to 0
restartPolicy: Always
volumes:
- name: openebs-nfs
persistentVolumeClaim:
claimName: openebs-nfs
status: {}
Set the terminationGracePeriodSeconds to 0 so the pod can be quickly removed when deleting it.
kubectl -n kube-system get all | grep nfs-pvc
pod/nfs-pvc-9226622c-10b0-4b1d-8d4d-5661c6fec8e3-7cfc9fdc76-x6746 1/1 Running 0 97s
service/nfs-pvc-9226622c-10b0-4b1d-8d4d-5661c6fec8e3 ClusterIP 10.105.148.166 <none> 2049/TCP,111/TCP 96s
deployment.apps/nfs-pvc-9226622c-10b0-4b1d-8d4d-5661c6fec8e3 1/1 1 1 97s
replicaset.apps/nfs-pvc-9226622c-10b0-4b1d-8d4d-5661c6fec8e3-7cfc9fdc76 1 1 1 97s
kubectl get po - o wide to get the node where the Pod is runningkubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
sleep 1/1 Running 0 2m11s 192.168.171.133 node-10-158-36-65 <none> <none>
kubectl delete -f <path_file_of_above_content>kubectl delete -f pod.yml
persistentvolumeclaim "openebs-nfs" deleted
pod "sleep" deleted
kubectl's view will be successfully removedkubectl get po
No resources found in default namespace.
kubectl -n kube-system get all | grep nfs-pvc
df -h which will get stuck. Then via mount will see the NFS volume is leftover# ssh to the node
mount | grep nfs
10.105.148.166:/ on /var/lib/kubelet/pods/947b2765-78f0-4908-8856-5fe09269999e/volumes/kubernetes.io~nfs/pvc-9226622c-10b0-4b1d-8d4d-5661c6fec8e3 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.158.36.65,local_lock=none,addr=10.105.148.166)
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelskubectl get pvc -n <openebs_namespace>kubectl get pvc -n <application_namespace>Anything else we need to know?:
Add any other context about the problem here.
Environment details:
kubectl get po -n openebs --show-labels):v0.9.0
kubectl version):Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"49499222b0eb0349359881bea01d8d5bd78bf444", GitTreeState:"clean", BuildDate:"2021-12-14T12:41:40Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
cat /etc/os-release):NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp3"
DOCUMENTATION_URL="https://documentation.suse.com/"
uname -a):Linux node-10-158-36-65 5.3.18-57-default #1 SMP Wed Apr 28 10:54:41 UTC 2021 (ba3c2e9) x86_64 x86_64 x86_64 GNU/Linux
The backend storage is Ceph CSI RBD.
StorageClass:
k get sc network-file -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
cas.openebs.io/config: |
- name: NFSServerType
value: kernel
- name: BackendStorageClass
value: network-block
- name: LeaseTime
value: 30
- name: GraceTime
value: 30
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"cas.openebs.io/config":"- name: NFSServerType\n value: kernel\n- name: BackendStorageClass\n value: network-block\n- name: LeaseTime\n value: 30\n- name: GraceTime\n value: 30\n","openebs.io/cas-type":"nfsrwx"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile"},"name":"network-file"},"provisioner":"openebs.io/nfsrwx","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
openebs.io/cas-type: nfsrwx
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2022-05-16T21:12:31Z"
labels:
addonmanager.kubernetes.io/mode: Reconcile
name: network-file
resourceVersion: "3104"
selfLink: /apis/storage.k8s.io/v1/storageclasses/network-file
uid: 1a02778d-391f-4e70-a9f1-cd3c7ad230da
provisioner: openebs.io/nfsrwx
reclaimPolicy: Delete
volumeBindingMode: Immediate
As of now when nfs-volume is provisioned provisioner will create NFS-server deployment and scheduling is limited to handover to backend-storage class topology and K8s scheduler.
Supporting an option to specify node information in NFS StorageClass (or) Provisioner will help to schedule NFS-Server on specific nodes.
Note:
For Mac M1, I am trying to dynamically provision a PV for my minikube cluster. I am following this docs https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/intro.md I have storage class, pvc in my cluster. But, pv is not dynamically getting provisioned. Did anyone saw this behaviour before?
โ ebs-nfs-provisioner k get sc | grep openebs-rwx
openebs-rwx openebs.io/nfsrwx Delete Immediate false 13m
โ ebs-nfs-provisioner k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc Pending openebs-rwx 10m
โ ebs-nfs-provisioner k describe sc openebs-rwx
Name: openebs-rwx
IsDefaultClass: No
Annotations: cas.openebs.io/config=- name: NFSServerType
value: "kernel"
- name: BackendStorageClass
value: "openebs-hostpath"
# LeaseTime defines the renewal period(in seconds) for client state
#- name: LeaseTime
# value: 30
# GraceTime defines the recovery period(in seconds) to reclaim locks
#- name: GraceTime
# value: 30
# FSGID defines the group permissions of NFS Volume. If it is set
# then non-root applications should add FSGID value under pod
# Supplemental groups
#- name: FSGID
# value: "120"
,kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"cas.openebs.io/config":"- name: NFSServerType\n value: \"kernel\"\n- name: BackendStorageClass\n value: \"openebs-hostpath\"\n# LeaseTime defines the renewal period(in seconds) for client state\n#- name: LeaseTime\n# value: 30\n# GraceTime defines the recovery period(in seconds) to reclaim locks\n#- name: GraceTime\n# value: 30\n# FSGID defines the group permissions of NFS Volume. If it is set\n# then non-root applications should add FSGID value under pod\n# Supplemental groups\n#- name: FSGID\n# value: \"120\"\n","openebs.io/cas-type":"nfsrwx"},"name":"openebs-rwx"},"provisioner":"openebs.io/nfsrwx","reclaimPolicy":"Delete"}
,openebs.io/cas-type=nfsrwx
Provisioner: openebs.io/nfsrwx
Parameters: <none>
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
โ ebs-nfs-provisioner k describe pvc nfs-pvc
Name: nfs-pvc
Namespace: default
StorageClass: openebs-rwx
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: openebs.io/nfsrwx
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 45s (x42 over 10m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openebs.io/nfsrwx" or manually created by system administrator
Add documentation and helm changes that are required to configure resource
requirements on NFS Server deployment(during provisioning time) as part of #92
Describe the problem/challenge you have
I want to pull the NFS Server container image from a private container registry, but cannot do it because the image pull secret used by the provisioner is not passed along to the NFS servers it creates.
Describe the solution you'd like
I would like an additional value in the Helm chart which allows me to specify an image pull secret to be used by the NFS servers, so that the NFS servers in Kubernetes can use container images that are from a private registry.
Anything else you would like to add:
I will begin working on this myself, but thought I should document it here.
Environment:
kubectl get po -n openebs --show-labels): 0.7.1kubectl version): 1.21.4cat /etc/os-release): Pop!_OS 21.04uname -a): 5.13.0-7614-genericVote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the bug: A clear and concise description of what the bug is.
Any container fails to start, when using the nfs provisioner
Expected behaviour: A concise description of what you expected to happen
Container starts and nfs storage is provisioned
Steps to reproduce the bug:
mkdir -p /mnt/datasudo chown -R admin:admin /mnt/datakubectl apply -f https://gist.github.com/tampler/44baacbd7606561a11cf949fa462ceddkubectl get pvc -> find the pvc created: ebs-rwx-pvc Bound pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc 10Gi RWX openebs-rwx 32mkubectl get pv -> find the pv created: pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc 10Gi RWX Delete Bound grant/ebs-rwx-pvc openebs-rwxkubectl get po | grep busybox ---> Find the following: busybox-76cbb94474-bq4dr 0/2 Init:0/1 0 24mkubectl describe deploy busybox. Find the following output:Mounts:
/mnt/otp-shared-data from nfs-volume (rw)
Volumes:
nfs-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ebs-rwx-pvc
ReadOnly: false
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: busybox-76cbb94474 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 26m deployment-controller Scaled up replica set busybox-76cbb94474 to 1
kubectl describe rs busybox. Find the following log:Environment: <none>
Mounts:
/mnt/otp-shared-data from nfs-volume (rw)
Volumes:
nfs-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ebs-rwx-pvc
ReadOnly: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 27m replicaset-controller Created pod: busybox-76cbb94474-bq4dr
So, as we see. Container is created and the nfs volume is provisioned. However, the container fails to boot
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelskubectl logs <upgrade_job_pod> -n <openebs_namespace>Anything else we need to know?:
Add any other context about the problem here.
Environment details:
kubectl get po -n openebs --show-labels):NAME READY STATUS RESTARTS AGE LABELS
nfs-pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc-6b6f5df896-7spqf 0/1 Pending 0 48m openebs.io/nfs-server=nfs-pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc,pod-template-hash=6b6f5df896
openebs-localpv-provisioner-66b6b855fc-km4nb 1/1 Running 4 18d name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=2.6.0,pod-template-hash=66b6b855fc
openebs-ndm-6xzkm 1/1 Running 8 18d controller-revision-hash=7994455ddf,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.6.0,pod-template-generation=1
openebs-ndm-lv7kb 1/1 Running 0 17d controller-revision-hash=7994455ddf,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=2.6.0,pod-template-generation=1
openebs-ndm-operator-6bd5949967-kx6wd 1/1 Running 4 18d name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=2.6.0,pod-template-hash=6bd5949967
openebs-nfs-provisioner-7b4c9b87d9-2b9lb 1/1 Running 0 24h name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=dev,pod-template-hash=7b4c9b87d9
kubectl version):cat /etc/os-release):uname -a):kubectl describe deploy nfs-pvc -n openebs yields the following:Pod Template:
Labels: openebs.io/nfs-server=nfs-pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc
Containers:
nfs-server:
Image: itsthenetwork/nfs-server-alpine
Ports: 2049/TCP, 111/TCP
Host Ports: 0/TCP, 0/TCP
Environment:
SHARED_DIRECTORY: /nfsshare
Mounts:
/nfsshare from exports-dir (rw)
Volumes:
exports-dir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: nfs-pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc
ReadOnly: false
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: nfs-pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc-6b6f5df896 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 51m deployment-controller Scaled up replica set nfs-pvc-505263e6-25d8-4b86-9362-b95ea79fa7fc-6b6f5df896 to 1
i change nfsd.sh in nfs-server-alpine:0.9.0 image.
add line ulimit -HSn 1048576 to nfsd.sh fix the problem.
echo "Starting Mountd in the background..."These
ulimit -HSn 1048576 #add line
/usr/sbin/rpc.mountd --debug all --no-udp --no-nfs-version 2 --no-nfs-version 3
create rwx pvc the nfs pod loop error:
Displaying /etc/exports contents:
/nfsshare *(rw,fsid=0,async,no_subtree_check,no_auth_nlm,insecure,no_root_squash)
Starting rpcbind...
rpcbind: another rpcbind is already running. Aborting
Displaying rpcbind status...
program version netid address service owner
100000 4 tcp6 ::.0.111 - superuser
100000 3 tcp6 ::.0.111 - superuser
100000 4 udp6 ::.0.111 - superuser
100000 3 udp6 ::.0.111 - superuser
100000 4 tcp 0.0.0.0.0.111 - superuser
100000 3 tcp 0.0.0.0.0.111 - superuser
100000 2 tcp 0.0.0.0.0.111 - superuser
100000 4 udp 0.0.0.0.0.111 - superuser
100000 3 udp 0.0.0.0.0.111 - superuser
100000 2 udp 0.0.0.0.0.111 - superuser
100000 4 local /var/run/rpcbind.sock - superuser
100000 3 local /var/run/rpcbind.sock - superuser
100003 4 tcp 0.0.0.0.8.1 - superuser
100003 4 tcp6 ::.8.1 - superuser
Starting NFS in the background...
rpc.nfsd: knfsd is currently up
Exporting File System...
exporting *:/nfsshare
/nfsshare <world>
Starting Mountd in the background...These
rpc.mountd: setrlimit (RLIMIT_NOFILE) failed: Operation not permitted
Startup of NFS failed, sleeping for 2s, then retrying...
kubectl get pods -n openebs --show-labelsNAME READY STATUS RESTARTS AGE LABELS
cspc-operator-85d95555c-pp8t6 1/1 Running 2 (34m ago) 8d name=cspc-operator,openebs.io/component-name=cspc-operator,openebs.io/version=3.3.0,pod-template-hash=85d95555c
cstor-storage-8l5t-75fffc7f94-pksht 3/3 Running 6 (34m ago) 8d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-8l5t,openebs.io/version=3.3.0,pod-template-hash=75fffc7f94
cstor-storage-dz8g-f7d4f5fcc-k95xn 3/3 Running 6 (34m ago) 8d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-dz8g,openebs.io/version=3.3.0,pod-template-hash=f7d4f5fcc
cstor-storage-m6dw-6499b4c96f-tgqxw 3/3 Running 6 (34m ago) 8d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-m6dw,openebs.io/version=3.3.0,pod-template-hash=6499b4c96f
cvc-operator-9f4bfc89-dkmnn 1/1 Running 2 (34m ago) 8d name=cvc-operator,openebs.io/component-name=cvc-operator,openebs.io/version=3.3.0,pod-template-hash=9f4bfc89
nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc-7f7cc94f4f-q5w7j 1/1 Running 1 (10m ago) 31m openebs.io/nfs-server=nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc,pod-template-hash=7f7cc94f4f
openebs-cstor-admission-server-75df668b7-n6g4c 1/1 Running 2 (34m ago) 8d app=cstor-admission-webhook,openebs.io/component-name=cstor-admission-webhook,openebs.io/version=3.3.0,pod-template-hash=75df668b7
openebs-cstor-csi-controller-0 6/6 Running 13 (34m ago) 8d app=openebs-cstor-csi-controller,controller-revision-hash=openebs-cstor-csi-controller-7c6b54bbbc,name=openebs-cstor-csi-controller,openebs.io/component-name=openebs-cstor-csi-controller,openebs.io/version=3.3.0,role=openebs-cstor-csi,statefulset.kubernetes.io/pod-name=openebs-cstor-csi-controller-0
openebs-cstor-csi-node-9v78c 2/2 Running 4 (34m ago) 8d app=openebs-cstor-csi-node,controller-revision-hash=56945c4cd9,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.3.0,pod-template-generation=1,role=openebs-cstor-csi
openebs-cstor-csi-node-fkmqg 2/2 Running 4 (34m ago) 8d app=openebs-cstor-csi-node,controller-revision-hash=56945c4cd9,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.3.0,pod-template-generation=1,role=openebs-cstor-csi
openebs-cstor-csi-node-tz8nq 2/2 Running 4 (34m ago) 8d app=openebs-cstor-csi-node,controller-revision-hash=56945c4cd9,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.3.0,pod-template-generation=1,role=openebs-cstor-csi
openebs-ndm-5bjcl 1/1 Running 3 (34m ago) 8d controller-revision-hash=c7c95b565,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-64kvp 1/1 Running 3 (33m ago) 8d controller-revision-hash=c7c95b565,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-bq6s9 1/1 Running 3 (34m ago) 8d controller-revision-hash=c7c95b565,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-cluster-exporter-6f4c89665f-5lzqg 1/1 Running 2 (34m ago) 8d name=openebs-ndm-cluster-exporter,openebs.io/component-name=ndm-cluster-exporter,openebs.io/version=3.3.0,pod-template-hash=6f4c89665f
openebs-ndm-node-exporter-4fbqd 1/1 Running 2 (34m ago) 8d controller-revision-hash=678d8684cd,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-node-exporter-c77qq 1/1 Running 2 (34m ago) 8d controller-revision-hash=678d8684cd,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-node-exporter-g5tqr 1/1 Running 2 (34m ago) 8d controller-revision-hash=678d8684cd,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-operator-696c55bfff-kj4wn 1/1 Running 3 (34m ago) 8d name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.3.0,pod-template-hash=696c55bfff
openebs-nfs-provisioner-6f5b7dcbc8-kchlk 1/1 Running 2 (34m ago) 8d name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=0.9.0,pod-template-hash=6f5b7dcbc8
pvc-7b50b611-c7e3-4406-b4a7-d9197434c504-target-5fc8d84f9cpxxm7 3/3 Running 10 (33m ago) 7d16h app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=test-r-w-o,openebs.io/persistent-volume=pvc-7b50b611-c7e3-4406-b4a7-d9197434c504,openebs.io/target=cstor-target,openebs.io/version=3.3.0,pod-template-hash=5fc8d84f9c
pvc-b5c5a903-6970-45c2-aa65-fdb923bc2d36-target-6d97d95dc7j7flf 3/3 Running 10 (33m ago) 7d16h app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc,openebs.io/persistent-volume=pvc-b5c5a903-6970-45c2-aa65-fdb923bc2d36,openebs.io/target=cstor-target,openebs.io/version=3.3.0,pod-template-hash=6d97d95dc7
kubectl get pvc -n openebsNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc Bound pvc-b5c5a903-6970-45c2-aa65-fdb923bc2d36 10Gi RWO cstor-csi 7d16h
nfs-pvc-a84b182f-d658-4ae4-9c67-0093867c5a6e Bound pvc-d63176c3-643f-439b-ad86-d09d02ceba35 10Gi RWO cstor-csi 8d
kubectl get pvc -n defaultNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-r-w-o Bound pvc-7b50b611-c7e3-4406-b4a7-d9197434c504 10Gi RWO cstor-csi 7d16h
test-r-w-x Bound pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc 10Gi RWX cstor-nfs 7d16h
Environment details:
kubectl version):Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
NAME="Rocky Linux"
VERSION="9.0 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.0 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.0"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"
5.14.0-70.22.1.el9_0.x86_64containerd containerd.io 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6Describe the bug: I am unable to install Dynamic NFS Provisioner on GKE with GKE storage class to ensure my pvc is backed to a disk which will withstand node drains, upgrades and failures. The nfs-pv-<volume_name> pod is stuck in ContainerCreating status. Pod description shows the error-
MountVolume.MountDevice failed for volume "pvc-4ec6bfba-b5e7-47af-a6ec-5c1afd82e2b7" : rpc error: code = Internal desc = Failed to format and mount device from ("/dev/disk/by-id/google-pvc-4ec6bfba-b5e7-47af-a6ec-5c1afd82e2b7_regional") to ("/var/lib/kubelet/plugins/kubernetes.io/csi/pd.csi.storage.gke.io/2a9b9ac5e5e297142fb243da54292afd54d46280869358967bea09e4cc96b1ba/globalmount") with fstype ("ext4") and options ([]): mount failed: exit status 32
Expected behaviour: The pvc should be mounted and nfs-pv-<volume_name> pod should be in running status
Steps to reproduce the bug:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cms-pv-claim spec: storageClassName: openebs-gcp-pd-rwx accessModes: - ReadWriteMany resources: requests: storage: 200Gi
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-gcp-pd-rwx annotations: openebs.io/cas-type: nfsrwx cas.openebs.io/config: | - name: NFSServerType value: "kernel" - name: BackendStorageClass value: "regionalpd-storageclass" provisioner: openebs.io/nfsrwx reclaimPolicy: Delete
`kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: regionalpd-storageclass
provisioner: pd.csi.storage.gke.io
parameters:
type: pd-standard
replication-type: regional-pd
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
Steps to reproduce the bug should be clear and easily reproducible to help people gain an understanding of the problem
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelskubectl get pvc -n <openebs_namespace>kubectl get pvc -n <application_namespace>https://gist.github.com/sanke-t/7a4d8cc41f1840c79c6261da92d62003
Anything else we need to know?:
The same setup works fine with BackendStorageClass as openebs-hostpath
Environment details:
kubectl get po -n openebs --show-labels): 3.5.0kubectl version): v1.24.10-gke.2300cat /etc/os-release): Ubuntu 22.04.2 LTSuname -a): 5.15.0-1027-gkeThis is not really an issue, but I would really appreciate any insights on this. I am trying to install dynamic NFS provisioner on GKE to with backend storage class with provisioner pd.csi.storage.gke.io
Challenge here is meeting the pre-requisite of installation of nfs-common/nfs-util package on the node. Container OS does not have a package manager and does not allow any operations on the node.
Is there a workaround for this other than having ubuntu nodes on the cluster?
Describe the problem/challenge you have
If the NFS PV capacity is being used up, there is no way to resize the capacity - even though the underlying PV supports resize.
Describe the solution you'd like
NFS PV resize should be supported, if the underlying storage class (PV) supports it.
Anything else you would like to add:
This feature needs to take into account the following:
NFS servers created with this operator are provisioned with an update policy of RollingUpdate, but the rolling update doesn't work because the volume is still claimed by a prior instance.
Expected behaviour: A concise description of what you expected to happen
I would expect the pod to get deleted to free the volume, and then the new pod would be created.
Note: Brian asked me to report this, so here goes.
Describe the bug:
After creating a pvc with storage class openebs-rwx, the pods backing the nfs service don't progress past ContainerCreating when scheduled on nodes that don't have /dev/nvme-fabrics.
Expected behaviour:
The pods backing the nfs service should run after a pvc is created, even if scheduled on nodes without /dev/nvme-fabrics. We're using a cluster with two mayastor nodes, and two client nodes without mayastor.
Steps to reproduce the bug:
#Sample storage classes for OpenEBS Local PV
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-rwx
annotations:
openebs.io/cas-type: nfsrwx
cas.openebs.io/config: |
- name: NFSServerType
value: "kernel"
- name: BackendStorageClass
value: "mayastor-nvmf"
provisioner: openebs.io/nfsrwx
reclaimPolicy: Delete
Some Debug Output:
kubectl describe pod nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc-6949b95788-4l59q -n openebsName: nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc-6949b95788-4l59q
Namespace: openebs
Priority: 0
Node: client-1/10.0.96.152
Start Time: Mon, 15 Feb 2021 13:47:34 -0800
Labels: openebs.io/nfs-server=nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc
pod-template-hash=6949b95788
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc-6949b95788
Containers:
nfs-server:
Container ID:
Image: itsthenetwork/nfs-server-alpine
Image ID:
Ports: 2049/TCP, 111/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
SHARED_DIRECTORY: /nfsshare
Mounts:
/nfsshare from exports-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7j2mg (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
exports-dir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc
ReadOnly: false
default-token-7j2mg:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7j2mg
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 28m (x2 over 28m) default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 28m default-scheduler Successfully assigned openebs/nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc-6949b95788-4l59q to client-1
Normal SuccessfulAttachVolume 28m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-a17791d0-b0a4-43d1-b6fa-a88430c27989"
Warning FailedMount 8m43s (x7 over 26m) kubelet Unable to attach or mount volumes: unmounted volumes=[exports-dir], unattached volumes=[exports-dir default-token-7j2mg]: timed out waiting for the condition
Warning FailedMount 6m26s (x3 over 22m) kubelet Unable to attach or mount volumes: unmounted volumes=[exports-dir], unattached volumes=[default-token-7j2mg exports-dir]: timed out waiting for the condition
Warning FailedMount 2m14s (x21 over 28m) kubelet MountVolume.MountDevice failed for volume "pvc-a17791d0-b0a4-43d1-b6fa-a88430c27989" : rpc error: code = Internal desc = Failed to stage volume a17791d0-b0a4-43d1-b6fa-a88430c27989: attach failed: connect failed: NVMe connect failed: /dev/nvme-fabrics, No such file or directory (os error 2)
kubectl get pods -n <openebs_namespace> --show-labelsNAME READY STATUS RESTARTS AGE LABELS
nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc-6949b95788-4l59q 0/1 ContainerCreating 0 12m openebs.io/nfs-server=nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc,pod-template-hash=6949b95788
openebs-nfs-provisioner-787fc9c5d8-pbkcg 1/1 Running 0 14m name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=dev,pod-template-hash=787fc9c5d8
Some additional info:
The nfs-common package had been installed on both non-mayastor nodes. You can recreate the cluster by using the Terraform scripts in the 'aws_heterogeneous_cluster' branch of this repository. You can run this command to recreate the cluster:
terraform apply -var 'use_worker_instances_spec=true' -var 'worker_instances_spec=[{type = "i3.2xlarge", count = 2, mayastor_node_label = true},{type = "t3.large", count = 2, prefix = "client"}]'
Environment details:
kubectl get po -n openebs --show-labels):NAME READY STATUS RESTARTS AGE LABELS
nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc-6949b95788-4l59q 0/1 ContainerCreating 0 25m openebs.io/nfs-server=nfs-pvc-10696478-e04c-4e22-8ab0-5495542186fc,pod-template-hash=6949b95788
openebs-nfs-provisioner-787fc9c5d8-pbkcg 1/1 Running 0 27m name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=dev,pod-template-hash=787fc9c5d8
kubectl version):Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration: AWS
OS (e.g: cat /etc/os-release):
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
uname -a):Linux mayastor-tf-playground-worker-2 5.4.0-1037-aws #39-Ubuntu SMP Thu Jan 14 02:56:06 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxThere are a few bugs with the helm chart in need of fixing.
The liveness probe is wrongly configured and the NFS pod keeps continuously restarting. (https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/deploy/helm/charts/templates/deployment.yaml#L93) The pgrep command seems to have the wrong version and does not have the -c option. Therefore it keeps crashing and failing the liveness probe
The PSP config does not work because of a typo (https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/deploy/helm/charts/templates/clusterrole.yaml#L40) - there is }}} instead of <space>}}
I have followed tutorial "Deploying WordPress using Dynamic NFS Provisioner"
wordpress was deployed and started to monitor pvc usage using prometheus.
query in prometheus using
kubelet_volume_stats_used_bytes
kubelet_volume_stats_capacity_bytes
seems to display total of deployed node.
How can I extract pvc usage of wordpress and usage of mariaDb separately?
I am not familar with monitoring and prometheus. Are there any way to display disk usage of containers and pvc?
Hi,
I've installed the NFS provisioner helm chart and it starts up correctly. On top of it I try to install Minio with an RWX storage class from the provisioner. The PVCs & PVs seem to be correctly provisioned, but Minio will not start because of a permission error:
ERROR Unable to initialize backend: Unable to write to the backend
> Please ensure MinIO binary has write permissions for the backend
HINT:
Run the following command to add write permissions: `sudo chown -R <your-username> /export && sudo chmod u+rxw /export`
I use the default helm chart values, where I remove the liveness check and set the nfsStorageClass.backendStorageClass to an existing cStor-based storage class of my Open EBS installation. I also install the PSP components.
If I install minio on the OpenEBS cStor-based RWO storage class, it works normally, but it fails with missing write permission when I use the storage class of this provisioner.
Is there a configuration when installing NFS provisioner that I am missing to solve this error?
Describe the bug: A clear and concise description of what the bug is.
installed openebs on top of k3d 5.4.4 as an helmrelease using the nfs-provisioner chart, with this values (indentation is because of the helmrelease yaml, this block is under spec.values):
ndm:
enabled: false
ndmOperator:
enabled: false
localprovisioner:
enabled: false
openebsNDM:
enabled: false
nfsProvisioner:
enabled: true
nfsStorageClass:
name: nfs
reclaimPolicy: Retain
backendStorageClass: "local-path"
I then create my deployments (always via helm releases) which request a couple of pvc using the given storageclass name "nfs", i can see the pvc correctly created, while they can't be then mounted from the requesting pods, with these errors in event logs:
Mounting arguments: -t nfs 10.43.4.138:/ /var/lib/kubelet/pods/c2027ba1-b6c0-4e85-a842-c336a357992b/volumes/kubernetes.io~nfs/pvc-32ae907a-2045-4658-aa86-33512a6b9867
Output: mount: mounting 10.43.4.138:/ on /var/lib/kubelet/pods/c2027ba1-b6c0-4e85-a842-c336a357992b/volumes/kubernetes.io~nfs/pvc-32ae907a-2045-4658-aa86-33512a6b9867 failed: Not supported
10.43.x.x is the service network, pods are on the 10.42.x.x network...
Expected behaviour:
pods starting with nfs pvc mounted
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelsNAME READY STATUS RESTARTS AGE LABELS
openebs-nfs-provisioner-78fbfdd8c6-nvhtq 1/1 Running 0 50m app=nfs-provisioner,chart=nfs-provisioner-0.9.0,component=nfs-provisioner,heritage=Helm,name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=0.9.0,pod-template-hash=78fbfdd8c6,release=openebs
nfs-pvc-b46d0e46-30e7-4f59-ba56-fecdfb57a3bf-6777f6d4dd-zs9tr 1/1 Running 0 46m openebs.io/nfs-server=nfs-pvc-b46d0e46-30e7-4f59-ba56-fecdfb57a3bf,pod-template-hash=6777f6d4dd
nfs-pvc-32ae907a-2045-4658-aa86-33512a6b9867-68c65f964d-p8zm7 1/1 Running 0 46m openebs.io/nfs-server=nfs-pvc-32ae907a-2045-4658-aa86-33512a6b9867,pod-template-hash=68c65f964d
kubectl get pvc -n <openebs_namespace>NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc-b46d0e46-30e7-4f59-ba56-fecdfb57a3bf Bound pvc-03d661be-d385-4abf-9d9b-5f05c1d062d3 1Gi RWO local-path 47m
nfs-pvc-32ae907a-2045-4658-aa86-33512a6b9867 Bound pvc-da5f9eca-8610-4927-9cb4-ab60625688d7 1Gi RWO local-path 47m
kubectl get pvc -n <application_namespace>NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-common-mongodb-0 Bound pvc-fe2b5244-5bd7-4fbd-bd3b-a079547ead14 8Gi RWO local-path 50m
inst1-core-data-transient Bound pvc-b46d0e46-30e7-4f59-ba56-fecdfb57a3bf 1Gi RWX nfs 47m
inst1-core-data-lts Bound pvc-32ae907a-2045-4658-aa86-33512a6b9867 1Gi RWX nfs 47m
Anything else we need to know?:
Add any other context about the problem here.
Environment details:
kubectl get po -n openebs --show-labels):NAME READY STATUS RESTARTS AGE LABELS
openebs-nfs-provisioner-78fbfdd8c6-nvhtq 1/1 Running 0 52m app=nfs-provisioner,chart=nfs-provisioner-0.9.0,component=nfs-provisioner,heritage=Helm,name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=0.9.0,pod-template-hash=78fbfdd8c6,release=openebs
nfs-pvc-b46d0e46-30e7-4f59-ba56-fecdfb57a3bf-6777f6d4dd-zs9tr 1/1 Running 0 48m openebs.io/nfs-server=nfs-pvc-b46d0e46-30e7-4f59-ba56-fecdfb57a3bf,pod-template-hash=6777f6d4dd
nfs-pvc-32ae907a-2045-4658-aa86-33512a6b9867-68c65f964d-p8zm7 1/1 Running 0 48m openebs.io/nfs-server=nfs-pvc-32ae907a-2045-4658-aa86-33512a6b9867,pod-template-hash=68c65f964d
kubectl version):Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:17:29Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14+k3s1", GitCommit:"982252d747f7e50701da7052383d9fd788d2b20e", GitTreeState:"clean", BuildDate:"2022-06-27T22:44:37Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration:
k3d 5.4.4
OS (e.g: cat /etc/os-release):
macos bigsur latest
Hello, i have installed k8s cluster with 3 master and 7 worker nodes (name of nodes, like this: k8s-node1 - k8s-node6 and powerbi)
on k8s-node4, k8s-node5, k8s-node6, i added disks and installed openebs with cstor
then installed openebs dynamic-nfs-provisioner with helm and set default backend cstor.
its was working, 2 days, but i cant to do ls in mount folder, then i reinstalled it again and its go to error state:
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs-pvc
spec:
storageClassName: openebs-kernel-nfs
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongo
spec:
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
nodeSelector:
kubernetes.io/hostname: k8s-node4
containers:
- name: mongo
image: mongo:3.6.17-xenial
ports:
- containerPort: 27017
volumeMounts:
- name: storage
mountPath: /data/db
volumes:
- name: storage
persistentVolumeClaim:
claimName: nfs-pvc
nfs-pvc-23ee81ed-cb93-4727-9319-45a76e2b2e27-85d9d9ff7b-49db9.16d0b19997a7c268
FailedMount
Unable to attach or mount volumes: unmounted volumes=[exports-dir], unattached volumes=[kube-api-access-tkpsj exports-dir]: timed out waiting for the condition
nfs-pvc-23ee81ed-cb93-4727-9319-45a76e2b2e27-85d9d9ff7b-49db9.16d0b17fb6a82632
FailedMount
MountVolume.MountDevice failed for volume "pvc-22237728-5a18-4c37-8d59-f088745a14af" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
kubectl get pods -n <openebs_namespace> --show-labelsNAME READY STATUS RESTARTS AGE LABELS
cstor-disk-pool1-9r9x-6b4d87f49b-7xd7l 3/3 Running 3 (37d ago) 52d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-disk-pool1,openebs.io/cstor-pool-instance=cstor-disk-pool1-9r9x,openebs.io/version=3.0.0,pod-template-hash=6b4d87f49b
cstor-disk-pool1-hzg5-6b79f9998f-xp2c2 3/3 Running 3 (37d ago) 52d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-disk-pool1,openebs.io/cstor-pool-instance=cstor-disk-pool1-hzg5,openebs.io/version=3.0.0,pod-template-hash=6b79f9998f
cstor-disk-pool1-z9bb-75c7b65f47-77n8h 3/3 Running 0 8d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-disk-pool1,openebs.io/cstor-pool-instance=cstor-disk-pool1-z9bb,openebs.io/version=3.0.0,pod-template-hash=75c7b65f47
cstor-disk-pool2-j2dh-58fdcfcfff-6txm2 3/3 Running 3 (37d ago) 52d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-disk-pool2,openebs.io/cstor-pool-instance=cstor-disk-pool2-j2dh,openebs.io/version=3.0.0,pod-template-hash=58fdcfcfff
cstor-disk-pool2-jsww-665fd66759-g289l 3/3 Running 0 8d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-disk-pool2,openebs.io/cstor-pool-instance=cstor-disk-pool2-jsww,openebs.io/version=3.0.0,pod-template-hash=665fd66759
cstor-disk-pool2-kr9c-7bd97c946f-l96lg 3/3 Running 3 (37d ago) 52d app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-disk-pool2,openebs.io/cstor-pool-instance=cstor-disk-pool2-kr9c,openebs.io/version=3.0.0,pod-template-hash=7bd97c946f
nfs-pvc-19d0f9ca-5475-44a8-84e6-dc8a59d095a5-84759bd48b-7gd2x 1/1 Running 0 7d8h openebs.io/nfs-server=nfs-pvc-19d0f9ca-5475-44a8-84e6-dc8a59d095a5,pod-template-hash=84759bd48b
nfs-pvc-23ee81ed-cb93-4727-9319-45a76e2b2e27-85d9d9ff7b-49db9 0/1 ContainerCreating 0 12m openebs.io/nfs-server=nfs-pvc-23ee81ed-cb93-4727-9319-45a76e2b2e27,pod-template-hash=85d9d9ff7b
nfs-pvc-884284a8-7ed5-4876-8d9f-f53ca19c8017-7765c88c4-zrsbg 1/1 Running 0 8d openebs.io/nfs-server=nfs-pvc-884284a8-7ed5-4876-8d9f-f53ca19c8017,pod-template-hash=7765c88c4
nfs-pvc-8eb1e4e3-8ce5-4951-b522-e55e320ab32c-5b5c789bfd-rhsld 1/1 Running 0 8d openebs.io/nfs-server=nfs-pvc-8eb1e4e3-8ce5-4951-b522-e55e320ab32c,pod-template-hash=5b5c789bfd
nfs-pvc-b71d8186-8a72-4cc8-81f7-19b2d64e67f4-dc95dddd6-2txtv 1/1 Running 0 8d openebs.io/nfs-server=nfs-pvc-b71d8186-8a72-4cc8-81f7-19b2d64e67f4,pod-template-hash=dc95dddd6
nfs-pvc-d05e7993-83c9-489f-a38c-c0b38c89402f-6ccd46fd9d-s94kd 1/1 Running 0 8d openebs.io/nfs-server=nfs-pvc-d05e7993-83c9-489f-a38c-c0b38c89402f,pod-template-hash=6ccd46fd9d
openebs-cstor-admission-server-5754659f4b-5hkmq 1/1 Running 1 (37d ago) 52d app=cstor-admission-webhook,chart=cstor-3.0.2,component=cstor-admission-webhook,heritage=Helm,openebs.io/component-name=cstor-admission-webhook,openebs.io/version=3.0.0,pod-template-hash=5754659f4b,release=openebs
kubectl get pvc -n <openebs_namespace> kubectl get pvc -n openebs
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc-19d0f9ca-5475-44a8-84e6-dc8a59d095a5 Bound pvc-05044693-9b69-4a99-91a1-815ff950851a 1Gi RWO cstor-csi-disk1 7d8h
nfs-pvc-23ee81ed-cb93-4727-9319-45a76e2b2e27 Bound pvc-22237728-5a18-4c37-8d59-f088745a14af 5Gi RWO cstor-csi-disk1 13m
nfs-pvc-884284a8-7ed5-4876-8d9f-f53ca19c8017 Bound pvc-61a30045-5c4c-4f97-8974-db733100c594 3Gi RWO cstor-csi-disk1 8d
nfs-pvc-8eb1e4e3-8ce5-4951-b522-e55e320ab32c Bound pvc-97b8d215-adac-4bfd-9df9-c74ac71d6ddb 10Gi RWO cstor-csi-disk1 8d
nfs-pvc-b71d8186-8a72-4cc8-81f7-19b2d64e67f4 Bound pvc-9f2e94ab-57f8-4db0-a6f1-1fb2ba13fae8 100Gi RWO cstor-csi-disk1 8d
nfs-pvc-d05e7993-83c9-489f-a38c-c0b38c89402f Bound pvc-4b31f1dc-0cc7-4456-8621-d062e7141a73 3Gi RWO cstor-csi-disk1 8d
kubectl get pvc -n <application_namespace>[root@k8s-master1 openebs]# kubectl get pvc -n default
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc Bound pvc-23ee81ed-cb93-4727-9319-45a76e2b2e27 5Gi RWO openebs-kernel-nfs 14m
nfs-pvc1 Bound pvc-1558b073-ab7e-4f40-bb8f-8fd7fe6f5336 6Gi RWO cstor-csi-disk2 14m
Environment details:
kubectl get po -n openebs --show-labels):kubectl version):cat /etc/os-release):uname -a):Describe the bug: Add development workflow document
Expected behaviour: development workflow document which helps developers to contribute
Describe the problem/challenge you have
Describe the solution you'd like
gid (group ID) option under nfs storageclass parameters.gid value (if omitted defaults to root).gid value (or) root users.Anything else you would like to add:
Backend storage class user for provisioning is LVM-LocalPV.
Environment:
kubectl get po -n openebs --show-labels):kubectl version):cat /etc/os-release):uname -a):Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the problem/challenge you have
There is a requirement that want to provision nfs PV by clone an original backend PVC (the backend PVC CSI should support snapshot/clone, for example, LocalPV lvm or cStor).
Describe the solution you'd like
In the StorageClass yaml file, add dataSource annotation, then provisioner create backend PVC by clone original PVC by dataSource.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-rwx
annotations:
openebs.io/cas-type: nfsrwx
cas.openebs.io/config: |
- name: NFSServerType
value: "kernel"
- name: BackendStorageClass
value: "openebs-hostpath"
- name: dataSource
value: "backend-origin-pvc"
Anything else you would like to add:
If we can directly add the snapshot/clone support for nfs PVC, that is fine also.
Environment:
kubectl get po -n openebs --show-labels):kubectl version):cat /etc/os-release):uname -a):Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the problem/challenge you have
I need high-availability replication support which would otherwise be provided by Jiva, but it doesn't support ReadWriteMany.
I tried to make Jiva use replica storage class of openebs-rwx but that doesn't work because of the aforementioned limitation.
Describe the solution you'd like
Perhaps it would work the other way around, putting replicated Jiva storage class to back the NFS provisioner? It would be nice to have some documentation about this use case.
Edit: Actually I tried it the other way around now, NFS provisioner using a replicated OpenEBS Jiva replication to back it, and it works perfectly!
Some more information about my set up here:
https://www.costacoders.es/news/2022-12-24_kubernetes/
Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the bug:
Unable to process command ::set-env for the github action
Expected behaviour:
set-env command is deprecated. append the Env Variables to GITHUB_ENV
Steps to reproduce the bug:
Trigger the github action
The output of the github action:
Environment details:
The helm chart has no option to disable creation of the storage class.
Would be nice to have this like all other OpenEBS components.
Keeping helm values in sync with the tagged release
Just an observation that if we club the code and helm chart changes, the helm chart ends up having a configurable value that is not yet available in the released version. This will need to be addressed once we hit beta by either adding a column that specifies the version from which the value will be available or indicating that a given value will be available in the next tagged release.
Originally posted by @kmova in #84 (comment)
NFS provisioner binds to a persistent volume claim in the back in the ReadWriteOnce mode.
This is otherwise all well and good, but in a hard reboot of a node, starting these NFS volume pods fails as they fail to get the volume mount.
Specifically with these events:
Warning FailedMount 93s (x25 over 37m) kubelet MountVolume.MountDevice failed for volume "pvc-542bf63c-575a-4a82-ab4d-96d319e58179" : rpc error: code = FailedPrecondition desc = volume {pvc-542bf63c-575a-4a82-ab4d-96d319e58179} is already mounted at more than one place: {{/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/jiva.csi.openebs.io/0f4d4b7188975f990ed572ae7bdb4f2f1c07aa967d6460d2a8472343e7c110e1/globalmount ext4 /dev/disk/by-path/ip-10.152.183.138:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-542bf63c-575a-4a82-ab4d-96d319e58179-lun-0}}
At least in microk8s I have found no way to find out what is mounting the volume behind the scenes exactly, or maybe the accounting is simply wrong. I suppose some weird ghost container could in principle be the one keeping the volume reserved, but I haven't managed to find out what and how.
What I have tried:
mounts. Nothing special there.Steps to reproduce the bug:
Have several NFS persistent volume claims which use ReadWriteOnce volumes behind them active and reboot a Kubernetes node.
Expected:
I have no clue how to investigate further and due to manual surgery to try to make the cluster up and running again after this problem, the whole cluster is now in a state of no return and I need to rebuild it from scratch.
Environment details:
openebs.io/version=3.3.0kubectl version):Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-19T15:26:36Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-19T15:27:17Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
uname -a): Linux curie 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/LinuxI'm not sure if this is a NFS Provisioner bug, OpenEBS Jiva bug or MicroK8S bug.
This happens to me about weekly, if anyone has suggestions on how to debug what happens, I'd be glad to hear such.
Describe the bug: A clear and concise description of what the bug is.
Expected behaviour: A concise description of what you expected to happen
all started?
Steps to reproduce the bug:
already detailed above
The output of the following commands will help us better understand what's going on:
kubectl get pods -n <openebs_namespace> --show-labelsNAME READY STATUS RESTARTS AGE LABELS
openebs-ndm-gkbjs 0/1 ContainerCreating 0 8m50s app=openebs,component=ndm,controller-revision-hash=5f7f54ffb8,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.2.0,pod-template-generation=1,release=openebs
openebs-nfs-provisioner-78fbfdd8c6-sxt7n 1/1 Running 0 8m50s app=nfs-provisioner,chart=nfs-provisioner-0.9.0,component=nfs-provisioner,heritage=Helm,name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=0.9.0,pod-template-hash=78fbfdd8c6,release=openebs
openebs-localpv-provisioner-6445497896-kfhrv 1/1 Running 0 8m50s app=openebs,component=localpv-provisioner,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=3.2.0,pod-template-hash=6445497896,release=openebs
openebs-ndm-operator-66bb9cbbcb-lb8vd 1/1 Running 0 8m50s app=openebs,component=ndm-operator,name=ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.2.0,pod-template-hash=66bb9cbbcb,release=openebs
kubectl get pvc -n <openebs_namespace>No resources found in openebs namespace.
kubectl get pvc -n <application_namespace>No resources found in openebs namespace.
Anything else we need to know?:
in events i see these:
MountVolume.SetUp failed for volume "udev" : hostPath type check failed: /run/udev is not a directory
Unable to attach or mount volumes: unmounted volumes=[udev], unattached volumes=[sparsepath kube-api-access-6qw9q config udev procmount devmount basepath]: timed out waiting for the condition
Environment details:
OpenEBS version (use kubectl get po -n openebs --show-labels):
already above
Kubernetes version (use kubectl version):
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.7+k3s1", GitCommit:"8432d7f239676dfe8f748c0c2a3fabf8cf40a826", GitTreeState:"clean", BuildDate:"2022-02-24T23:03:47Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1
cat /etc/os-release):Describe the problem/challenge you have
Currently, a nfs-server-image (itsthenetwork/nfs-server-alpine, provisioner/helper_kernel_nfs_server.go ) is used, which is only available for amd64 at latest tag.
Describe the solution you'd like
I think the best solution would be to use a Image with arm64-support. I solved this issue locally by building the used image for amd64 + arm64 (i'm using a mixed setup) and pushed it to the kubernetes-registry. Then I changed the deployment-yaml using this image instead.
Used build-command: docker buildx build --platform linux/amd64,linux/arm64 -t [REGISTRY_IP]:32000/itsthenetwork/nfs-server-alpine:latest --push .
The second solution would be imho to make it configurable which image is used.
Info: With this changes it is possible to mount a pv.
Environment:
kubectl get po -n openebs --show-labels): 2.7.0kubectl version): v1.20.4-34+1ae8c29bbb48f7 (microk8s)cat /etc/os-release): 3x Ubuntu 20.04uname -a): Linux nuc 5.4.0-70-generic, Linux rock64 5.10.21-rockchip64, Linux pi4 5.4.0-1032-raspirwx nfs mount failed:
Normal Scheduled 3m12s default-scheduler Successfully assigned default/fio to worker-2
Warning FailedMount 69s kubelet, worker-2 Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[ms-volume default-token-f2g9z]: timed out waiting for the condition
Warning FailedMount 67s kubelet, worker-2 MountVolume.SetUp failed for volume "pvc-54c72f37-b8c0-403c-9c31-0ec0b775515c" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e5fd4e64-c07f-461b-b5ee-7c44ccbd0a7c/volumes/kubernetes.io~nfs/pvc-54c72f37-b8c0-403c-9c31-0ec0b775515c --scope -- mount -t nfs nfs-pvc-54c72f37-b8c0-403c-9c31-0ec0b775515c.openebs.svc.cluster.local:/ /var/lib/kubelet/pods/e5fd4e64-c07f-461b-b5ee-7c44ccbd0a7c/volumes/kubernetes.io~nfs/pvc-54c72f37-b8c0-403c-9c31-0ec0b775515c
Output: Running scope as unit: run-rbd1219ab70cb486690fc0a70f20935d8.scope
mount.nfs: Connection timed out
Expected behaviour: A concise description of what you expected to happen
volume should mount and app should run.
Steps to reproduce the bug:
install mayastor-nvmf enabled cluster on i3 nodes, then
install nfs provisioner, then
modify Mayastor/deploy/pvc.yaml to suit
apply pvc.yaml, fio.yaml
fio pod will fail here
run fiotest.sh
Describe the problem/challenge you have
Currently node affinity can only be specified for the entire provisioner.
Describe the solution you'd like
Our use-case requires that certain nfs servers must be run on specific nodes. The best case would be adding the affinity parameters to the storageclass perhaps via the cas.openebs.io/config configmap.
Vote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
Describe the problem/challenge you have
I work with Magento 2 daily and currently, our setup, uses containers that, due to security reasons, have their default user as "nobody" because this way, in the event of a security breach or any kind of problem the exposed user can't do much besides reading and writing to the folder that he is isolated.
The problem is that, with 10 pods running 1 container each, we have a ton of logs so we use Open-EBS NFS on top of Jiva to volume-mount an NFS PVC that allows all the containers to write to the logs at the same time and we have one centralized solutions for the logs.
Currently there is only support for FSGID, this sets the Group ID and also sets the permissions for that PVC in such a way that allows an user on the group 'nobody' to read and write in that volume-mount.
Describe the solution you'd like
In our entire infrastructure, the logs folder is the one with the biggest exposure because it's the one that has an actual access, all the other ones are isolated to the container, no one can access them.
I would like to be able to set the owner of the PVC and the permissions of the PVC the same way as I set the FSGID.
Anything else you would like to add:
Here is a code snippet of what I currently use with a few touches of what I would like to see being possible:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-nobody-rwx
annotations:
openebs.io/cas-type: nfsrwx
cas.openebs.io/config: |
- name: NFSServerType
value: "kernel"
- name: BackendStorageClass
value: "openebs-jiva-csi-default"
- name: UID
value: "65534"
- name: FSGID
value: "65534"
- name: Permissions
value: "744"
provisioner: openebs.io/nfsrwx
reclaimPolicy: DeleteAs you can see I added UID and Permissions to the cas.openebs.io config and this would allow me to set the owner and the permissions to drw-r--r--, which would be great for my use case.
Environment:
kubectl get po -n openebs --show-labels): openebs.io/version=2.12.0kubectl version):Client Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-3+67c78072cd01dd", GitCommit:"67c78072cd01dda1a3f9cd846849611be47a9ada", GitTreeState:"clean", BuildDate:"2021-10-14T05:08:53Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-3+67c78072cd01dd", GitCommit:"67c78072cd01dda1a3f9cd846849611be47a9ada", GitTreeState:"clean", BuildDate:"2021-10-14T05:03:53Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}microk8s v1.22.2cat /etc/os-release): Ubuntu 20.4 (3x nodes in one cluster, all the same)uname -a): Linux main 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxVote on this issue!
This is an invitation to the OpenEBS community to vote on issues.
Use the "reaction smiley face" up to the right of this comment to vote.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google โค๏ธ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.