Comments (2)
Since when does the local path provisioner want to view pod logs? I guess this was added in rancher/local-path-provisioner#324 but noone updated the RBAC over here.
This will need to be updated in
k3s/manifests/local-storage.yaml
Lines 7 to 23 in 7f65975
from k3s.
##Environment Details
Reproduced using VERSION=v1.29.3+k3s1
Validated using COMMIT=81cd630f87ba3c0c720862af4cd02850303083a5
for what it's worth I was able to hit and reproduce this issue on the v1.28.8 branch #9833
Infrastructure
- Cloud
Node(s) CPU architecture, OS, and version:
Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
Cluster Configuration:
NAME STATUS ROLES AGE VERSION
ip-3-2-1-1 Ready control-plane,etcd,master 3h35m v1.29.3+k3s-81cd630f
Config.yaml:
node-external-ip: 3.2.1.1
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
profile: cis
protect-kernel-defaults: true
cluster-init: true
embedded-registry: true
Reproduction
$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ COMMIT=81cd630f87ba3c0c720862af4cd02850303083a5
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=server ./install-k3s.sh
$ set_kubefig
$ vim pv-test.yaml
$ vim pod-test.yaml
$ k get deploy -n kube-system local-path-provisioner -o jsonpath='{$.spec.template.spec.containers[:1].image}'
$ k apply -f pvc-test.yaml
$ k apply -f pod-test.yaml
$ kgp -A -o wide
$ k delete -f pod-test.yaml -f pvc-test.yaml
$ kg pv -A
$ k logs pod/local-path-provisioner
$ k logs pod/local-path-provisioner-6c86858495-9lkr6 -n kube-system
$ k logs pod/local-path-provisioner-6c86858495-9lkr6 -n kube-system
$ kg clusterrole local-path-provisioner-role -o yaml
Results:
//both new COMMIT_IDs and existing release retain the same versions of local-path-provisioner
$ k get deploy -n kube-system local-path-provisioner -o jsonpath='{$.spec.template.spec.containers[:1].image}'
rancher/local-path-provisioner:v0.0.26
// existing release clusterrole resource permissions attention to missing resources: pod/logs
$ kg clusterrole local-path-provisioner-role -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
objectset.rio.cattle.io/applied: H4sIAAAAAA
objectset.rio.cattle.io/id: ""
objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
objectset.rio.cattle.io/owner-name: local-storage
objectset.rio.cattle.io/owner-namespace: kube-system
creationTimestamp: "2024-04-15T18:42:20Z"
labels:
objectset.rio.cattle.io/hash: 183f35c65ffbc3064603f43f1580d8c68a2dabd4
name: local-path-provisioner-role
resourceVersion: "273"
uid: 6c447fa9-505f-43f3-b3d7-fa289476146f
rules:
- apiGroups:
- ""
resources:
- nodes
- persistentvolumeclaims
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- endpoints
- persistentvolumes
- pods
verbs:
- '*'
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
// latest commit install now includes the pods/log resources to the clusterrole
$ kg clusterrole local-path-provisioner-role -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
objectset.rio.cattle.io/applied: H4sIAAAAAAAYDAAA
objectset.rio.cattle.io/id: ""
objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
objectset.rio.cattle.io/owner-name: local-storage
objectset.rio.cattle.io/owner-namespace: kube-system
creationTimestamp: "2024-04-15T18:42:20Z"
labels:
objectset.rio.cattle.io/hash: 183f35c65ffbc3064603f43f1580d8c68a2dabd4
name: local-path-provisioner-role
resourceVersion: "278"
uid: f8302ce3-6990-416b-9afa-b545f373707d
rules:
- apiGroups:
- ""
resources:
- nodes
- persistentvolumeclaims
- configmaps
- pods/log
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- endpoints
- persistentvolumes
- pods
verbs:
- '*'
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
I did not hit the error during reproduction in the pod logs for what it's worth. But as the change is a permissions change on the clusterrole it's pretty straightforward to check if it has the right permissions from the kubectl api.
$ k logs pod/local-path-provisioner-6c86858495-9lkr6 -n kube-system
I0415 18:42:38.079959 1 controller.go:811] Starting provisioner controller rancher.io/local-path_local-path-provisioner-6c86858495-9lkr6_62958260-9704-4ca4-ab3a-6038ed1fef65!
I0415 18:42:38.180437 1 controller.go:860] Started provisioner controller rancher.io/local-path_local-path-provisioner-6c86858495-9lkr6_62958260-9704-4ca4-ab3a-6038ed1fef65!
I0415 21:31:29.264836 1 controller.go:1337] provision "default/test-pvc" class "local-path": started
time="2024-04-15T21:31:29Z" level=info msg="Creating volume pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 at ip-1-1-23:/var/lib/rancher/k3s/storage/pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7_default_test-pvc"
time="2024-04-15T21:31:29Z" level=info msg="create the helper pod helper-pod-create-pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 into kube-system"
I0415 21:31:29.268005 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-pvc", UID:"1cfed247-e7e4-4da8-b7d7-ffcefe3288c7", APIVersion:"v1", ResourceVersion:"29078", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/test-pvc"
time="2024-04-15T21:31:32Z" level=info msg="Volume pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 has been created on ip-1-1-23:/var/lib/rancher/k3s/storage/pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7_default_test-pvc"
time="2024-04-15T21:31:32Z" level=info msg="Start of helper-pod-create-pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 logs"
time="2024-04-15T21:31:32Z" level=info msg="Illegal option -a"
time="2024-04-15T21:31:32Z" level=info msg="End of helper-pod-create-pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 logs"
I0415 21:31:32.343240 1 controller.go:1442] provision "default/test-pvc" class "local-path": volume "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7" provisioned
I0415 21:31:32.343275 1 controller.go:1455] provision "default/test-pvc" class "local-path": succeeded
I0415 21:31:32.343283 1 volume_store.go:212] Trying to save persistentvolume "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7"
I0415 21:31:32.349700 1 volume_store.go:219] persistentvolume "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7" saved
I0415 21:31:32.349918 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-pvc", UID:"1cfed247-e7e4-4da8-b7d7-ffcefe3288c7", APIVersion:"v1", ResourceVersion:"29078", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7
I0415 21:34:46.530546 1 controller.go:1471] delete "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7": started
time="2024-04-15T21:34:46Z" level=info msg="Deleting volume pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 at ip-1-1-23:/var/lib/rancher/k3s/storage/pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7_default_test-pvc"
time="2024-04-15T21:34:46Z" level=info msg="create the helper pod helper-pod-delete-pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 into kube-system"
time="2024-04-15T21:34:48Z" level=info msg="Volume pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 has been deleted on ip-1-1-23:/var/lib/rancher/k3s/storage/pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7_default_test-pvc"
time="2024-04-15T21:34:48Z" level=info msg="Start of helper-pod-delete-pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 logs"
time="2024-04-15T21:34:48Z" level=info msg="Illegal option -a"
time="2024-04-15T21:34:48Z" level=info msg="End of helper-pod-delete-pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7 logs"
I0415 21:34:48.607227 1 controller.go:1486] delete "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7": volume deleted
I0415 21:34:48.611467 1 controller.go:1531] delete "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7": persistentvolume deleted
I0415 21:34:48.611485 1 controller.go:1536] delete "pvc-1cfed247-e7e4-4da8-b7d7-ffcefe3288c7": succeeded
from k3s.
Related Issues (20)
- Option service-node-port-range gets ignored HOT 2
- A potential risk in k3s that could lead to takeover of the cluster HOT 2
- x509: certificate has expired or is not yet valid HOT 2
- Add support for SLE Micro 6.0
- Reducing K3s Org Equinix usage HOT 8
- flannel-v6.1 MAC address changes every boot HOT 19
- k3s node becomes unresponsive with error InvalidDiskCapacity HOT 1
- Set cri-dockerd `streaming-bind-addr` to fixed address
- ErrImagePull when using `latest` docker image tag & embedded registry in air-gapped cluster HOT 2
- K3S agent starting on Google Coral crashes the host kernel due to nf_conntrack_netlink kernel module HOT 8
- Flannel crashoff on k3s <<<rasperry pi 4 arm64 debian bookworm HOT 1
- Support containerd config_path for stargz snapshotter HOT 1
- Adding incorrect flag during install results in a error which is hard to figure out what is wrong HOT 4
- K3s fails to start in a multi-network card environment with error "IPv6 was enabled but no IPv6 address was found on node" HOT 7
- K3S agent failed to join server, CA cert validation failed HOT 1
- k3s uninstall shouldn't delete k3s config.yaml HOT 2
- etcd-snapshot save ignores s3-folder param provided and saves only in s3-bucket location HOT 1
- Invalid `hosts.toml` for private registry is generated HOT 2
- etcd-snapshot save times out in 10 seconds the first try HOT 1
- ServiceLB back after auto-upgrade; k3s seems to ignore parameters from systemd unit after upgrade HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.