Get device path fails with error `no fc disk found` about ibm-powervs-block-csi-driver

[Madhan-SWE]

Manual debugging logs:

Volume ID: e0066551-8dd6-4244-b002-2e1ba31f8f8c, Name: pvc-4d7d2aff-a4d1-4f9d-ac17-9a53548da00b is attached to the PVM Instance 9552c51d-5916-4ce5-a061-1e8bd7315ca8, state is in-use, wwn is 6005076810830198A000000000001494

2021/12/08 09:15:28 Calling the VolumeGet Method..
2021/12/08 09:15:28 The input volume name is e0066551-8dd6-4244-b002-2e1ba31f8f8c and  to the cloudinstance id 7845d372-d4e1-46b8-91fc-41051c984601
2021/12/08 09:15:28 Calling the New Auth Method in the IBMPower Session Code
2021/12/08 09:15:28 Calling the crn constructor that is to be passed back to the caller  65b64c1f1c29460e8c2e4bbfbd893c2c
2021/12/08 09:15:28 the region is lon and the zone is  lon04
2021/12/08 09:15:28 the crndata is ... crn:v1:bluemix:public:power-iaas:lon04:a/65b64c1f1c29460e8c2e4bbfbd893c2c:7845d372-d4e1-46b8-91fc-41051c984601::
GET /pcloud/v1/cloud-instances/7845d372-d4e1-46b8-91fc-41051c984601/volumes/e0066551-8dd6-4244-b002-2e1ba31f8f8c HTTP/1.1^M
Host: lon.power-iaas.cloud.ibm.com^M
User-Agent: Go-http-client/1.1^M
Accept: application/json^M
Authorization: Bearer
Crn: crn:v1:bluemix:public:power-iaas:lon04:a/65b64c1f1c29460e8c2e4bbfbd893c2c:7845d372-d4e1-46b8-91fc-41051c984601::^M
Accept-Encoding: gzip^M
^M

HTTP/1.1 200 OK^M
Transfer-Encoding: chunked^M
Cf-Cache-Status: DYNAMIC^M
Cf-Ray: 6ba4e3ad4ab94303-FRA^M
Connection: keep-alive^M
Content-Type: application/json^M
Date: Wed, 08 Dec 2021 09:15:28 GMT^M
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"^M
Server: cloudflare^M
Strict-Transport-Security: max-age=15724800; includeSubDomains^M
^M
1a9^M


{"bootable":false,"creationDate":"2021-12-08T09:14:05.000Z","diskType":"tier3","lastUpdateDate":"2021-12-08T09:15:24.000Z","name":"pvc-4d7d2aff-a4d1-4f9d-ac17-9a53548da00b","pvmInstanceIDs":["9552c51d-5916-4ce5-a061-1e8bd7315ca8"],"shareable":false,"size":1,"state":"in-use","volumeID":"e0066551-8dd6-4244-b002-2e1ba31f8f8c","volumePool":"Tier3-Flash-2","volumeType":"Tier3-Flash-2","wwn":"6005076810830198A000000000001494"}

Scanned all the host adapters:

[root@madhan-1-kube-1-22-2 ~]# echo "- - -" > /sys/class/scsi_host/host0/scan
[root@madhan-1-kube-1-22-2 ~]# echo "- - -" > /sys/class/scsi_host/host1/scan
[root@madhan-1-kube-1-22-2 ~]# echo "- - -" > /sys/class/scsi_host/host2/scan
[root@madhan-1-kube-1-22-2 ~]# echo "- - -" > /sys/class/scsi_host/host3/scan
[root@madhan-1-kube-1-22-2 ~]# echo "- - -" > /sys/class/scsi_host/host4/scan

Disk path would be : "/dev/disk/by-id/scsi-3" + wwn
Disk path "/dev/disk/by-id/scsi-36005076810830198A000000000001494" not found for the disk.

[root@madhan-1-kube-1-22-2 ~]# ls /dev/disk/by-id/
[root@madhan-1-kube-1-22-2 ~]# ls /dev/disk/by-id/
dm-name-mpatha    dm-name-mpathyu                                  dm-uuid-mpath-36005076810830198a000000000001426        scsi-36005076810830198a000000000001364        wwn-0x6005076810830198a0000000000011ff
dm-name-mpatha1   dm-name-mpathyv                                  dm-uuid-mpath-36005076810830198a000000000001427        scsi-36005076810830198a000000000001367        wwn-0x6005076810830198a000000000001200
.....
.....

[mkumatag]

This is an interesting issue, need to debug why this is really happening?!

Any hints in dmesg?

[Madhan-SWE]

dmesg logs:

[695699.935222] sd 1:0:6:8: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695701.951694] sd 3:0:6:8: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695712.097039] sd 2:0:7:9: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695716.200239] sd 1:0:6:2: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695717.240177] device-mapper: multipath: 253:130: Failing path 69:432.
[695731.421155] device-mapper: multipath: 253:96: Failing path 8:16.
[695732.436161] device-mapper: multipath: 253:97: Failing path 65:128.
[695733.471222] device-mapper: multipath: 253:96: Failing path 66:0.
[695735.503686] sd 4:0:7:2: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695740.600839] sd 1:0:6:7: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695742.619556] sd 1:0:7:7: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695743.656309] device-mapper: multipath: 253:97: Failing path 67:112.
[695751.788139] device-mapper: multipath: 253:138: Failing path 69:400.
[695753.826654] device-mapper: multipath: 253:138: Failing path 71:256.
[695763.971163] device-mapper: multipath: 253:130: Failing path 70:464.
[695768.051212] sd 1:0:6:2: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695787.792099] ibmvfc 30000006: 146200: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695787.794076] ibmvfc 30000008: 146300: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695787.794671] ibmvfc 30000006: 142200: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695787.796421] ibmvfc 30000008: 142300: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695788.577983] ibmvfc 30000007: A6200: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695788.578010] ibmvfc 30000007: A2200: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695788.582998] ibmvfc 30000004: A2300: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695788.583050] ibmvfc 30000004: A6300: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
[695794.460908] sd 1:0:6:9: Cancelling outstanding commands.
[695794.499533] sd 1:0:6:9: Successfully cancelled outstanding commands
[695794.500591] sd 1:0:6:9: [sdmj] tag#34 Command (00) : transaction cancelled (2:6) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0
[695794.500609] sd 1:0:6:9: [sdmj] tag#34 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=6s
[695794.500619] sd 1:0:6:9: [sdmj] tag#34 CDB: Test Unit Ready 00 00 00 00 00 00
[695794.500711] sd 1:0:6:9: rejecting I/O to offline device
[695795.100898] sd 4:0:7:2: Cancelling outstanding commands.
[695795.139365] sd 4:0:7:2: Successfully cancelled outstanding commands
[695795.139889] sd 4:0:7:2: [sdko] tag#79 Command (00) : transaction cancelled (2:6) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0
[695795.139906] sd 4:0:7:2: [sdko] tag#79 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=7s
[695795.139916] sd 4:0:7:2: [sdko] tag#79 CDB: Test Unit Ready 00 00 00 00 00 00
[695795.140025] sd 4:0:7:2: rejecting I/O to offline device
[695795.440401] device-mapper: multipath: 253:98: Failing path 66:448.
[695795.444816] device-mapper: multipath: 253:122: Failing path 69:352.
[695796.450733] device-mapper: multipath: 253:122: Failing path 69:448.
[695819.837796] device-mapper: multipath: 253:98: Failing path 8:144.
[695836.084905] sd 1:0:6:1: Power-on or device reset occurred
[695836.086984] sd 4:0:7:2: Power-on or device reset occurred
[695836.102320] sd 4:0:7:2: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695836.112291] sd 1:0:6:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695836.122298] sd 4:0:7:2: alua: port group 00 state A non-preferred supports tolusna
[695836.142324] sd 1:0:6:1: alua: port group 01 state N non-preferred supports tolusna
[695837.098312] sd 1:0:7:1: Power-on or device reset occurred
[695837.099692] device-mapper: multipath: 253:96: Reinstating path 8:16.
[695837.102014] device-mapper: multipath: 253:98: Reinstating path 66:448.
[695837.112305] sd 1:0:7:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[695837.152333] sd 1:0:7:1: alua: port group 00 state A non-preferred supports tolusna
[695838.124356] device-mapper: multipath: 253:96: Reinstating path 66:0.
[695840.162379] sd 1:0:6:2: Power-on or device reset occurred

Below errors could be the cause:

• Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
• 146200: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
• : [sdmj] tag#34 Command (00) : transaction cancelled (2:6) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0
• Power-on or device reset occurred

[Madhan-SWE]

All of my new workloads were failing as the fc disk was not found.
Had to restart the cluster(single node) to get rid of this issue.

[mkumatag]

All of my new workloads were failing as the fc disk was not found.
Had to restart the cluster(single node) to get rid of this issue.

If the issue is resolved post-reboot, and also if you see those previously attached volumes post-reboot then something wrong with the OS/kernel, more investigation is required at the OS level to understand why new devices aren't visible post-scan?!

[mkumatag]

Below errors could be the cause:

• Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
• 146200: Query Target failed: command failed (2:8000) unknown fc type (0) no additional explanation (0) rc=0xF7
• : [sdmj] tag#34 Command (00) : transaction cancelled (2:6) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0
• Power-on or device reset occurred

this may be just noise at times, try displaying the dmesg with user-readable time format like dmesg -T and see what messages it is logging while rescan time next time.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

This issue is still seen in the upstream and we are actively debugging this issue: https://bugzilla.linux.ibm.com/show_bug.cgi?id=199311

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[yussufsh]

/close
Fixed as part of code improvements with #392 #414 #422

[k8s-ci-robot]

@yussufsh: Closing this issue.

In response to this:

/close
Fixed as part of code improvements with #392 #414 #422

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.