Format and Mount fails about ibm-powervs-block-csi-driver

[Madhan-SWE]

Error snippet:

options:("defaults") errcode:(exit status 1) output:(mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: failed - Remote I/O error
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: 35dad7ea-e90d-490d-891e-fed7c2524ebd
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: Input/output error while writing out and closing file system
) 

Formatting the volume attached is failing during stage volume operation.

[Madhan-SWE]

Format operation for manually created and attached volume is passing as epxected.

[root@madhan-multinode-kube-worker-2 ~]# mkfs -t ext4 -F -m0 /dev/dm-16
mke2fs 1.45.6 (20-Mar-2020)
/dev/dm-16 contains a ext4 file system
	created on Mon Nov 15 06:53:42 2021
Discarding device blocks: done                            
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: c2086ebc-0d1d-4e46-96b3-1b02dce394e3
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

However the format fails for the disk created from automation

[root@madhan-multinode-kube-worker-2 ~]# mkfs -t ext4 -F -m0 /dev/dm-15
mke2fs 1.45.6 (20-Mar-2020)
Discarding device blocks: failed - Remote I/O error
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: e744392c-1753-4391-ab1f-75fff6a7660f
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: Input/output error while writing out and closing file system

[Madhan-SWE]

Volume details of a manually created volume:

{
         "bootVolume":false,
         "bootable":false,
         "creationDate":"2021-11-15T11:26:58.000Z",
         "diskType":"tier3",
         "href":"/pcloud/v1/cloud-instances/7031b049297e4588a3eafb21335d6a2b/volumes/df20f2b4-e82d-4d3a-a519-c137ca57b93b",
         "lastUpdateDate":"2021-11-15T11:27:57.000Z",
         "name":"testvol2",
         "pvmInstanceIDs":[
            "49d8a35d-40ff-46d3-8091-0553183e7daa"
         ],
         "shareable":false,
         "size":4,
         "state":"in-use",
         "volumeID":"df20f2b4-e82d-4d3a-a519-c137ca57b93b",
         "volumePool":"Tier3-Flash-2",
         "volumeType":"Tier3-Flash-2",
         "wwn":"6005076810830198A000000000000C67"
      }

Volume details of the volume created from automation:

     {
         "bootVolume":false,
         "bootable":false,
         "creationDate":"2021-11-15T05:32:02.000Z",
         "diskType":"tier3",
         "href":"/pcloud/v1/cloud-instances/7031b049297e4588a3eafb21335d6a2b/volumes/0a6edf14-1a53-4ee3-9a24-fb87ecc5088e",
         "lastUpdateDate":"2021-11-15T05:32:32.000Z",
         "name":"pvc-2222243566121843473",
         "pvmInstanceIDs":[
            "49d8a35d-40ff-46d3-8091-0553183e7daa"
         ],
         "shareable":false,
         "size":4,
         "state":"in-use",
         "volumeID":"0a6edf14-1a53-4ee3-9a24-fb87ecc5088e",
         "volumePool":"Tier3-Flash-2",
         "volumeType":"Tier3-Flash-2",
         "wwn":"6005076810830198A000000000000C48"
      }
    

[Madhan-SWE]

dmesg logs shows the Buffer I/O error on format failed disks

[3383621.275952] Buffer I/O error on dev dm-27, logical block 6291333, async page read

Root cause could be :

• damaged disk platters/cells
• connection problems (ie: bad cable, a “lost” iSCSI target, etc)
• thinly provisioned block device whose parent pool space was exausted

Reference: https://serverfault.com/questions/866109/centos7-buffer-i-o-error-on-dev-sda-logical-block-xxxxxxxxx-lost-async-page

[mkumatag]

Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

I see the number of blocks for which you are hitting with error got double the count of blocks compared to passing volume, can you create the same size volume with UI/CLI and retry.

one more way of reproducing this issue is also by creating/mounting a bunch of volumes from CLI/REST API and retrying the steps.

[Madhan-SWE]

@mkumatag
Should I create multiple volumes and try formatting?

[mkumatag]

@mkumatag
Should I create multiple volumes and try formatting?

yes

[Madhan-SWE]

Created 10 disks together from UI and tried formatting this one by one.
Formats were passing.

[mkumatag]

May be try back to back operation like create , attach, mount only one volume..

[Madhan-SWE]

Additional note:
This issue is not specific to one particular test case in the e2e tests.
If a test case fails due to format issue, it passes when I re run the same case.

[Madhan-SWE]

demsg

[507312.915299] scsi_io_completion_action: 64 callbacks suppressed
[507312.915307] sd 3:0:1:22: [sdgx] tag#37 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507312.915342] sd 3:0:1:22: [sdgx] tag#37 Sense Key : Illegal Request [current] 
[507312.915353] sd 3:0:1:22: [sdgx] tag#37 Add. Sense: Logical block address out of range
[507312.915365] sd 3:0:1:22: [sdgx] tag#37 CDB: Read(10) 28 00 01 3f ff 80 00 00 80 00
[507312.915379] print_req_error: 138 callbacks suppressed
[507312.915381] blk_update_request: critical target error, dev sdgx, sector 20971392 op 0x0:(READ) flags 0x4200 phys_seg 1 prio class 0
[507312.915417] blk_update_request: critical target error, dev dm-52, sector 20971392 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[507312.915613] sd 2:0:1:22: [sdgv] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507312.915624] sd 2:0:1:22: [sdgv] tag#51 Sense Key : Illegal Request [current] 
[507312.915634] sd 2:0:1:22: [sdgv] tag#51 Add. Sense: Logical block address out of range
[507312.915644] sd 2:0:1:22: [sdgv] tag#51 CDB: Read(10) 28 00 01 3f ff 80 00 00 80 00
[507312.915653] blk_update_request: critical target error, dev sdgv, sector 20971392 op 0x0:(READ) flags 0x4200 phys_seg 1 prio class 0
[507312.915670] blk_update_request: critical target error, dev dm-52, sector 20971392 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[507312.915681] buffer_io_error: 1016 callbacks suppressed
[507312.915683] Buffer I/O error on dev dm-52, logical block 163839, async page read
[507322.033993] sd 3:0:1:22: [sdgx] tag#82 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034005] sd 3:0:1:22: [sdgx] tag#82 Sense Key : Illegal Request [current] 
[507322.034018] sd 3:0:1:22: [sdgx] tag#82 Add. Sense: Logical block address out of range
[507322.034025] sd 3:0:1:22: [sdgx] tag#82 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034031] blk_update_request: critical target error, dev sdgx, sector 8388608 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034045] sd 3:0:1:22: [sdgx] tag#83 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034052] sd 3:0:1:22: [sdgx] tag#83 Sense Key : Illegal Request [current] 
[507322.034057] sd 3:0:1:22: [sdgx] tag#83 Add. Sense: Logical block address out of range
[507322.034063] sd 3:0:1:22: [sdgx] tag#83 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034069] blk_update_request: critical target error, dev sdgx, sector 8396800 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034078] sd 4:0:1:22: [sdgz] tag#70 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034083] sd 4:0:1:22: [sdgz] tag#70 Sense Key : Illegal Request [current] 
[507322.034089] sd 4:0:1:22: [sdgz] tag#70 Add. Sense: Logical block address out of range
[507322.034094] sd 4:0:1:22: [sdgz] tag#70 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034099] blk_update_request: critical target error, dev sdgz, sector 8392704 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034107] sd 4:0:1:22: [sdgz] tag#71 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034113] sd 4:0:1:22: [sdgz] tag#71 Sense Key : Illegal Request [current] 
[507322.034118] sd 4:0:1:22: [sdgz] tag#71 Add. Sense: Logical block address out of range
[507322.034123] sd 4:0:1:22: [sdgz] tag#71 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034129] blk_update_request: critical target error, dev sdgz, sector 8400896 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034136] sd 3:0:1:22: [sdgx] tag#88 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034142] sd 3:0:1:22: [sdgx] tag#88 Sense Key : Illegal Request [current] 
[507322.034147] sd 3:0:1:22: [sdgx] tag#88 Add. Sense: Logical block address out of range
[507322.034151] sd 3:0:1:22: [sdgx] tag#88 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034157] blk_update_request: critical target error, dev sdgx, sector 8409088 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034165] blk_update_request: critical target error, dev dm-52, sector 8388608 op 0x3:(DISCARD) flags 0x4800 phys_seg 0 prio class 0
[507322.034173] sd 4:0:1:22: [sdgz] tag#72 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034178] sd 4:0:1:22: [sdgz] tag#72 Sense Key : Illegal Request [current] 
[507322.034183] sd 4:0:1:22: [sdgz] tag#72 Add. Sense: Logical block address out of range
[507322.034188] sd 4:0:1:22: [sdgz] tag#72 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034193] blk_update_request: critical target error, dev sdgz, sector 8413184 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034201] sd 2:0:1:22: [sdgv] tag#55 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034207] sd 2:0:1:22: [sdgv] tag#55 Sense Key : Illegal Request [current] 
[507322.034213] sd 2:0:1:22: [sdgv] tag#55 Add. Sense: Logical block address out of range
[507322.034218] sd 2:0:1:22: [sdgv] tag#55 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034223] blk_update_request: critical target error, dev sdgv, sector 8404992 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034234] sd 2:0:1:22: [sdgv] tag#54 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.034241] sd 2:0:1:22: [sdgv] tag#54 Sense Key : Illegal Request [current] 
[507322.034246] sd 2:0:1:22: [sdgv] tag#54 Add. Sense: Logical block address out of range
[507322.034251] sd 2:0:1:22: [sdgv] tag#54 CDB: Unmap/Read sub-channel 42 00 00 00 00 00 00 00 18 00
[507322.034256] blk_update_request: critical target error, dev sdgv, sector 8417280 op 0x3:(DISCARD) flags 0x4a00 phys_seg 0 prio class 0
[507322.034263] blk_update_request: critical target error, dev dm-52, sector 8396800 op 0x3:(DISCARD) flags 0x4800 phys_seg 0 prio class 0
[507322.705820] sd 2:0:1:22: [sdgv] tag#54 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.705834] sd 2:0:1:22: [sdgv] tag#54 Sense Key : Illegal Request [current] 
[507322.705840] sd 2:0:1:22: [sdgv] tag#54 Add. Sense: Logical block address out of range
[507322.705846] sd 2:0:1:22: [sdgv] tag#54 CDB: Read(10) 28 00 01 00 00 00 00 00 80 00
[507322.706028] sd 3:0:1:22: [sdgx] tag#89 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[507322.706034] sd 3:0:1:22: [sdgx] tag#89 Sense Key : Illegal Request [current] 
[507322.706039] sd 3:0:1:22: [sdgx] tag#89 Add. Sense: Logical block address out of range
[507322.706044] sd 3:0:1:22: [sdgx] tag#89 CDB: Read(10) 28 00 01 00 00 00 00 00 80 00
[507322.767034] Buffer I/O error on dev dm-52, logical block 67624, lost async page write
[507322.767053] Buffer I/O error on dev dm-52, logical block 67625, lost async page write
[507322.767061] Buffer I/O error on dev dm-52, logical block 67626, lost async page write
[507322.767067] Buffer I/O error on dev dm-52, logical block 67627, lost async page write
[507322.767073] Buffer I/O error on dev dm-52, logical block 67628, lost async page write
[507322.767079] Buffer I/O error on dev dm-52, logical block 67629, lost async page write
[507322.767085] Buffer I/O error on dev dm-52, logical block 67630, lost async page write
[507322.767091] Buffer I/O error on dev dm-52, logical block 67631, lost async page write
[507322.767098] Buffer I/O error on dev dm-52, logical block 67632, lost async page write
[507322.767105] Buffer I/O error on dev dm-52, logical block 67633, lost async page write

[Madhan-SWE]

Note: Adding a slight delay before formatting the disk may solve the issue for a thick provisioned disk.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Madhan-SWE]

/reopen

[k8s-ci-robot]

@Madhan-SWE: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Madhan-SWE]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

This issue is not seen recently. Lets wait and see if the issue is reproduced while running e2e tests.
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Madhan-SWE]

This issue is not seen recently. Lets wait and see if the issue is reproduced while running e2e tests.
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[yussufsh]

/remove-lifecycle stale
We are not seeing mount errors in the latest code when running e2e locally. That said, IMO we should wait till we have e2e CI running to confirm the issue is in fact fixed.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[yussufsh]

/close
This issue was never seen again after the code improvement last year.

[k8s-ci-robot]

@yussufsh: Closing this issue.

In response to this:

/close
This issue was never seen again after the code improvement last year.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.