ControllerCreateVolume fails as the volume already exists about ibm-powervs-block-csi-driver

[Madhan-SWE]

/assign

[Madhan-SWE]

Since the multiple volumes are created at the same time, IBM Cloud takes some more time to return the response back.
In the mean time, controller plugin creates another new request to create volume and ignores the previous request's response.

Explored the External-Provisioner side car component and added timeout as 100s as part of deployment.
With this change create volume requests are passing.

Fix:

[root@madhan-1-kube-1-22-2 powervs-csi-driver]# cat deploy/kubernetes/base/controller.yaml 
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: powervs-csi-controller
  namespace: kube-system
  labels:
    app.kubernetes.io/name: powervs-csi-driver
spec:
      ....
      ....
      containers:
        ....
        ....
        - name: csi-provisioner
          image: k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4
          args:
            - --csi-address=$(ADDRESS)
            - --v=5
            - --feature-gates=Topology=true
            - --leader-election
            - --timeout=100s
            #- --leader-election-type=leases
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        ....
        ....

[Madhan-SWE]

Eventhough increasing the timeout fixes the issue, need to benchmark the driver with multiple 100s of workloads and then fix constant timeout and make changes in the code.

Shouldn't close this issue until the constant is decided and made changes in the repo.