purestorage / pso-csi Goto Github PK

View Code? Open in Web Editor NEW

30.0 30.0 31.0 5.57 MB

PSO CSI helm chart

License: Apache License 2.0

csi-driver flasharray flashblade helm-chart installation-pso pso purestorage

pso-csi's People

Contributors

Stargazers

Watchers

pso-csi's Issues

databases tolerations not considered for pso-db

Need to change pullPolicy from Always -> IfNotPresent

In the helm chart values.yaml the pullPolicies need to be changed from Always to IfNotPresent

  plugin:
    name: purestorage/k8s
    tag: v6.2.0
    pullPolicy: Always
  csi:
    provisioner:
      name: quay.io/k8scsi/csi-provisioner
      pullPolicy: Always
    snapshotter:
      name: quay.io/k8scsi/csi-snapshotter
      pullPolicy: Always
    attacher:
      name: quay.io/k8scsi/csi-attacher
      pullPolicy: Always
    resizer:
      name: quay.io/k8scsi/csi-resizer
      pullPolicy: Always
    nodeDriverRegistrar:
      name: quay.io/k8scsi/csi-node-driver-registrar
      pullPolicy: Always
    livenessProbe:
      name: quay.io/k8scsi/livenessprobe
      pullPolicy: Always
  database:
    deployer:
      name: purestorage/dbdeployer
      tag: v1.2.0
      pullPolicy: Always
    cockroachOperator:
      name: purestorage/cockroach-operator
      tag: v1.2.0
      pullPolicy: Always
    psctl:
      name: purestorage/psctl
      tag: v1.1.0
    cockroachdb:
      name: cockroachdb/cockroach
      tag: v20.2.6

   --- Should be ---

images:
  plugin:
    name: purestorage/k8s
    tag: v6.2.0
    pullPolicy: IfNotPresent
  csi:
    provisioner:
      name: quay.io/k8scsi/csi-provisioner
      pullPolicy: IfNotPresent
    snapshotter:
      name: quay.io/k8scsi/csi-snapshotter
      pullPolicy: IfNotPresent
    attacher:
      name: quay.io/k8scsi/csi-attacher
      pullPolicy: IfNotPresent
    resizer:
      name: quay.io/k8scsi/csi-resizer
      pullPolicy: IfNotPresent
    nodeDriverRegistrar:
      name: quay.io/k8scsi/csi-node-driver-registrar
      pullPolicy: IfNotPresent
    livenessProbe:
      name: quay.io/k8scsi/livenessprobe
      pullPolicy: IfNotPresent
  database:
    deployer:
      name: purestorage/dbdeployer
      tag: v1.2.0
      pullPolicy: IfNotPresent
    cockroachOperator:
      name: purestorage/cockroach-operator
      tag: v1.2.0
      pullPolicy: IfNotPresent
    psctl:
      name: purestorage/psctl
      tag: v1.1.0
    cockroachdb:
      name: cockroachdb/cockroach
      tag: v20.2.6

Support setting export rules for database volumes

End users may want to restrict database volume access to only nodes in the cluster (reasonably enough). Enable setting this from values.yaml.

Add schema validation for values.yaml file

Can you clarify the requirement: "FC Supported on Bare-metal K8s installations only"? If we intend to run multiple KVMs for k8s compute on top of our physical nodes, would that impact FC connectivity for those virtual machines?

failed to create snapshot in k8s 1.20

We recently upgraded to k8s 1.20. I am having troubles creating volumesnapshots. Here is the error:

spec:
snapshotClassName: pure-snapshotclass
snapshotContentName: ""
source:
apiGroup: null
kind: PersistentVolumeClaim
name: pvc-nginx
status:
creationTime: "2021-04-05T13:22:48Z"
error:
message: 'Failed to create snapshot: selfLink was empty, can''t make reference'
time: "2021-04-05T13:23:11Z"
readyToUse: false
restoreSize: null

I am using snapshot.storage.k8s.io/v1alpha1, which is still included in k8s 1.20.

$ kubectl api-versions | grep snap
snapshot.storage.k8s.io/v1alpha1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-31T15:33:39Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-31T16:42:32Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

We are using: image: purestorage/k8s:5.2.0

It looks like this issue is referenced here, but I'm not sure how that helps us. Is an upgrade to PSO 6.x required?
kubernetes/kubernetes#94660

Memory Leak on pso-db ?

Memory usage of pso-db pods in increasing at very high level.

Topology not working?

I'm having difficulty getting the CSI topology to work correctly. I have two Pure FlashBlades in separate racks, with a node in each rack, and am trying to get pods to get PVs on the Pure in the same rack.

Version is pure-csi v6.1.0, cluster is okd 4.7.0.

I have re-run the helm chart to update the flashblade labels:

arrays:
  FlashArrays: []
  FlashBlades:
    - MgmtEndPoint: "10.13.6.47"
      APIToken: **
      NFSEndPoint: 10.13.10.1
      Labels:
        topology.purestorage.com/rack: "rack-1"
    - MgmtEndPoint: "10.13.6.50"
      APIToken: **
      NFSEndPoint: 10.13.10.2
      Labels:
        topology.purestorage.com/rack: "rack-2"

and labelled my nodes:

jreynolds@laptop helm$ oc get node openshift-01 -oyaml | grep purestorage
    topology.purestorage.com/rack: rack-1
jreynolds@laptop helm$ oc get node openshift-02 -oyaml | grep purestorage
    topology.purestorage.com/rack: rack-2

and added a delayed pure-file storage class

jreynolds@laptop eqty3-oshift$ oc get storageclass
NAME                      PROVISIONER   RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
pure-block                pure-csi      Delete          Immediate              true                   7d3h
pure-file                 pure-csi      Delete          Immediate              true                   7d3h
pure-file-delay-binding   pure-csi      Delete          WaitForFirstConsumer   false                  3h39m

I've then created a stateful set with 5 replicas, each of which is scheduled to a node with a PVC of type pure-file-delay-binding.

  volumeClaimTemplates:
    - metadata:
        name: webserver-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: pure-file-delay-binding
        resources:
          requests:
            storage: 1Gi

# oc get pv
pvc-df23b3d4-296c-4fd4-85cf-c8c4f60319d4   1Gi        RWO            Delete           Bound    infra/webserver-data-webserver-statefulset-4      pure-file-delay-binding            113m
pvc-e0fa19c6-c3a6-4a98-974d-ac8acb3c9673   1Gi        RWO            Delete           Bound    infra/webserver-data-webserver-statefulset-0      pure-file-delay-binding            121m
pvc-fed6d8e7-dd5d-4a90-a457-58cfb2cca7f9   1Gi        RWO            Delete           Bound    infra/webserver-data-webserver-statefulset-1      pure-file-delay-binding            119m

However, I'm seeing all of my PVs created on the second Pure device, the one in "rack-2", regardless of if the pod is scheduled on node -01 or node -02. I'm expecting pods scheduled on node -01 get PVs created on the Pure device in rack-1, but this doesn't seem to be happening.

Have I missed a configuration step? I'm assuming the functionality is the same for pure-file as it is for the documented pure-block.

Create PVs in sync pod

Hi,

at the moment it looks like the created PVs by PVCs are only available in the standard container on the Flasharray.
We would like to have the feature to specify the sync-pod where the PV gets created.
Is this planned or possible somehow ?
We are using version 6.1.1 but did not find an option yet

Lehbot

error with openshift 4.6

tried a fresh install which worked on openshift 4.5

pso-db-public.pure-storage-orchestrator

time="2020-11-13T20:12:52Z" level=debug msg="Creating a persistent store service" logrus_error="can not add field "func"" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-11-13T20:12:52Z" level=debug msg="Create a cockroach store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-11-13T20:12:52Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.pure-storage-orchestrator port=26257 user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt" error="dial tcp: lookup pso-db-public.pure-storage-orchestrator on 192.172.0.10:53: no such host"
time="2020-11-13T20:12:52Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"
time="2020-11-13T20:12:52Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=13616.378826435386 span="{}"
time="2020-11-13T20:13:06Z" level=debug msg="Creating a persistent store service" logrus_error="can not add field "func"" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-11-13T20:13:06Z" level=debug msg="Create a cockroach store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-11-13T20:13:06Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.pure-storage-orchestrator port=26257 user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt" error="dial tcp: lookup pso-db-public.pure-storage-orchestrator on 192.172.0.10:53: no such host"
time="2020-11-13T20:13:06Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"

Explanation of DB failover

I'm after a bit of clarity as the resilience/purpose of the cockroach DB volumes in a failover scenario.

I have 2 Pure devices, PureA and PureB, in a k8s cluster. I deploy pso on it, and PureA gets 3 DB volumes and PureB gets 2 DB volumes.

What happens if I loose PureA? Does the database continue to function, or does it require a quorum of the volumes? I'm not clear if they're shards or 5 identical replicas, and if they require quorum or can run independently.

Thanks!

Process to enable snapshots on PSO6

We had the ability to run snapshots under PSO5 and kubernetes 1.18/1.19.
When we upgraded to 1.20 we could no longer create snapshots. Details: #158
We were not able to run the helm upgrade due to this: #164
We ran the helm installation with upgrade=false and a new clusterID.
PSO6 looks like it installed correctly. I can create PVCs, but I cannot create snapshots.

I still see the old pure-snapshotclass:
$ k get volumesnapshotclass pure-snapshotclass
NAME AGE
pure-snapshotclass 240d

When I try to apply the new definition (https://raw.githubusercontent.com/purestorage/pso-csi/master/pure-pso/snapshotclass.yaml), I get:
$ k apply -f snapshotclass.yaml
error: unable to recognize "snapshotclass.yaml": no matches for kind "VolumeSnapshotClass" in version "snapshot.storage.k8s.io/v1beta1"

I have the following snapshot CRDs:
$ k get crds | grep volumesnap
volumesnapshotclasses.snapshot.storage.k8s.io 2020-08-17T21:27:12Z
volumesnapshotcontents.snapshot.storage.k8s.io 2020-08-17T21:27:12Z
volumesnapshots.snapshot.storage.k8s.io 2020-08-17T21:27:12Z

And the snapshot.storage.k8s.io API still looks like it is the alpha.
$ k api-versions | grep snap
snapshot.storage.k8s.io/v1alpha1

ImagePullSecrets in Helm charts for Docker Registry

I recently encounter a situation where pods -not only PSO's- were evicted due to some pressure on nodes.

Then it went into a kind of scheduling loop:

PSO operator was down
Apps' Pods were not able to dettach/attach volume, no CSI available
PSO Pods/DB could not terminate because of the finalizer not being available
Apps and PSO Pods were getting into a crashloop and imagepull
Rate limiting on dockerhub was reached for image pulling
PSO could never start even after forcing finalizing the pods, because of docker hub blocking the image pull.

I sorted it out by creating a secret with my dockerhub credentials -we have a paid account- and setting the imagePullSecrets on PSO's Deployments&StatefulSets. It appears I could not do that through the helm charts and it will be replaced in next update.

Once nodes health OK, PSO was back again thanks to the secret, the cluster recovered.

It is the first time I am getting blocked in the middle of a troubleshooting because of docker's limits.

Is there a way to specify the secret in the helm values.yaml ?

After upenshift 4.5.x to 4.6.1 upgrade storage is broken

here are the logs from everything
urrent issues
time="2020-11-16T13:04:29Z" level=debug msg="Creating a persistent store service" logrus_error="can not add field "func"" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"

time="2020-11-16T13:08:15Z" level=debug msg="Creating a persistent store service" logrus_error="can not add field "func"" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-11-16T13:08:15Z" level=debug msg="Create a cockroach store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-11-16T13:08:16Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.pure-storage-orchestrator port=26257 user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt" error="dial tcp 192.172.178.152:26257: connect: connection refused"
time="2020-11-16T13:08:16Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"
time="2020-11-16T13:08:16Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=17235.615697477253 span="{}"

W201116 13:05:45.427545 1279 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:05:45.428777 1411 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:45.429246 1411 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:05:45.458427 1365 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:45.822423 1386 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:46.214857 1390 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:46.431366 1397 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:05:46.440555 1428 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:47.431666 1397 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:47.432905 1406 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:05:47.439857 1311 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:48.433062 1406 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:48.434717 1451 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
W201116 13:05:49.434838 1451 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:05:49.435085 1499 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:49.435806 1499 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:05:49.462496 1503 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:49.829804 1506 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:50.226650 1464 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:50.438001 1524 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:05:50.447764 1511 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:51.438172 1524 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:51.439281 1517 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:05:51.440664 1547 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:52.439497 1517 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:52.442148 1532 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:05:53.441284 1589 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:53.441742 1589 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:05:53.442300 1532 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:05:53.471004 1608 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:53.889971 1610 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:54.234375 1593 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:54.443852 1599 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:05:54.452825 1622 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:55.444146 1599 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:55.445561 1658 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:05:55.461444 1629 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:56.445730 1658 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:56.446560 1673 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:05:57.446547 1714 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:57.446716 1673 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:05:57.446998 1714 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:05:57.484859 1705 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:57.921633 1718 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:05:58.245716 1707 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:58.448679 1712 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:05:58.460870 1747 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:05:59.449110 1712 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:05:59.449878 1727 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:05:59.466085 1765 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:00.450034 1727 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:00.450642 1784 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
W201116 13:06:01.450803 1784 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:01.453036 1802 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:01.453383 1802 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:01.491607 1806 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:01.941085 1817 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:02.258000 1809 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:02.455028 1849 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:02.464566 1859 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:03.455149 1849 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:03.456162 1869 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:03.470844 1877 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:04.456244 1869 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:04.457517 1885 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:04.910355 1923 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:05.457358 1914 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:05.457668 1885 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:05.457927 1914 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:05.496328 1740 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:05.956418 1928 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:06.275910 1744 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:06.459713 1936 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:06.470566 1960 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:07.459894 1936 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:07.460386 1971 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:07.475532 1976 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:08.460550 1971 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:08.462468 1969 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:08.913222 2042 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:09.462642 1969 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:09.463921 2022 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:09.464358 2022 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:09.499958 2026 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:09.962595 2030 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:10.286984 2011 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:10.465685 1996 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:10.481454 2016 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:11.465833 1996 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:11.467576 2089 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:11.482867 2095 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:12.467836 2089 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:12.468323 2106 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:12.937239 2120 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:13.468277 2125 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:13.468496 2106 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:13.468635 2125 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:13.512637 2058 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:13.981840 2136 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:14.297412 2148 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:14.470840 2164 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:14.486742 2139 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:15.471020 2164 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:15.471118 2177 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:15.487137 2142 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:16.471390 2177 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:16.472875 2184 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:16.949848 2230 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:17.473015 2184 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:17.473230 2251 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:17.473620 2251 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:17.522888 1440 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:18.119094 2203 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:18.306235 2254 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:18.475347 2268 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:18.493729 2292 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:19.475565 2268 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:19.476402 2280 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:19.493537 2312 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:20.476698 2280 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:20.477541 2320 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:20.974825 2286 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:21.477605 2351 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:21.477899 2320 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:21.477999 2351 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:21.532068 2353 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:22.126840 2386 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:22.314464 2372 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:22.479830 2361 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:22.499152 2369 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:23.479982 2361 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:23.481391 2380 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:23.502805 2406 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:24.481691 2380 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:24.481808 2416 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:25.016855 2439 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:25.481861 2442 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:25.481945 2416 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:25.482238 2442 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:25.535832 2457 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:26.138112 2482 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:26.322950 2459 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:26.484566 2499 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:26.506518 2507 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:27.484725 2499 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:27.485662 2533 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:27.511636 2487 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:28.485816 2533 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:28.487086 2548 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:29.038593 2569 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:29.486656 2524 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:29.486983 2524 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:06:29.487232 2548 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:29.540471 2583 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:30.149945 2594 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:30.341763 2596 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:30.489044 2217 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.27:26257: connect: connection refused". Reconnecting...
I201116 13:06:30.510732 2591 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:31.489224 2217 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-5-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:31.492868 2575 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:31.515369 2616 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:32.492730 2664 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
W201116 13:06:32.493117 2575 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:33.050430 2671 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:33.068448 2675 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:33.492662 2676 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:33.492910 2664 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:33.493060 2676 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:33.546950 2681 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:34.157601 2695 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:34.350389 2698 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:34.503366 2706 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 [n?] circuitbreaker: gossip [::]:26257->pso-db-5-0.pso-db.pure-storage-orchestrator:26257 event: BreakerReset
I201116 13:06:34.503389 2706 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:34.503707 2706 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:34.515093 2731 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:35.499124 2754 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:35.519464 2760 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:36.499266 2754 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:36.500276 2768 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:37.068524 2737 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:37.119840 2803 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:37.500119 2752 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:37.500410 2768 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:37.500501 2752 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:37.551616 2808 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:38.165895 2813 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:38.360249 2822 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:38.500902 2823 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:38.501443 2823 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:38.523593 2800 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:39.503481 2867 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:39.526776 2855 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:40.503625 2867 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:40.504297 2879 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:41.078092 2889 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:41.127999 2840 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:41.504564 2879 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:41.515655 2896 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:41.516037 2896 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:41.555879 2860 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:41.610321 2905 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:42.189895 2930 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:42.382556 2906 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:42.516517 2922 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:42.516984 2922 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:42.530401 2908 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:43.523943 2929 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:43.537402 2978 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:44.524155 2929 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:44.524767 2974 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:45.086070 2956 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:45.147670 2992 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:45.524502 2999 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:45.524843 2999 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:06:45.524904 2974 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:45.560763 3014 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:45.689612 2959 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:46.217791 3016 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:46.391538 3027 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:46.525391 3009 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:46.525856 3009 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:46.537426 3019 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:47.527506 3074 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:47.541958 3054 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:48.527664 3074 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:48.528655 3070 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:49.090919 3106 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:49.174315 3139 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:49.528501 3113 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:49.528854 3070 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:49.528857 3113 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:49.566432 3117 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:49.697620 3093 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:50.247709 3154 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:50.406126 3156 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:50.529818 3126 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:50.530245 3126 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:50.541342 3130 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:51.531821 3186 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:51.546742 3193 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:52.531980 3186 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:52.534415 3203 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:53.115718 3218 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:53.185103 3213 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:53.533842 3148 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:53.534168 3148 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:06:53.534560 3203 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:53.573291 3217 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:53.710097 3237 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:54.269997 3223 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:54.420414 3252 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:54.536438 3255 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:54.549960 3225 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:54.554558 3255 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:06:55.539718 3271 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:55.550665 3265 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:56.539865 3271 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
W201116 13:06:56.546534 3304 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:06:57.123929 3316 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:57.209863 3310 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:57.541785 3182 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:57.542115 3182 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:06:57.547573 3304 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:06:57.578503 3332 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:57.727140 3312 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:58.283467 3364 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:58.432742 3348 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:58.542676 3373 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:06:58.542991 3373 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:06:58.554106 3350 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
W201116 13:06:59.551120 3391 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.4.28:26257: connect: connection refused". Reconnecting...
I201116 13:06:59.555122 3356 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:00.549010 3399 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
W201116 13:07:00.551254 3391 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-6-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:01.130556 3412 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:01.228403 3406 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:01.548715 3442 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:01.549089 3442 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:01.549170 3399 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:01.583084 3446 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:01.737649 3447 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:02.311430 3450 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:02.440073 3359 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:02.447426 3361 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:02.549821 3436 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:02.550271 3436 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:02.562996 3482 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:03.560016 3491 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:03.567390 3465 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 [n?] circuitbreaker: gossip [::]:26257->pso-db-6-0.pso-db.pure-storage-orchestrator:26257 event: BreakerReset
I201116 13:07:03.567411 3465 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:03.567729 3465 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:04.557020 3513 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:05.177163 3537 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:05.249632 3539 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:05.557177 3513 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:05.557542 3546 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:05.557881 3546 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:05.591702 3550 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:05.799909 3556 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:06.318740 3518 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:06.456003 3520 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:06.456651 3570 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:06.567742 3521 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:06.569397 3577 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:06.588766 3521 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:07.568991 3569 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:07.569394 3569 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:07.569449 3584 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:08.571008 3613 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:09.212266 3623 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:09.262348 3625 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:09.571160 3613 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:09.571872 3668 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:09.572183 3668 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:09.595852 3627 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:09.819313 3656 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:10.355970 3632 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:10.463455 3683 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:10.466014 3658 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:10.572676 3661 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:10.573034 3661 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:10.578400 3665 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:11.573993 3642 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:11.573998 3702 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:11.575400 3642 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:12.575716 3708 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:13.241145 3740 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:13.278229 3744 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:13.575866 3708 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:13.576056 3674 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:13.576357 3674 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:13.613437 3678 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:13.827251 3778 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:14.371120 3724 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:14.472466 3726 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:14.481690 3768 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:14.577001 3785 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:14.577380 3785 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:14.583796 3770 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:15.578119 3776 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:15.578605 3776 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:15.579363 3812 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:16.582648 3832 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:17.275505 3807 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:17.286853 3809 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:17.582387 3862 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:17.582714 3862 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:17.582787 3832 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:17.617310 3849 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:17.855669 3824 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:18.384207 3851 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:18.490119 3867 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:18.501422 3869 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:18.583665 3891 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:18.584093 3891 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:18.590077 3895 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:19.584712 3902 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:19.585171 3902 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:19.585792 3886 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:20.587237 3924 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:21.283236 3943 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:21.292172 3960 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:21.586761 3963 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:21.587102 3963 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:21.587416 3924 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:21.622762 3968 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:21.866437 3949 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:22.391111 3972 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:22.499181 3975 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:22.514542 3911 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:22.587712 3976 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:22.589098 3976 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:22.603169 3980 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:23.588607 3913 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:23.589062 3913 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:23.589949 4024 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:24.590910 4026 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:25.290944 4054 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:25.297422 4056 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:25.590581 4059 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:25.590924 4059 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:25.591054 4026 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:25.640775 4000 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:25.935090 4069 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:26.402639 4098 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:26.520801 4101 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:26.523478 4086 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:26.592160 4104 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:26.592704 4104 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:26.608932 4088 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:27.593097 4075 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:27.593509 4075 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:27.596410 4120 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:28.596971 4149 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:29.309349 4006 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:29.319126 4008 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:29.596883 4009 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:29.597113 4149 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:29.597211 4009 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:29.646548 4013 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:29.944520 4167 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:30.414935 4178 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:30.535286 4195 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:30.543249 4197 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:30.597925 4179 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:30.598452 4179 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:30.615910 4183 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:31.599059 4215 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:31.599589 4215 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:31.600699 4219 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:32.601177 4246 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:33.331619 4260 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:33.390707 4262 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:33.601328 4246 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:33.601783 4233 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:33.602162 4233 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:33.653203 4269 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:33.959981 4273 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:34.425260 3929 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:34.554264 3934 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:34.578934 4276 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:34.604947 4239 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:34.605386 4239 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:34.623302 4280 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:35.605749 4287 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:35.606190 4287 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:35.606502 4296 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
W201116 13:07:36.608230 4338 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp 192.168.3.47:26257: connect: connection refused". Reconnecting...
I201116 13:07:37.348473 4346 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:37.413026 4360 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:37.607872 4347 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:37.608205 4347 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
W201116 13:07:37.608335 4338 vendor/google.golang.org/grpc/clientconn.go:1206 grpc: addrConn.createTransport failed to connect to {pso-db-7-0.pso-db.pure-storage-orchestrator:26257 0 }. Err :connection error: desc = "transport: Error while dialing cannot reuse client connection". Reconnecting...
I201116 13:07:37.657979 4371 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:37.971793 4373 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:38.046204 4380 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:38.435157 4384 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:38.563375 4335 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:38.586456 4337 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:38.608965 4387 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:38.609554 4387 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:38.627388 4407 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:39.610166 4430 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:39.610689 4430 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:39.611862 4391 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:40.636595 4393 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 [n?] circuitbreaker: gossip [::]:26257->pso-db-7-0.pso-db.pure-storage-orchestrator:26257 event: BreakerReset
I201116 13:07:40.636617 4393 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:40.637298 4393 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:41.385876 4454 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:41.435360 4466 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:41.615300 4468 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:41.615607 4468 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:41.662679 4490 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:41.980045 4455 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:42.035552 4492 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:42.444459 4457 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:42.577758 4478 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:42.606314 4460 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:42.616556 4514 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:42.633114 4514 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:42.635199 4462 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:43.617534 4532 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:43.617599 4535 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:43.617964 4535 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:44.618406 4529 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:44.618857 4529 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:45.394787 4509 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:45.440794 4544 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:45.619384 4510 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:45.619727 4510 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:45.667532 4594 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:45.992794 4580 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:46.046370 4596 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:46.448229 4583 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:46.585700 4612 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:46.613617 4600 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:46.620768 4616 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:46.625058 4616 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:46.648730 4620 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:47.621028 4642 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:47.627330 4630 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:47.627782 4630 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:48.631094 4623 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:48.631789 4623 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:49.400046 4664 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:49.480837 4637 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:49.632768 4678 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:49.633104 4678 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:49.673437 4667 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.006754 4690 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.062877 4693 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.486816 4706 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.594084 4685 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.634062 4696 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.634431 4696 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:50.652617 4565 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:50.655660 4725 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:51.625793 4568 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:51.635266 4739 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:51.637587 4739 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:52.636632 4736 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:52.637291 4736 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:53.405676 4773 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:53.512029 4775 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:53.638011 4781 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:53.638355 4781 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:53.690478 4766 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.026419 4714 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.076546 4202 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.499919 4716 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.603141 4746 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.638969 4208 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.639369 4208 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:54.657794 4718 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:54.683434 4788 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:55.631020 4751 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:55.640274 4810 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:55.640670 4810 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:56.641606 4834 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:56.642346 4834 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:57.410943 4850 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:57.539727 4867 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:57.642576 4868 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:57.642926 4868 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:57.695624 4873 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.053171 4844 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.084621 4875 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.531973 4880 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.622532 4882 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.643726 4883 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.644089 4883 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:07:58.662804 4846 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:58.705036 4887 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:59.635854 4793 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:59.644908 4908 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:07:59.645335 4908 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:00.647557 4922 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:00.649570 4922 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:01.430755 4936 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:01.551928 4950 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:01.648429 4943 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:01.648842 4943 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:01.711313 4951 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.062097 4978 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.096448 4965 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.541092 4953 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.638519 4968 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.649526 4958 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.649871 4958 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:02.669121 4994 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:02.715699 4991 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:03.643772 4993 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:03.650765 4893 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:03.651201 4893 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:04.652269 5029 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:04.653150 5029 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:05.453631 5050 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:05.586044 5034 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:05.653200 4970 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:05.653594 4970 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:05.721358 5038 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.072558 5057 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.119807 5059 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.585228 5020 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.647299 5074 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.654335 5021 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.663089 5021 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:06.678457 5025 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:06.738936 5085 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:07.655272 5101 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:07.655288 5062 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:07.655718 5062 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:08.656386 5127 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:08.657099 5127 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:09.475613 5145 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:09.606975 5134 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:09.657593 5148 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:09.657945 5148 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:09.725742 5115 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.085627 5119 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.142044 5154 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.590244 5158 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.658953 5161 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.670580 5173 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.685412 5161 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:10.693336 5165 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:10.745441 5190 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:11.660123 5174 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:11.660361 5200 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:11.669587 5174 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:12.661252 5212 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:12.661956 5212 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:13.493087 5185 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:13.662423 5225 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:13.663062 5225 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:13.667097 5229 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:13.740194 5251 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.097323 5254 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.156162 5256 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.607934 5259 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.663831 5233 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.664244 5233 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:14.679664 5248 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.704370 5067 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:14.809509 5270 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:15.664931 5274 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:15.665067 5302 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:15.665381 5274 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:16.666112 5288 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:16.666669 5288 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:17.510744 5346 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:17.669973 5345 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:17.670304 5345 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:17.679838 5365 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:17.744715 5367 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.111553 5379 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.169545 5369 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.617737 5371 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.670899 5384 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.671439 5384 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:18.687563 5354 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.708161 5388 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:18.840010 5390 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:19.672573 5356 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:19.672917 5356 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:19.674013 5360 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:20.673480 5320 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:20.674196 5320 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:21.543150 5404 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:21.674536 5405 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:21.674840 5405 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:21.684738 5443 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:21.749509 5409 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.120715 5444 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.178645 5445 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.630979 5450 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.675630 5277 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.684947 5277 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:22.708633 5478 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.712574 5464 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:22.856161 5490 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:23.677882 5483 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:23.678229 5508 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:23.678340 5483 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:24.681112 5516 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:24.681866 5516 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:25.557292 5526 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:25.682122 5527 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:25.682871 5527 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:25.699047 5455 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:25.758037 5457 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.134483 5420 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.195242 5422 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.639730 5543 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.683004 5489 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.687205 5489 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:26.717198 5589 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.720706 5575 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:26.884789 5583 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:27.684204 5557 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:27.684564 5557 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:27.687856 5603 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:28.685286 5624 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:28.685929 5624 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:29.598595 5631 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:29.686747 5615 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:29.687097 5615 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:29.703972 5651 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:29.762364 5637 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.145391 5666 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.207242 5639 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.645403 5670 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.690607 5676 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.691092 5676 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:30.735147 5641 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.737621 5642 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:30.897356 5687 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:31.692730 5658 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:31.693169 5658 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:31.694645 5662 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:32.694088 5706 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:32.694819 5706 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:33.619686 5725 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:33.695177 5709 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:33.695510 5709 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:33.752219 5713 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:33.770742 5552 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.157997 5733 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.217156 5729 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.656003 5763 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.698193 5736 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.700133 5736 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:34.749385 5769 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.749425 5751 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:34.937294 5743 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:35.699041 5647 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:35.699471 5647 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:35.704043 5795 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:36.700655 5783 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:36.704409 5783 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:37.655852 5842 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:37.701536 5830 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:37.701839 5830 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:37.758691 5834 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:37.776243 5787 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.167383 5838 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.227730 5840 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.680041 5791 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.702565 5793 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.702969 5793 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:38.753965 5862 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.764003 5876 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:38.965811 5866 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:39.703685 5818 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:39.704081 5818 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:39.718891 5893 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:40.704837 5882 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:40.705548 5882 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:41.662845 5938 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:41.709787 5929 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:41.710079 5929 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:41.780377 5933 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:41.844019 5911 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.187693 5913 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.238170 5915 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.701446 5943 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.711089 5921 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.711546 5921 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:42.761308 5960 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.787326 5961 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:42.992884 5945 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:43.712160 5982 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:43.712569 5982 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:43.725748 5950 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:44.713304 5901 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:44.713877 5901 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:45.672862 6009 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:45.714591 5994 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:45.714911 5994 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:45.795290 6020 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:45.855373 6014 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.195517 6016 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.250936 6024 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.715751 6036 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.723619 6036 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:46.765860 6000 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.766790 6044 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.815758 6050 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:46.998578 6066 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:47.716733 6072 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:47.717128 6072 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:47.729551 6063 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:48.717690 6100 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:48.718646 6100 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:49.683258 6114 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:49.718764 6117 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:49.719060 6117 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:49.802017 6111 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:49.870048 6092 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:50.205968 6122 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:50.259069 6031 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:50.719886 6094 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:50.720479 6094 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:50.780081 6149 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:50.787518 6151 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:50.827657 6166 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:51.007971 6170 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:51.721370 6159 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:51.722062 6159 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:51.733550 6179 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:52.729401 6207 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:52.730797 6207 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:53.703601 6216 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:53.730424 6187 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:53.730728 6187 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:53.816444 6219 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:53.888535 6221 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:54.214911 6193 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:54.266770 6242 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:54.731327 6237 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:54.731687 6237 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:54.792354 6249 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:54.799510 6258 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:54.835866 6260 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:55.018985 6262 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:55.732059 6290 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:55.732471 6290 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:55.746790 6279 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:56.733653 6283 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:56.734549 6283 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:57.709413 6135 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:57.734904 6269 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:57.735273 6269 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:57.829623 6273 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:57.904031 6323 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:58.224429 6316 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:58.283437 6141 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:58.736125 6341 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:58.736500 6341 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:58.809325 6304 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:58.812796 6356 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:58.852939 6343 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:59.037865 6358 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:59.738416 6377 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:08:59.738858 6377 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:08:59.751282 6381 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:00.739634 6391 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:00.740709 6391 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:01.719141 6404 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:01.740915 6396 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:01.741190 6396 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:01.834102 6411 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:01.938194 6400 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:02.242745 6434 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:02.295714 6416 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:02.742565 6436 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:02.742938 6436 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:02.818857 6432 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:02.819006 6466 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:02.867259 6468 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:03.061659 6440 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:03.743462 6477 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:03.743799 6477 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:03.761033 6481 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:04.744452 6446 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:04.745268 6446 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:05.729788 6491 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:05.746580 6495 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:05.746900 6495 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:05.841174 6328 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:05.974858 6505 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:06.252192 6329 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:06.308853 6511 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:06.747796 6535 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:06.749162 6535 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:06.824799 6539 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:06.827362 6541 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:06.876714 6519 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:07.080715 6331 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:07.748775 6583 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:07.749181 6583 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:07.767182 6587 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:08.749738 6570 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:08.750619 6570 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:09.738040 6597 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:09.750756 6601 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:09.751134 6601 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:09.846481 6613 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:09.981813 6605 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:10.280828 6617 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:10.319290 6619 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:10.751635 6644 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:10.752278 6644 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:10.830330 6648 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:10.832803 6628 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:10.894785 6630 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:11.099320 6652 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:11.752509 6675 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:11.756608 6675 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:11.772146 6635 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:12.756296 6687 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:12.758785 6687 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:13.757434 6670 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:13.757714 6670 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:13.775305 6548 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:13.853724 6712 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.043812 6714 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.289815 6723 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.327806 6553 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.758748 6742 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.759318 6742 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:14.837931 6746 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.867842 6728 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:14.903305 6748 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:15.114837 6718 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:15.761844 6730 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:15.762312 6730 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:15.781643 6759 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:16.762795 6764 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:16.763318 6764 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:17.764386 6797 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:17.764840 6797 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:17.810467 6801 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:17.861694 6803 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.065968 6807 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.301373 6809 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.340834 6810 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.765523 6822 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.765933 6822 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:18.842685 6812 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.876997 6828 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:18.910516 6814 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:19.115150 6830 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:19.766804 6850 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:19.767198 6850 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:19.786069 6842 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:20.768523 6871 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:20.769276 6871 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:21.770280 6878 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:21.770619 6878 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:21.842714 6883 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:21.866910 6845 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.075251 6847 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.309331 6849 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.350932 6901 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.771773 6938 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.772210 6938 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:22.848083 6903 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.889436 6944 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:22.923964 6922 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:23.131104 6947 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:23.772991 6913 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:23.773452 6913 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:23.793054 6925 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:24.774551 6953 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:24.775825 6953 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:25.782305 6995 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:25.782633 6995 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:25.863319 6966 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:25.873298 7011 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.086960 6968 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.326626 7001 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.368050 7003 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.784636 6972 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.785167 6972 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:26.852656 7028 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.896076 7020 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:26.932792 6976 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:27.142477 7022 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:27.786022 7067 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:27.789586 7067 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:27.796763 7032 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:28.787308 7034 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:28.787888 7034 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:29.788829 7092 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:29.789190 7092 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:29.878219 7041 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:29.881071 7107 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.104915 7083 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.335585 7085 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.379143 7109 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.795672 6884 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.796217 6884 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:30.858282 6890 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.917685 7130 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:30.944665 7132 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:31.176940 7099 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:31.797037 7137 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:31.797617 7137 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:31.805888 7154 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:32.798519 7149 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:32.799377 7149 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:33.799768 7170 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:33.800039 7170 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:33.889858 7187 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:33.902751 7117 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.125031 7191 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.343430 7120 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.386342 7202 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.801444 7174 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.802024 7174 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:34.862911 7180 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.924234 7209 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:34.961093 7200 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:35.198815 7182 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:35.804408 7225 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:35.805070 7225 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:35.812364 7229 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:36.805530 7230 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:36.806399 7230 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:37.806848 7283 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:37.807178 7283 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:37.907047 7273 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:37.911316 7255 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.134591 7288 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.351747 7290 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.400453 7277 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.807983 7260 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.808480 7260 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:38.867106 7302 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.940073 7306 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:38.971079 7297 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:39.211083 7264 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:39.813035 7332 gossip/client.go:124 [n?] started gossip client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:39.816509 7332 gossip/client.go:131 [n?] closing client to pso-db-6-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-6-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:39.821016 7336 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-21-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:40.814133 7343 gossip/client.go:124 [n?] started gossip client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:40.814806 7343 gossip/client.go:131 [n?] closing client to pso-db-7-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-7-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:41.815109 7380 gossip/client.go:124 [n?] started gossip client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:41.815527 7380 gossip/client.go:131 [n?] closing client to pso-db-8-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-8-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:41.913351 7384 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-8-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:41.950142 7350 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:42.149132 7371 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-14-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:42.370844 7386 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-20-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:42.414849 7388 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-7-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:42.817636 7395 gossip/client.go:124 [n?] started gossip client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:42.818859 7395 gossip/client.go:131 [n?] closing client to pso-db-5-0.pso-db.pure-storage-orchestrator:26257: stopping outgoing client to n0 (pso-db-5-0.pso-db.pure-storage-orchestrator:26257); loopback connection
I201116 13:09:42.878855 7411 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-16-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:42.951549 7358 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-22-0.pso-db.pure-storage-orchestrator:26257
I201116 13:09:43.024856 7413 gossip/server.go:227 [n?] received initial cluster-verification connection from pso-db-15-0.pso-db.pure-storage-orchestrator:26257

time="2020-11-16T13:11:23Z" level=debug msg="HTTP request completed" status="200 OK"
time="2020-11-16T13:11:23Z" level=info msg="node live" node=pso-db-9 reconciler_name=controller_intrusion request=pure-storage-orchestrator/pso-db
time="2020-11-16T13:11:23Z" level=info msg="new nodes need to be created" count=5 reconciler_name=controller_intrusion request=pure-storage-orchestrator/pso-db
time="2020-11-16T13:11:23Z" level=error msg="too many stateful sets already exist, stopping to avoid flooding" reconciler_name=controller_intrusion request=pure-storage-orchestrator/pso-db
time="2020-11-16T13:11:23Z" level=info msg="updated intrusion status" intrusion_status=UnableToStart reconciler_name=controller_intrusion replica_statuses="BackendID: e36ba64f-7c70-4d60-9744-37a8b783590a, BackendType:FlashArray, CurrentStatusStart: 2020-11-14T19:36:22Z, ReplicaPhase: Down, Volume:pso-db_0000000000; BackendID: 63d6beee-7df5-4914-8128-e8d84bd929f9, BackendType:FlashArray, CurrentStatusStart: 2020-11-14T19:16:13Z, ReplicaPhase: Down, Volume:pso-db_0000000000; BackendID: e36ba64f-7c70-4d60-9744-37a8b783590a, BackendType:FlashArray, CurrentStatusStart: 2020-11-13T19:05:22Z, ReplicaPhase: Down, Volume:pso-db_0000000001; BackendID: 63d6beee-7df5-4914-8128-e8d84bd929f9, BackendType:FlashArray, CurrentStatusStart: 2020-11-14T19:16:13Z, ReplicaPhase: Down, Volume:pso-db_0000000001; BackendID: 63d6beee-7df5-4914-8128-e8d84bd929f9, BackendType:FlashArray, CurrentStatusStart: 2020-11-14T19:16:13Z, ReplicaPhase: Down, Volume:pso-db_0000000002; BackendID: e36ba64f-7c70-4d60-9744-37a8b783590a, BackendType...
time="2020-11-16T13:11:23Z" level=error msg="requeuing to see if nodes self-heal" error="intrusion previously initialized, but now not enough nodes are ready (0/5 ready, need a majority)" reconciler_name=controller_intrusion request=pure-storage-orchestrator/pso-db
time="2020-11-16T13:11:23Z" level=error msg="failed to update intrusion status" error="Operation cannot be fulfilled on intrusions.pso.purestorage.com "pso-db": the object has been modified; please apply your changes to the latest version and try again" reconciler_name=controller_intrusion request=pure-storage-orchestrator/pso-db
time="2020-11-16T13:11:23Z" level=info msg="--------------------- Begin reconcile ---------------------" reconciler_name=controller_intrusion request=pure-storage-orchestrator/pso-db
time="2020-11-16T13:11:23Z" level=debug msg="finalizer is already in correct state" finalizer_present=true reconciler_name=controller_intrusion
time="2020-11-16T13:11:23Z" level=debug msg="Beginning to ensure child owners set" reconciler_name=controller_intrusion
time="2020-11-16T13:11:23Z" level=debug msg="Child object owners already set correctly" object_name=pure object_namespace=pure-storage-orchestrator reconciler_name=controller_intrusion
time="2020-11-16T13:11:23Z" level=debug msg="Child object owners already set correctly" object_name=pso-db-runner object_namespace=pure-storage-orchestrator reconciler_name=controller_intrusion

time="2020-11-16T13:12:49Z" level=debug msg="Sending request:" body= method=GET url="https://10.127.216.104/api/1.6/volume"
time="2020-11-16T13:12:49Z" level=debug msg="Received response" response="[]" status="401 UNAUTHORIZED"
time="2020-11-16T13:12:49Z" level=debug msg="Attempting POST" body="{b628d192-4803-4357-97f2-7dcf0e16450d}" path=/auth/session
time="2020-11-16T13:12:49Z" level=debug msg="Sending request:" body= method=POST url="https://10.127.216.104/api/1.6/auth/session"
time="2020-11-16T13:12:49Z" level=debug msg="Received response" response="{"username": "srvCaaS_PureS_NP"}" status="200 OK"
time="2020-11-16T13:12:49Z" level=debug msg="Sending request:" body= method=GET url="https://10.127.216.104/api/1.6/volume"
time="2020-11-16T13:12:49Z" level=debug msg="Received response" response="[{"name": "lab_storage_ose-pso-db_0000000000", "created": "2020-11-13T21:02:52Z", "source": null, "serial": "63D6BEEE7DF5491400011991", "size": 34359738368}, {"name": "lab_storage_ose-pso-db_0000000001", "created": "2020-11-13T21:02:52Z", "source": null, "serial": "63D6BEEE7DF5491400011992", "size": 34359738368}, {"name": "lab_storage_ose4-pso-db_0000000000", "created": "2020-11-13T21:32:03Z", "source": null, "serial": "63D6BEEE7DF5491400011993", "size": 34359738368}, {"name": "lab_storage_ose4-pso-db_0000000001", "created": "2020-11-13T21:32:03Z", "source": null, "serial": "63D6BEEE7DF5491400011994", "size": 34359738368}, {"name": "lab_storage_ose4-pso-db_0000000002", "created": "2020-11-13T21:32:03Z", "source": null, "serial": "63D6BEEE7DF5491400011995", "size": 34359738368}, {"name": "ose_np_pure-pso-db_0000000000", "created": "2020-11-14T18:46:...
time="2020-11-16T13:12:49Z" level=debug msg="Attempting GET" path=/array
time="2020-11-16T13:12:49Z" level=debug msg="Sending request:" body= method=GET url="https://10.127.216.104/api/1.6/array"
time="2020-11-16T13:12:49Z" level=debug msg="Received response" response="{"revision": "202006250209+12678a3", "version": "5.3.8", "array_name": "OPD50R2-A1K-2012D", "id": "63d6beee-7df5-4914-8128-e8d84bd929f9"}" status="200 OK"
time="2020-11-16T13:12:49Z" level=debug msg="Attempting GET" path=/array
time="2020-11-16T13:12:49Z" level=debug msg="Sending request:" body= method=GET url="https://10.127.216.104/api/1.6/array?controllers=true"
time="2020-11-16T13:12:49Z" level=debug msg="Received response" response="[{"status": "ready", "name": "CT0", "version": "5.3.8", "mode": "secondary", "model": "FA-m50r2"}, {"status": "ready", "name": "CT1", "version": "5.3.8", "mode": "primary", "model": "FA-m50r2"}]" status="200 OK"
time="2020-11-16T13:12:49Z" level=debug msg="Discovered replica backends" responsiveBackends="map[10.127.216.104:FlashArray{name: OPD50R2-A1K-2012D, id: 63d6beee-7df5-4914-8128-e8d84bd929f9, mgmtEndPoint: https://10.127.216.104/api/1.6, labels:[access-mode.purestorage.com/ROX=false,access-mode.purestorage.com/RWO=true,access-mode.purestorage.com/RWX=true,protection.purestorage.com/snapshot=true,purestorage.com/backend=block,purestorage.com/family=FlashArray,purestorage.com/hostname=OPD50R2-A1K-2012D,purestorage.com/id=63d6beee-7df5-4914-8128-e8d84bd929f9,purestorage.com/model=FA-m50r2,purestorage.com/version=5.3.8]} 10.127.216.107:FlashArray{name: OPD50R2-A1K-1009A, id: e36ba64f-7c70-4d60-9744-37a8b783590a, mgmtEndPoint: https://10.127.216.107/api/1.6, labels:[access-mode.purestorage.com/ROX=false,access-mode.purestorage.com/RWO=true,access-mode.purestorage.com/RWX=true,protection.purestorage.com/snapshot=true,purestorage.com/backend=block,purestorage.com/family=FlashArray,purestorage.com/hostname=OPD50R2-...
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=info msg="not reporting volume with non-nil decommission phase" reconciler_name=dbdeployer_controller
time="2020-11-16T13:12:49Z" level=debug msg="Discovered replicas" activeReplicas="[]" desiredReplicaCount=5 startingReplicas="[]" unhealthyReplicas="[]"
time="2020-11-16T13:12:49Z" level=debug msg="Burst-starting replicas"
time="2020-11-16T13:12:49Z" level=debug msg="Applying replicas" replicas="[{{e36ba64f-7c70-4d60-9744-37a8b783590a pso-db_0000000006} c8cbbbd0-6a5f-d34a-14af-7006929e4ab0 FlashArray 10.127.216.107 } {{e36ba64f-7c70-4d60-9744-37a8b783590a pso-db_0000000007} c8cbbbd0-6a5f-d34a-14af-7006929e4ab0 FlashArray 10.127.216.107 } {{63d6beee-7df5-4914-8128-e8d84bd929f9 pso-db_0000000015} b628d192-4803-4357-97f2-7dcf0e16450d FlashArray 10.127.216.104 } {{63d6beee-7df5-4914-8128-e8d84bd929f9 pso-db_0000000016} b628d192-4803-4357-97f2-7dcf0e16450d FlashArray 10.127.216.104 } {{63d6beee-7df5-4914-8128-e8d84bd929f9 pso-db_0000000017} b628d192-4803-4357-97f2-7dcf0e16450d FlashArray 10.127.216.104 }]"
time="2020-11-16T13:12:49Z" level=debug msg="Result of reconcile iteration" error="" result=BurstStart

Cannot use imported volume

Imported volume created as per the doc

PV and PVC created and bound.

when pod is deploying it triggers

  Normal   Scheduled               <unknown>             default-scheduler         Successfully assigned dna-backup/rsync-ssh-677b8c779-8b56x to mdc-k8s-worker5
  Normal   SuccessfulAttachVolume  7m48s                 attachdetach-controller   AttachVolume.Attach succeeded for volume "pv-dnabackup"
  Warning  FailedMount             69s (x11 over 7m39s)  kubelet, mdc-k8s-worker5  MountVolume.MountDevice failed for volume "pv-dnabackup" : rpc error: code = Internal desc = exit status 32
  Warning  FailedMount             69s (x3 over 5m45s)   kubelet, mdc-k8s-worker5  Unable to attach or mount volumes: unmounted volumes=[dnabackup], unattached volumes=[dnabackup default-token-9zqgp]: timed out waiting for the condition

Used to work with 5.x.

PV and PVC have been deleted and recreated as per 6.x doc.

clusterID characters allowed

The values.yaml states that clusterID allows "alphanumeric and underscores".

In PSO5 our namespace.pure value is "pure-csi". We are forced to re-use that value in the clusterID field.

Will the dash in "pure-csi" cause any problems?

HashiCorp Nomad Support

This is less of an issue, and more of a series of questions (with implicit improvement suggestions).

CSI support has been available in HashiCorp Nomad (as a beta feature) for almost a year.
My understanding is that it is also available in at least a couple other orchestrators (Mesos, Cloud Foundry).

I haven't tried setting up pso-csi in our Nomad environment, but presumably 5.x should just work (after reading through all the YAML and translating things into job specs for the controller and node services).

With the advent of pso-csi 6.x, Pure's CSI plugin is no longer stateless.
Is it possible to leverage a manually deployed/managed cockroach DB cluster, or does the architecture of pso-csi 6.x require that the plugin be able to directly manage the cockroach cluster via K8S.
That is, is pso-csi a CSI plugin with an external metadata storage requirement; or is pso-csi actually dependent on Kubernetes APIs?

Is there any hope for running pso-csi 6.x+ in a non-Kubernetes environment with CSI plugin support available?
If so, is there any intent on making this an officially supported deployment option?

Does anybody reading this have examples of pso-csi 5.x or 6.x working with Nomad in the wild?
If so, are you willing to contribute examples to the documentation?

Release 6.1.1

v6.1.1 (04/12/2021)

Bug fixes and enhancements:

[NSTK-2256] Add missing toleration to db pods
[NSTK-2259] Support DB pods StatefulSet node-affinity and topology
#90 Enhance PSO log debug level
#141 Set Default Cockroach DB version to v20.2.6

SmartAgent permission issue in OpenShift 4.7

Executing the smart-agent script fails the ping command with permission denied.

If I log into one of the csi-node pods smart-config container and try the command manually this is the error I get:

/ # ping pso-db-1-0.pso-db.pso-csi -w 2
PING pso-db-1-0.pso-db.pso-csi (10.134.2.16): 56 data bytes
ping: permission denied (are you root?)

Unknown if this error occurs on earlier versions of OpenShift.

FlashBlade NFS 4.1 Support and FB-NFS Access Control via Export Rules

Per request by PSO customers:

Customers want to enable NFS 4.1 for FlashBlade to achieve better performance and mount NFS the way they want via mount options (rw/ro, hard/soft, fg/bg, vers).
Customers want to enable the FB-NFS Access Control via Export Rules that they can define their own CIDR access range and different permission for volumes.

pso-csi-node pods restarts frequently

$ kubectl get pods -o wide -n storage
NAME                                         READY   STATUS    RESTARTS   AGE     IP             NODE            NOMINATED NODE   READINESS GATES
pso-csi-controller-0                         6/6     Running   6          6h1m    172.20.2.53    init22b-8       <none>           <none>
pso-csi-node-24qd8                           2/3     Running   53         6h1m    10.64.112.52   k8s-worker-02   <none>           <none>
pso-csi-node-mfjg7                           3/3     Running   0          6h1m    10.64.230.41   init22b-8       <none>           <none>
pso-csi-node-ncqmn                           3/3     Running   0          6h1m    10.64.230.40   init22b-7       <none>           <none>
pso-csi-node-p6b97                           2/3     Running   56         6h1m    10.64.112.51   k8s-worker-01   <none>           <none>
pso-db-0-0                                   1/1     Running   0          6h1m    172.20.1.47    init22b-7       <none>           <none>
pso-db-1-0                                   1/1     Running   0          6h1m    172.20.2.54    init22b-8       <none>           <none>
pso-db-2-0                                   1/1     Running   0          6h1m    172.20.3.4     k8s-worker-01   <none>           <none>
pso-db-3-0                                   1/1     Running   0          6h1m    172.20.4.5     k8s-worker-02   <none>           <none>
pso-db-4-0                                   1/1     Running   0          6h1m    172.20.1.48    init22b-7       <none>           <none>
pso-db-cockroach-operator-5f494b6b5b-wpskw   1/1     Running   0          6h1m    172.20.1.45    init22b-7       <none>           <none>
pso-db-deployer-5df48fb79b-5hj82             1/1     Running   0          6h1m    172.20.1.46    init22b-7       <none>           <none>
pso-explorer-84d6cf69f9-lgqjc                2/2     Running   0          5h35m   172.20.2.57    init22b-8       <none>           <none>
psox-agent-2bj4j                             1/1     Running   0          5h51m   172.20.3.5     k8s-worker-01   <none>           <none>
psox-agent-2vx4x                             1/1     Running   0          5h51m   172.20.4.7     k8s-worker-02   <none>           <none>
psox-agent-pc5bh                             1/1     Running   0          5h51m   172.20.2.56    init22b-8       <none>           <none>
psox-agent-z2hnz                             1/1     Running   0          5h51m   172.20.1.49    init22b-7       <none>           <none>
psox-agent-zqlnl                             1/1     Running   1          5h51m   172.20.0.9     k8s-master-01   <none>           <none>

node-driver-registrar

W0508 00:42:08.409027   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:42:18.409174   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:42:28.409061   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:42:38.409045   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:42:48.409081   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:42:58.408966   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:43:08.408997   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:43:18.409026   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:43:28.409013   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
W0508 00:43:38.409098   56628 connection.go:170] Still connecting to unix:///csi/csi.sock
I0508 00:43:42.144113   56628 node_register.go:51] Starting Registration Server at: /registration/pure-csi-reg.sock
I0508 00:43:42.144325   56628 node_register.go:60] Registration Server started at: /registration/pure-csi-reg.sock
I0508 00:43:43.563343   56628 main.go:77] Received GetInfo call: &InfoRequest{}
I0508 00:43:43.584165   56628 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
E0508 00:49:40.845200   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 01:00:54.861780   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 01:05:12.763918   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 01:14:36.757668   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 01:33:18.765391   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 02:28:59.840505   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 02:46:42.755703   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.
E0508 02:57:45.872492   56628 connection.go:129] Lost connection to unix:///csi/csi.sock.

pso-csi-container

Original NodeID: k8s-worker-01
processIPHost
time="2021-05-08T06:50:10Z" level=error msg="Failed to retrieve configmap from cluster." configmap="&ConfigMap{ObjectMeta:{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Data:map[string]string{},BinaryData:map[string][]byte{},}" configmap_name=pure-csi-container-configmap error="an empty namespace may not be set when a resource name is provided"
time="2021-05-08T06:50:11Z" level=info msg="Starting server" debug=false endpoint="unix:///csi/csi.sock" nodeID=k8s-worker-01 span="{}"
time="2021-05-08T06:50:11Z" level=info msg="Created listener" addr=//csi/csi.sock protocol=unix span="{}"
time="2021-05-08T06:50:11Z" level=info msg="Listening for connections" address=//csi/csi.sock network=unix span="{}"
time="2021-05-08T06:50:11Z" level=info msg="Parsing ISCSI target ranges for CIDR filter." ISCSITargetRanges=
time="2021-05-08T06:50:11Z" level=info msg="CIDR Filters constructed." CIDRs="[]"
(/go/src/pso.purestorage.com/k8s/vendor/pso.purestorage.com/nsm/pkg/datastores/services/cockroach/collection.go:166)
[2021-05-08 06:50:12]  sql: no rows in result set 
time="2021-05-08T06:51:00Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-05-08T06:51:00Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-05-08T06:51:00Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}"
time="2021-05-08T06:51:02Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-05-08T06:51:02Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-05-08T06:51:02Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}"
time="2021-05-08T06:51:04Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-05-08T06:51:04Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-05-08T06:51:04Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}"
time="2021-05-08T06:51:06Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-05-08T06:51:06Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-05-08T06:51:06Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}"
time="2021-05-08T06:51:08Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-05-08T06:51:08Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-05-08T06:51:08Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}"
time="2021-05-08T06:51:10Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-05-08T06:51:10Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-05-08T06:51:10Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}"
time="2021-05-08T06:51:12Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1

[Question] Deploy PSO on control plane / master nodes ?

Some examples of pod scheduling mentions the master nodes as appropriate nodes to run PSO.
https://github.com/purestorage/pso-csi/blob/master/docs/csi-controller-attach-detach.md

Being a component of the control-plane, this makes sense. Is this considered as a best practice, assuming the nodes are capable of hosting them ?

I tend not to mix K8s components with other's on master nodes, and PSO is quite a heavy workload with its cockroachDB.
My clusters are too small (8 nodes) to imagine a (worker) node pool designated to host just that
My worker nodes would be glad to save some memory for user workloads rather than a CSI driver.

Any insights on what's the best practice ?

RWX on FlashArray

PSO docs indicate that FlashArrays can be used for RWX accessmode on PVCs. Does this require a different storageClass be created? My pvc creation failed with this error:
"failed to provision volume with StorageClass "pure-block": rpc error: code = InvalidArgument desc = mount volume type with block backend and read-write-many access mode is not supported"

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-nginx-rwx
spec:
storageClassName: pure-block
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi

pure-block StorageClass:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
csi.storage.k8s.io/fstype: xfs
meta.helm.sh/release-name: pure-pso
meta.helm.sh/release-namespace: default
creationTimestamp: "2021-04-15T20:45:35Z"
labels:
app.kubernetes.io/managed-by: Helm
chart: pure-pso
generator: helm
kubernetes.io/cluster-service: "true"
release: pure-pso
name: pure-block
resourceVersion: "104787191"
uid: 0a64621a-0574-4671-a9fc-660273dc0bd9
mountOptions:

discard
parameters:
backend: block
createoptions: -q
csi.storage.k8s.io/fstype: xfs
provisioner: pure-csi
reclaimPolicy: Delete
volumeBindingMode: Immediate

[Question] CockroachDB version

This chart is using cockroach 19.2.3 which is a year old.

During testing of a supposed memory-leak in a previous issue #91 (which was not, limits had to be set), we tried more recent image of the DB.

It was not part of the resolution but I cannot downgrade to default version of this chart since I migrated to 20.x.

Forcing this image is still OK with current version of the chart.

I wonder if you plan to update the version used ?

Support for Sharing a Volume Among Different Kubernetes Namespaces

Requirement

We have a requirement to share a volume with different PODs in different Kubernetes namespaces.

Problem

In Kubernetes, a volume can only be shared within one namespace by using PersistentVolumeClaim. It is not easy to share a purestorage volume among different namespaces.

Can you provide an official way to fulfill this requirement?

Workaround

Currently, we are using some workaround to share the same purestorage among different namespaces. Is there any problem with it?

Create a PersistentVolumeClaim in one namespace. Because we already have purestorage Storage Class being configured. a new PV and purestorage volume are provisioned.
run "kubectl get -o yaml <the_new_pv_name>" on the new PV
the output would look like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv1
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 2Gi
  persistentVolumeReclaimPolicy: Retain
  flexVolume:
    driver: pure/flex-norelabel
    fsType: nfs
    options:
      accessMode: exclusive
      backend: file
      labels: ""
      namespace: k8s
      volumeMode: ""
      volumeName: pvc-xxxx
  storageClassName: pure

create a new PV in a different namespace with same flexVolume settings retrieved from the first PV.
because this new PV have the same flexVolume setting. It will actually use the same purestorage volume.

Debug Level Log

Default in values.yaml is:

app:
  debug: false

Still the container is very talky with debug level messages:

pso-csi-container time="2020-09-30T11:27:49Z" level=debug msg="executing atomic update" collection=nodecollection.Node filter="id = '150ebce8-754f-4a63-8346-c9e8c8dd630c'"
--

Release v6.0.5

bug report email address not reachable

We're writing to let you know that the group you tried to contact (pso-tses) may not exist, or you may not have permission to post messages to the group. A few more details on why you weren't able to post:

You might have spelled or formatted the group name incorrectly.
The owner of the group may have removed this group.
You may need to join the group before receiving permission to post.
This group may not be open to posting.

If you have questions related to this or any other Google Group, visit the Help Center at https://support.google.com/a/purestorage.com/bin/topic.py?topic=25838.

Thanks,

purestorage.com admins

PSO v6.2.1 Release

PSO v6.2.1 Release

Fixes and improvements:

Fix KB unit issue for NodeGetVolumeStats CSI call
Support FlashBlade Safe Mode.
Upgrade apiextensions.k8s.io API version from v1beta1 to v1.
#196
Upgrade CSIDriver apiVersion from v1beta1 to v1
FibreChannel + NVME bug fixes.
Fix volume resize failures

Release v6.0.3

v6.0.3 (10/07/2020)

Enhancements:

Enabled support to import snapshots from Pure Flash Array(FA). This applies to snapshots that were created directly on the Flash Array outside of K8S
Added support for Platform9.
Improved handling of NFS versions and support for mounting volumes that were provisioned prioir to NFS v4.1 upgrade
Fixed race condition issue in cockrachdb operator that created more pods than needed
Fixed the bug: log level in values.yaml not honored
Support for REST API v2.0

FlashArray IOPS and Bandwidth QoS Support

See this design doc for more details.

Unable to attach or mount volumes

Hi! How are you guys?!! I have an error when trying to attach a volume to a pod.

Although it shows this...

fernando@devvm02:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
pvc-1cd6e909-bf32-448b-83bf-accea73d57b8   1Gi        RWO            Delete           Bound    default/minio-pvc               pure-block              6m31s
pvc-c21207b8-f160-4982-8cdb-5021e55dbb59   10Gi       RWO            Delete           Bound    gitea/data-gitea-0              pure-block              78m
pvc-c634e890-122f-4260-aa3c-4317ed58a562   10Gi       RWO            Delete           Bound    gitea/data-gitea-postgresql-0   pure-block              78m

and this..

fernando@devvm02:~$ kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
minio-pvc            Bound    pvc-1cd6e909-bf32-448b-83bf-accea73d57b8   1Gi        RWO            pure-block     7m12s

Can't mount it on the pod

Events:
  Type     Reason              Age                  From                     Message
  ----     ------              ----                 ----                     -------
  Normal   Scheduled           6m36s                default-scheduler        Successfully assigned default/minio-deployment-pure-8f9d74c6d-dlrxr to rocky01
  Warning  FailedAttachVolume  34s (x3 over 4m36s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-1cd6e909-bf32-448b-83bf-accea73d57b8" : Attach timeout for volume b3b270fb-74d6-4ac9-9e2b-86017c9d80ae
  Warning  FailedMount         1s (x3 over 4m33s)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[storage kube-api-access-pgdjj]: timed out waiting for the condition

Client Version: v1.22.3
Server Version: v1.22.2
Flannel: quay.io/coreos/flannel:v0.14.0
iscsi multipath installed.
Kubeadm on Rocky Linux release 8.4 (Green Obsidian)
Helm version : pure/pure-pso	6.2.1

Pods

NAME                                         READY   STATUS    RESTARTS   AGE
pso-csi-controller-0                         6/6     Running   0          38h
pso-csi-node-f9znt                           3/3     Running   0          38h
pso-csi-node-hzh4n                           3/3     Running   0          13h
pso-csi-node-j7h82                           3/3     Running   0          38h
pso-csi-node-mlwqb                           3/3     Running   0          38h
pso-csi-node-wsk4r                           3/3     Running   0          38h
pso-db-0-0                                   1/1     Running   0          38h
pso-db-1-0                                   1/1     Running   0          38h
pso-db-2-0                                   1/1     Running   0          38h
pso-db-3-0                                   1/1     Running   0          38h
pso-db-4-0                                   1/1     Running   0          38h
pso-db-cockroach-operator-7c67c87c67-wsvm9   1/1     Running   0          38h
pso-db-deployer-5b4d466485-wg6dl             1/1     Running   0          38h

pso-csi-controller-0 extract log

time="2021-11-18T02:59:38Z" level=error msg="waiters found" bucket= req="(*csi.ProbeRequest) " span="{}" waiters=1
time="2021-11-18T02:59:38Z" level=error msg="failed to queue request" error="rpc error: code = Aborted desc = requests pending" req="(*csi.ProbeRequest) " span="{}"
time="2021-11-18T02:59:38Z" level=error msg="GRPC handler error" error="rpc error: code = Aborted desc = requests pending" method=/csi.v1.Identity/Probe request= span="{}

pso-csi-controller-0 describe pod

Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  5m9s (x34860 over 38h)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  9s (x34879 over 38h)    kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500

I attached all the logs and send it to [email protected]

Could you help me please? Thanks in advance!!!

Readiness probe failed with connection refused

Hi, I've tried deploying PSO to verify function with our //FA M20, but faced "Readiness probe failed" and some PSO pods doesn't boot well.
May I have any advice?

Out environment

PSO 6.0.5
CentOS Linux release 7.9.2009 (Core)
-3master, 3worker of 6 nodes, and 1 HAproxy node for control plain
-Kubernetes Ver.
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2",
GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z",
GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7",
GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z",
GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
pod network: Calico with default values
https://docs.projectcalico.org/manifests/calico.yaml

I got several output of kubectl command.
kubectl.log

Issued "helm install" with following values.yaml.
values.yaml.log

Best way to redeploy/refresh pso-dbs?

I have updated my values.yaml to include resource requests for my pso-db's (which are currently using 70% of the clusters memory), and then ran a helm upgrade.

What is the safest way to trigger the pso-db redeployment with new resource request suggestions?

[Enhancement] List chart on artifacthub

Currently the chart is not listed on artifacthub.
Given the fact that some registries (e.g. harbor) impliment their sync with artifacthub it would be nice to have the chart there.

Question: Import pv/pvc created with 5.x

When installing PSO 6.x, shall we recreate every volume ?

Also:
despite what the doc says here pure storage class is not created by default anymore. Nor is storageClass: available in values.yaml (documented here but not there)

I have recreated the storage class manually.

I have a bunch of errors but for some reasons volume is actually attached and pod is working:

AttachVolume.Attach failed for volume "pvc-b74531d6-2b86-4a4c-9bce-8c4ad1f79cf3" : rpc error: code = NotFound desc = Failed to import backend volume pure-csi/pvc-b74531d6-2b86-4a4c-9bce-8c4ad1f79cf3, not found

Question: pso reinstallation

Question:

What happens if we delete completely the pso helm charts and leave existing volumes, and then reinstall the helm chart (same version), do we recover our setup ?

unpublish job for non-existing node get recreated

We deploy our clusters using CAPI, thus our node names are ephermal.
Im currently noticing an unpublish job, that wants to run on a node that already got deleted and im unable to get reid of it.
If i delete the job, PSO just recreates it.
If i check on ly storage arrays, the volume it tries to unmount doesnt have a mount listed.

So, how do i get rid of that job.

Config option flasharray.iSCSIAllowedCIDR is ignored

I installed the pso-csi by helm (helm install pure-pso pure/pure-pso --namespace pure -f values.yaml) and configured the values.yaml as follow:

...
flasharray:
  iSCSIAllowedCIDR: 10.4.1.0/26
...

But on some containers, which fail to start, I still can see that those containers try to connect to other target which are not in the allowed list above:

time="2021-04-22T09:01:01Z" level=debug msg="Running command" args="[--mount=/proc/1/ns/mnt --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net -- iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.1234567812345678 -p 10.3.0.14:3260 --login]" command=nsenter timeout=20
time="2021-04-22T09:01:01Z" level=debug msg="Running command" args="[--mount=/proc/1/ns/mnt --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net -- iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.1234567812345678 -p 10.3.0.15:3260 --login]" command=nsenter timeout=20
time="2021-04-22T09:01:01Z" level=debug msg="Running command" args="[--mount=/proc/1/ns/mnt --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net -- iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.1234567812345678 -p 10.3.0.16:3260 --login]" command=nsenter timeout=20
time="2021-04-22T09:01:01Z" level=debug msg="Running command" args="[--mount=/proc/1/ns/mnt --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net -- iscsiadm -m node -T iqn.2010-06.com.purestorage:flasharray.1234567812345678 -p 10.3.0.17:3260 --login]" command=nsenter timeout=20

The overall status of the pure pso-csi pods are as follow:

kubectl get pods -n pure
NAME                                         READY   STATUS                  RESTARTS   AGE
pso-csi-controller-0                         5/6     Running                 7          30m
pso-csi-node-r6vbf                           2/3     Running                 7          30m
pso-csi-node-vv9mj                           2/3     Running                 7          30m
pso-db-0-0                                   0/1     Running                 0          30m
pso-db-1-0                                   0/1     Running                 0          30m
pso-db-2-0                                   0/1     Init:CrashLoopBackOff   6          30m
pso-db-3-0                                   0/1     Init:CrashLoopBackOff   7          30m
pso-db-4-0                                   0/1     Init:CrashLoopBackOff   6          30m
pso-db-cockroach-operator-5867dc494d-l8vxf   1/1     Running                 0          30m
pso-db-deployer-6cff94ff4d-8txsh             1/1     Running                 0          30m

It's quiet interesting that on all containers the ENV variable isn't set properly:

kubectl describe pods pso-db-0-0 -n pure|grep PURE_ISCSI_ALLOWED_CIDRS:
      PURE_ISCSI_ALLOWED_CIDRS:

kubectl describe pods pso-db-2-0 -n pure|grep PURE_ISCSI_ALLOWED_CIDRS:
      PURE_ISCSI_ALLOWED_CIDRS:

Any help is highly appreciated.

databases tolerations not added for pso-db pods

When we set the values.yaml with this settings:

database:
  # nodeSelector is the simplest way to limit which kubernetes nodes will run the database-related pods
  # Please refer to the top-level description of nodeSelector for an example
  nodeSelector:
    node.is.dtr: "yes"
  tolerations:
    - operator: Exists
      effect: "NoSchedule"
    - operator: Exists
      effect: "NoExecute"
    - operator: Exists
      effect: "NoSchedule"
      key: "com.docker.ucp.manager"

The tolerations doesn't add on pso-db pods. So we cannot schedule pso-db on some nodes.

We currently used the charts from master branch.

[Bug] PSO DB scheduler schedules pods which will never join the cluster.

Hey got an issue with PSO on

K8s: v.1.19.4
PSO: v6.0.4

The db scheduler goes wild and creates a bunch of statefulsets.
The issue is that they are not able to join the existing cluster and stay in a not running state.
Is that wanted behavior?

pso-db-0-0                                   1/1     Running   0          6d18h
pso-db-1-0                                   1/1     Running   0          6d18h
pso-db-10-0                                  0/1     Running   0          6d18h
pso-db-11-0                                  0/1     Running   0          6d18h
pso-db-12-0                                  0/1     Running   0          6d18h
pso-db-13-0                                  0/1     Running   0          6d18h
pso-db-14-0                                  0/1     Running   0          6d18h
pso-db-15-0                                  0/1     Running   0          6d17h
pso-db-16-0                                  0/1     Running   0          6d17h
pso-db-17-0                                  0/1     Running   0          6d17h
pso-db-18-0                                  0/1     Running   0          6d17h
pso-db-19-0                                  0/1     Running   0          6d17h
pso-db-2-0                                   0/1     Running   0          6d18h
pso-db-20-0                                  0/1     Running   0          6d16h
pso-db-21-0                                  0/1     Running   0          6d16h
pso-db-22-0                                  0/1     Running   0          6d16h
pso-db-23-0                                  0/1     Running   0          6d16h
pso-db-24-0                                  0/1     Running   0          6d16h
pso-db-25-0                                  0/1     Running   0          6d16h
pso-db-26-0                                  0/1     Running   0          6d16h
pso-db-27-0                                  0/1     Running   0          6d16h
pso-db-28-0                                  0/1     Running   0          6d16h
pso-db-29-0                                  0/1     Running   0          6d16h

pso 5 to 6 upgrade. namespace concern

I am planning an upgrade from PSO 5 to 6. I have read the warnings regarding the requirement to keep PSO in the same namespace when upgrading to PSO 6.1. Our challenge is that we specified 'pure-csi' as the namespace in the PSO 5 helm installation, but it installed in the default namespace.

[(default)]$ helm get values pure-storage-driver
USER-SUPPLIED VALUES:
arrays:
FlashArrays:
{ REDACTED }
namespace:
pure: pure-csi

[(default)]$ helm ls -n default
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
pure-storage-driver default 1 2020-08-17 21:26:44.021943316 +0000 UTC deployed pure-csi-1.2.0 1.2.0

[(default)]$ helm ls -n pure-csi
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
[(default)]$

Will this be a problem when we run the PSO 6 upgrade steps?

6.1.1 DB pod known issue clarification

Can you clarify this known issue from the 6.1.1 changelog:
"Database volume is mounted as read-only and result in CrashLoopBackOff status of a db pod."

When does that occur and will that put PSO's overall state in jeopardy?

[FB] Storage class to create volumes with hard limits ?

I could not find a parameter enabling hard limits on created volumes on flashblade, is it possible ?

prometheus metrics ?

Does latest version of PSO provide prometheus metrics of pvc usage ?

PSO Explorer don't offer such functionality, pso-analytics was but it's not maintained anymore

Thanks

'unable to parse env variables for psctl image' when following Dark-Site installation process

When following the Dark Site installation process and using mirror_pso_containers.sh I get the following error on the cockroach operator and the installation stops progressing. All of the images are successfully pulled from the internal registry.

$ oc get pods
NAME                                         READY   STATUS             RESTARTS   AGE
pso-csi-controller-0                         5/6     Running            5          3m24s
pso-csi-node-8zjm9                           2/3     Running            5          3m24s
pso-csi-node-dwzht                           2/3     Running            5          3m24s
pso-csi-node-td2rj                           2/3     Running            5          3m24s
pso-db-cockroach-operator-7d8469c9c8-smcpq   0/1     CrashLoopBackOff   5          3m24s
pso-db-deployer-778f97db5b-ckp8g             1/1     Running            0          3m24s

$ oc logs pso-db-cockroach-operator-7d8469c9c8-smcpq -n pure-pso                                                                         
{"level":"info","ts":1605907706.4262958,"logger":"cmd","msg":"","is_debug_mode":false}
{"level":"info","ts":1605907706.4263296,"logger":"cmd","msg":"Go Version: go1.15"}
{"level":"info","ts":1605907706.4263344,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1605907706.4263387,"logger":"cmd","msg":"Version of operator-sdk: v0.6.0"}
{"level":"error","ts":1605907706.426353,"logger":"cmd","msg":"unable to parse env variables for psctl image","error":"psctl image tag was specified as both gitlab.example.com:5005/oscp/pure-pso/psctl and v1.0.1","st
acktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/pso.purestorage.com/cockroach-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nmain.parseEnv\n\t/go/src/pso.purestorage.com/cockroach-operator/
cmd/cockroach-operator/main.go:152\nmain.main\n\t/go/src/pso.purestorage.com/cockroach-operator/cmd/cockroach-operator/main.go:269\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}```

Failure to install with default values, pso-db pods missing

On a healthy cluster with prereqs fulfilled and default values used, pso-db-* pods don't get created, even though pso-db-cockroach-operator doesn't log errors apparently.

Any help will be appreciated.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9", GitCommit:"9dd794e454ac32d97cde41ae10be801ae98f75df", GitTreeState:"clean", BuildDate:"2021-03-18T01:09:28Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9", GitCommit:"9dd794e454ac32d97cde41ae10be801ae98f75df", GitTreeState:"clean", BuildDate:"2021-03-18T01:00:06Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

$ helm ls -n cloud-system
pure-pso                cloud-system    1               2021-10-21 17:18:17.102638878 -0400 EDT deployed        pure-pso-6.2.0                  6.2.0

$ helm get values --namespace=cloud-system pure-pso
USER-SUPPLIED VALUES:
DBTopology:
  enforce: false
affinity: {}
app:
  debug: false
arrays:
  FlashArrays:
  - APIToken: xxxxxxxxxx
    MgmtEndPoint: xxxxxx
  FlashBlades: []
clusterID: xxxxxxxxx
clusterrolebinding:
  serviceAccount:
    name: pure
controllerServer:
  affinity: {}
  nodeSelector: {}
  tolerations: []
database:
  affinity: {}
  maxStartupSeconds: 600
  maxSuspectSeconds: 3600
  nodeSelector: {}
  resources:
    limits:
      memory: 1Gi
  tolerations: []
flasharray:
  defaultFSOpt: -q
  defaultFSType: xfs
  defaultMountOpt:
  - discard
  iSCSIAllowedCIDR: ""
  iSCSILoginTimeout: 20
  preemptAttachments: "true"
  sanType: ISCSI
flashblade:
  exportRules: '*(rw,no_root_squash)'
  snapshotDirectoryEnabled: "false"
images:
  csi:
    attacher:
      name: quay.io/k8scsi/csi-attacher
      pullPolicy: Always
    livenessProbe:
      name: quay.io/k8scsi/livenessprobe
      pullPolicy: Always
    nodeDriverRegistrar:
      name: quay.io/k8scsi/csi-node-driver-registrar
      pullPolicy: Always
    provisioner:
      name: quay.io/k8scsi/csi-provisioner
      pullPolicy: Always
    resizer:
      name: quay.io/k8scsi/csi-resizer
      pullPolicy: Always
    snapshotter:
      name: quay.io/k8scsi/csi-snapshotter
      pullPolicy: Always
  database:
    cockroachOperator:
      name: purestorage/cockroach-operator
      pullPolicy: Always
      tag: v1.2.0
    cockroachdb:
      name: cockroachdb/cockroach
      tag: v20.2.6
    deployer:
      name: purestorage/dbdeployer
      pullPolicy: Always
      tag: v1.2.0
    psctl:
      name: purestorage/psctl
      tag: v1.1.0
  plugin:
    name: purestorage/k8s
    pullPolicy: Always
    tag: v6.2.0
nodeSelector: {}
nodeServer:
  affinity: {}
  nodeSelector: {}
  tolerations: []
orchestrator:
  basePath: /var/lib/kubelet
  name: k8s
storagetopology:
  enable: false
  strictTopology: false
tolerations: []
upgrade: false

$ kubectl get pod -n cloud-system | grep pso
pso-csi-controller-0                                     5/6     Running   0          3m26s
pso-csi-node-4rtll                                       2/3     Running   0          3m26s
pso-csi-node-87mgj                                       2/3     Running   0          3m26s
pso-csi-node-9mrzs                                       2/3     Running   0          3m26s
pso-csi-node-pgkz9                                       2/3     Running   0          3m26s
pso-csi-node-wr24z                                       2/3     Running   0          3m26s
pso-db-cockroach-operator-7544d54f88-rddx2               1/1     Running   0          3m26s
pso-db-deployer-6748ddc94-zrclh                          1/1     Running   0          3m26s

$ kubectl logs -n cloud-system pso-db-cockroach-operator-7544d54f88-rddx2
{"level":"info","ts":1634851114.5586638,"logger":"cmd","msg":"","is_debug_mode":false}
{"level":"info","ts":1634851114.5587416,"logger":"cmd","msg":"Go Version: go1.15.12"}
{"level":"info","ts":1634851114.5587597,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1634851114.558776,"logger":"cmd","msg":"Version of operator-sdk: v0.6.0"}
{"level":"info","ts":1634851114.5588267,"logger":"cmd","msg":"Setting max startup duration to 600 seconds"}
{"level":"info","ts":1634851114.5588372,"logger":"cmd","msg":"Setting max suspect duration to 3600 seconds"}
{"level":"info","ts":1634851114.5588984,"logger":"cmd","msg":"Did not find config file for affinity"}
{"level":"info","ts":1634851114.5589225,"logger":"cmd","msg":"Successfully parsed config file","file":"affinity","value":"&Affinity{NodeAffinity:nil,PodAffinity:nil,PodAntiAffinity:nil,}"}
{"level":"info","ts":1634851114.5635457,"logger":"cmd","msg":"Did not find config file for tolerations"}
{"level":"info","ts":1634851114.5637438,"logger":"cmd","msg":"Successfully parsed config file","file":"tolerations","value":null}
{"level":"info","ts":1634851114.564393,"logger":"cmd","msg":"Successfully parsed config file","file":"resources","value":"&ResourceRequirements{Limits:ResourceList{memory: {{1073741824 0} {<nil>} 1Gi BinarySI},},Requests:ResourceList{},}"}
{"level":"info","ts":1634851114.5646725,"logger":"cmd","msg":"Did not find config file for nodeSelector"}
{"level":"info","ts":1634851114.5646825,"logger":"cmd","msg":"Successfully parsed config file","file":"nodeSelector","value":{}}
{"level":"info","ts":1634851114.565181,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1634851115.1553597,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1634851115.1731136,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1634851115.4093795,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1634851115.4163942,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"intrusion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1634851115.4169033,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"intrusion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1634851115.4232342,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"intrusion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1634851115.424368,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"intrusion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1634851115.42497,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"intrusion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1634851115.4254565,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"intrusion-controller","source":"kind source: /, Kind="}
time="2021-10-21T21:18:35Z" level=info msg="Successfully started watches." reconciler_name=controller_intrusion
{"level":"info","ts":1634851115.8073902,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"cockroach-operator","Service.Namespace":"cloud-system"}
{"level":"info","ts":1634851115.8074362,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1634851116.115943,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"intrusion-controller"}
{"level":"info","ts":1634851116.2169046,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"intrusion-controller","worker count":1}

$ kubectl get intrusion --all-namespaces
No resources found

$ kubectl logs -n cloud-system pso-db-deployer-6748ddc94-zrclh 
time="2021-10-21T21:18:35Z" level=info is_debug_mode=false name=dbdeployer_cmd
time="2021-10-21T21:18:35Z" level=info msg="version info" OS=linux arch=amd64 go=go1.15.12 name=dbdeployer_cmd
{"level":"info","ts":1634851115.0486882,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1634851115.4552867,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1634851115.4713528,"logger":"leader","msg":"Became the leader."}
time="2021-10-21T21:18:35Z" level=info msg="Registering Components." name=dbdeployer_cmd
{"level":"info","ts":1634851115.8281531,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"dbdeployer-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1634851115.8284357,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"dbdeployer-controller","source":"kind source: /, Kind="}
time="2021-10-21T21:18:35Z" level=info msg="Starting the Cmd." name=dbdeployer_cmd
{"level":"info","ts":1634851116.0308127,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"dbdeployer-controller"}
{"level":"info","ts":1634851116.1313999,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"dbdeployer-controller","worker count":1}
time="2021-10-21T21:18:36Z" level=info msg="--------------------- Begin reconcile ---------------------" reconciler_name=dbdeployer_controller request=cloud-system/sh.helm.release.v1.topolvm.v2
time="2021-10-21T21:18:36Z" level=info msg="--------------------- Begin reconcile ---------------------" reconciler_name=dbdeployer_controller request=cloud-system/sh.helm.release.v1.topolvm.v5
time="2021-10-21T21:18:36Z" level=info msg="--------------------- Begin reconcile ---------------------" reconciler_name=dbdeployer_controller request=cloud-system/pure-provisioner-secret
time="2021-10-21T21:18:36Z" level=info msg="Discovery json validated successfully"
time="2021-10-21T21:18:36Z" level=warning msg="could not get intrusion" error="Intrusion.pso.purestorage.com \"pso-db\" not found" reconciler_name=dbdeployer_controller
time="2021-10-21T21:18:36Z" level=warning msg="could not get intrusion" error="Intrusion.pso.purestorage.com \"pso-db\" not found" reconciler_name=dbdeployer_controller
time="2021-10-21T21:18:36Z" level=error msg="unable to create initial intrusion: Intrusion.pso.purestorage.com \"pso-db\" is invalid: [status.underreplicatedRanges: Invalid value: \"null\": status.underreplicatedRanges in body must be of type integer: \"null\", status.asOf: Invalid value: \"null\": status.asOf in body must be of type string: \"null\", status.initialized: Invalid value: \"null\": status.initialized in body must be of type boolean: \"null\", status.readyNodes: Invalid value: \"null\": status.readyNodes in body must be of type string: \"null\", status.replicas: Invalid value: \"null\": status.replicas in body must be of type array: \"null\", status.status: Invalid value: \"null\": status.status in body must be of type string: \"null\", status.totalRanges: Invalid value: \"null\": status.totalRanges in body must be of type integer: \"null\", status.unavailableRanges: Invalid value: \"null\": status.unavailableRanges in body must be of type integer: \"null\"]" reconciler_name=dbdeployer_controller
time="2021-10-21T21:18:36Z" level=error msg="Failed applying replicas" error="Intrusion.pso.purestorage.com \"pso-db\" is invalid: [status.underreplicatedRanges: Invalid value: \"null\": status.underreplicatedRanges in body must be of type integer: \"null\", status.asOf: Invalid value: \"null\": status.asOf in body must be of type string: \"null\", status.initialized: Invalid value: \"null\": status.initialized in body must be of type boolean: \"null\", status.readyNodes: Invalid value: \"null\": status.readyNodes in body must be of type string: \"null\", status.replicas: Invalid value: \"null\": status.replicas in body must be of type array: \"null\", status.status: Invalid value: \"null\": status.status in body must be of type string: \"null\", status.totalRanges: Invalid value: \"null\": status.totalRanges in body must be of type integer: \"null\", status.unavailableRanges: Invalid value: \"null\": status.unavailableRanges in body must be of type integer: \"null\"]"

purestorage pso-db Init:CrashLoopBackOff

hello i am pure-storage user

Environment

k8s version: v1.20.10
node 3ea / master 1ea
os version : ubuntu 20.04.3
pso version : 6.2.0/6.0.1/6.0.5
cni : weave/clium/calico
maintenance : vsphere on vm
======================================================
Connect k8s and pure as a test.
but

kubectl get pod -n storage
pso-csi-controller-0 5/6 Running 3 2m9s
pso-csi-node-r6frf 2/3 Running 3 2m9s
pso-csi-node-sk6qp 2/3 Running 3 2m9s
pso-csi-node-vttgc 2/3 Running 3 2m9s
pso-db-0-0 0/1 Init:CrashLoopBackOff 4 118s
pso-db-1-0 0/1 Init:CrashLoopBackOff 4 118s
pso-db-2-0 0/1 Init:CrashLoopBackOff 4 118s
pso-db-3-0 0/1 Init:CrashLoopBackOff 4 118s
pso-db-4-0 0/1 Init:CrashLoopBackOff 4 118s
pso-db-cockroach-operator-65c697676-xlnwr 1/1 Running 0 2m9s
pso-db-deployer-649bcbf568-dqjbw 1/1 Running 0 2m9s

kubectl logs -n storage pso-csi-controller-0 -c pso-csi-container

time="2021-10-13T08:21:09Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.storage port=26257
user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt"
error="dial tcp: lookup pso-db-public.storage on 10.96.0.10:53: write udp 10.100.31.127:44716->10.96.0.10:53: write: operation not permitted"

time="2021-10-13T08:21:09Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"

time="2021-10-13T08:21:09Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=84235.33392566246 span="{}"

kubectl logs -n storage pso-csi-node-r6frf

time="2021-10-13T08:24:22Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.storage port=26257
user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt"
error="dial tcp: lookup pso-db-public.storage on 10.96.0.10:53: read udp 10.0.3.241:45466->10.96.0.10:53: i/o timeout"

time="2021-10-13T08:24:22Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"

time="2021-10-13T08:24:22Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=158.62037146736316 span="{}"

kubectl logs -n storage pso-db-0-0

Error from server (BadRequest): container "cockroachdb" in pod "pso-db-0-0" is waiting to start: PodInitializing

Not working in Openshift 4.5

I followed the instructions that you have. However it continues to fail the issue is:
on pso-db-cockroach-operator pod
msg="failed to get liveness for node" error="Get http://pso-db-13-0.pso-db.pure-storage-orchestrator:8080/health: dial tcp 192.168.5.28:8080: connect: connection refused"

on pso-csi-controller-0 pso-csi-container
host=pso-db-public.pure-storage-orchestrator port=26257 user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt" error="dial tcp 192.172.48.36:26257: connect: connection refused"
time="2020-09-01T15:35:27Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"
time="2020-09-01T15:35:27Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=2550.6642033801327 span="{}"
time="2020-09-01T15:35:30Z" level=debug msg="Creating a persistent store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}" func=0x136b970
time="2020-09-01T15:35:30Z" level=debug msg="Create a cockroach store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-09-01T15:35:31Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.pure-storage-orchestrator port=26257 user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt" error="dial tcp 192.172.48.36:26257: connect: connection refused"
time="2020-09-01T15:35:31Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"
time="2020-09-01T15:35:31Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=6552.449958720005 span="{}"
time="2020-09-01T15:35:37Z" level=debug msg="Creating a persistent store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}" func=0x136b970
time="2020-09-01T15:35:37Z" level=debug msg="Create a cockroach store service" config="&{true cockroach pure pso-db-public.pure-storage-orchestrator /cockroach/cockroach-certs map[]}"
time="2020-09-01T15:35:38Z" level=error msg="failed to connect to cockroach" addr="host=pso-db-public.pure-storage-orchestrator port=26257 user=root dbname=pure sslmode=require sslcert=/cockroach/cockroach-certs/client.root.crt sslkey=/cockroach/cockroach-certs/client.root.key sslrootcert=/cockroach/cockroach-certs/ca.crt" error="dial tcp 192.172.48.36:26257: connect: connection refused"
time="2020-09-01T15:35:38Z" level=error msg="Failed to instantiate persistent store service in readiness probe, not ready" error="failed to connect to cockroach"
time="2020-09-01T15:35:38Z" level=info msg="Probe indicates plugin is not ready yet, waiting and retrying" error="" sleep_ms=7417.006379185561 span="{}"

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.