helm deploy incomplete - backbeat dockers restarting about zenko

[ssalaues]

@patriziobassi Since we don't deploy in an ordered sequence of dependencies, typically you will see CrashLoopBackoff as a normal part of the installation until all the replicas of the stateful services stabilize. For 3 node clusters, the typical stabilization period is between 5-10 minutes (some managed clusters that have slower scheduling can take longer).

The specific error you are seeing is from the async engine Backbeat is typical of this service waiting on all 3 of the MongoDB replicas to startup and be available. Have you continued to see this error even after all MongoDB replicas are "Green"? I ask because in the snippet you posted I can only see 1 MongoDB still in the initializing state.

Edit:
Also what would be useful for further debugging is some more info on how you installed Zenko. Such as custom values added to the Helm chart installation (values.yaml changes, cli --set options, etc.

[patriziobassi]

Hi

of course i waited mongodb to be up and running, infact the 3 istances were deployed and running.
i have a 5 workers cluster, infact all the pods/containers become fully running in a few minutes, less than 10. then all the containers based on backbeat images start to restart, i waited a whole night and i got hundreds of restarts.

i just did a plain helm install as in the installation guide without any change in yml

[ssalaues]

@patriziobassi This is a strange issue and typically should never happen beyond the first few minutes of initialization. Can you post or update your original post with the current status of all the pods? This will help us better understand what is going on with your install.

[patriziobassi]

kubectl get pods
NAME                                                          READY   STATUS             RESTARTS   AGE
zenko-backbeat-api-5566bdfd68-w6bkq                           0/1     CrashLoopBackOff   2226       4d16h
zenko-backbeat-gc-consumer-574d5d69cf-gvg9m                   0/1     CrashLoopBackOff   2224       4d16h
zenko-backbeat-gc-consumer-574d5d69cf-m4xhf                   0/1     CrashLoopBackOff   2229       4d16h
zenko-backbeat-gc-consumer-574d5d69cf-p4ngk                   0/1     CrashLoopBackOff   2227       4d16h
zenko-backbeat-ingestion-consumer-db4b76459-4hskl             0/1     CrashLoopBackOff   2221       4d16h
zenko-backbeat-ingestion-consumer-db4b76459-4tzs8             0/1     CrashLoopBackOff   2216       4d16h
zenko-backbeat-ingestion-consumer-db4b76459-ww7ww             0/1     CrashLoopBackOff   2227       4d16h
zenko-backbeat-ingestion-producer-69589dfdc6-979g6            0/1     CrashLoopBackOff   2230       4d16h
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-8b6mq    0/1     CrashLoopBackOff   2228       4d16h
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-bp5rw    0/1     CrashLoopBackOff   2233       4d16h
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-jwb8r    0/1     CrashLoopBackOff   2232       4d16h
zenko-backbeat-lifecycle-conductor-65d5ccb9f6-pr4x9           1/1     Running            19         4d16h
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-88sz9    0/1     CrashLoopBackOff   2230       4d16h
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-nrgsn    0/1     CrashLoopBackOff   2219       4d16h
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-zpg7v    0/1     CrashLoopBackOff   2225       4d16h
zenko-backbeat-replication-data-processor-86f5dbcd9-7vbt6     0/1     CrashLoopBackOff   2224       4d16h
zenko-backbeat-replication-data-processor-86f5dbcd9-hwbhw     1/1     Running            2226       4d16h
zenko-backbeat-replication-data-processor-86f5dbcd9-m4qj6     0/1     CrashLoopBackOff   2214       4d16h
zenko-backbeat-replication-populator-749cfd8d59-m27zz         1/1     Running            20         4d16h
zenko-backbeat-replication-status-processor-b8cfd5785-2x652   1/1     Running            2222       4d16h
zenko-backbeat-replication-status-processor-b8cfd5785-4chcx   1/1     Running            2218       4d16h
zenko-backbeat-replication-status-processor-b8cfd5785-tzgl2   0/1     CrashLoopBackOff   2220       4d16h
zenko-cloudserver-dd76fb887-4wvcl                             1/1     Running            12         4d16h
zenko-cloudserver-dd76fb887-6cjsc                             1/1     Running            6          4d16h
zenko-cloudserver-dd76fb887-6svzz                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-6t8kp                             1/1     Running            6          4d16h
zenko-cloudserver-dd76fb887-6wlhq                             1/1     Running            13         4d16h
zenko-cloudserver-dd76fb887-74c5h                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-74c8w                             1/1     Running            9          4d16h
zenko-cloudserver-dd76fb887-7jpmc                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-88p95                             1/1     Running            5          4d16h
zenko-cloudserver-dd76fb887-8fx99                             1/1     Running            13         4d16h
zenko-cloudserver-dd76fb887-8shr7                             1/1     Running            6          4d16h
zenko-cloudserver-dd76fb887-9c7r4                             1/1     Running            6          4d16h
zenko-cloudserver-dd76fb887-9g2lg                             1/1     Running            7          4d16h
zenko-cloudserver-dd76fb887-bpdl8                             1/1     Running            7          4d16h
zenko-cloudserver-dd76fb887-ddvsc                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-dzpj7                             1/1     Running            6          4d16h
zenko-cloudserver-dd76fb887-h79w2                             1/1     Running            9          4d16h
zenko-cloudserver-dd76fb887-kg5hb                             1/1     Running            7          4d16h
zenko-cloudserver-dd76fb887-lp24g                             1/1     Running            14         4d16h
zenko-cloudserver-dd76fb887-mhjwb                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-mnxzb                             1/1     Running            14         4d16h
zenko-cloudserver-dd76fb887-nqzz4                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-q88tv                             1/1     Running            11         4d16h
zenko-cloudserver-dd76fb887-qvm5k                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-r85k6                             1/1     Running            8          4d16h
zenko-cloudserver-dd76fb887-vxlg9                             1/1     Running            6          4d16h
zenko-cloudserver-dd76fb887-wj4l2                             1/1     Running            5          4d16h
zenko-cloudserver-dd76fb887-x6zlw                             1/1     Running            5          4d16h
zenko-cloudserver-dd76fb887-z4qn2                             1/1     Running            10         4d16h
zenko-cloudserver-dd76fb887-zcg74                             1/1     Running            10         4d16h
zenko-cloudserver-manager-75c489df7-fkh4n                     1/1     Running            8          4d16h
zenko-cosmos-operator-5b86756ff7-plpd5                        1/1     Running            0          4d16h
zenko-cosmos-scheduler-5c96f7ff9d-k2jvd                       0/1     CrashLoopBackOff   1310       4d16h
zenko-grafana-fd8d85b84-q6wbx                                 3/3     Running            4          4d16h
zenko-mongodb-replicaset-0                                    2/2     Running            0          4d16h
zenko-mongodb-replicaset-1                                    2/2     Running            0          4d16h
zenko-mongodb-replicaset-2                                    2/2     Running            0          4d16h
zenko-prometheus-server-0                                     2/2     Running            0          4d16h
zenko-prometheus-server-1                                     2/2     Running            0          4d16h
zenko-redis-ha-server-0                                       2/2     Running            0          4d16h
zenko-redis-ha-server-1                                       2/2     Running            0          4d16h
zenko-redis-ha-server-2                                       2/2     Running            0          4d16h
zenko-s3-data-78f46ccd8d-dd4g5                                1/1     Running            0          4d16h
zenko-zenko-queue-0                                           2/2     Running            3          4d16h
zenko-zenko-queue-1                                           2/2     Running            0          4d16h
zenko-zenko-queue-2                                           2/2     Running            0          4d16h
zenko-zenko-queue-config-ab364206-5h79r                       0/1     Completed          4          4d16h
zenko-zenko-queue-exporter-6b995f54fc-hcxlx                   1/1     Running            5          4d16h
zenko-zenko-queue-manager-5475c68d5c-6sn8v                    1/1     Running            2          4d16h
zenko-zenko-queue-manager-bootstrap-4b2b1b55-fs4kl            0/1     Completed          5          4d16h
zenko-zenko-quorum-0                                          2/2     Running            0          4d16h
zenko-zenko-quorum-1                                          2/2     Running            0          4d16h
zenko-zenko-quorum-2                                          2/2     Running            0          4d16h
zenko-zenko-reporting-count-items-1572854400-hgkq7            0/1     Completed          0          5m40s

[patriziobassi]

i tried to "wake up" things by scaling mongo to a 4th instance.

mong specific commands show it's working fine

for i in 0 1 2 3; do kubectl exec zenko-mongodb-replicaset-$i -- sh -c 'mongo --eval="printjson(rs.isMaster())"'; done
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-0 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : true,
"secondary" : false,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"electionId" : ObjectId("7fffffff0000000000000003"),
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864205, 4),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:25Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864205, 4),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:25Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:26.371Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864205, 4),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864205, 4),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-1 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : false,
"secondary" : true,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:26Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:26Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:22.599Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864206, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864206, 3),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-2 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : false,
"secondary" : true,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:26Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:26Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:21.870Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864206, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864206, 2),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-3 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : false,
"secondary" : true,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864207, 2),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:27Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864207, 2),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:27Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:30.558Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864207, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864208, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}

[ssalaues]

Could you post the output of kubectl -lapp=cloudserver-manager? It seems like something occurred at startup that couldn't be reconciled and would be interested in finding the cause to prevent this from happening in the future.

However after seeing the last outputs you posted, I believe kubectl delete pod -l'app in (cloudserver,cloudserver-manager)' would fix the transient issue. In this specific state, it will trigger a re-initialization by the cloudserver service allowing the remaining services to stabilize.

[dashagurova]

Hello @patriziobassi. My name is Dasha, I am in Zenko community team.
Was the issue resolved?
I would love to learn more about your project and how you are using Zenko.
This is Zenko forum where we can chat or if you prefer any other channel let me know :)