Comments (7)
@patriziobassi Since we don't deploy in an ordered sequence of dependencies, typically you will see CrashLoopBackoff
as a normal part of the installation until all the replicas of the stateful services stabilize. For 3 node clusters, the typical stabilization period is between 5-10 minutes (some managed clusters that have slower scheduling can take longer).
The specific error you are seeing is from the async engine Backbeat
is typical of this service waiting on all 3 of the MongoDB replicas to startup and be available. Have you continued to see this error even after all MongoDB replicas are "Green"? I ask because in the snippet you posted I can only see 1 MongoDB still in the initializing state.
Edit:
Also what would be useful for further debugging is some more info on how you installed Zenko. Such as custom values added to the Helm chart installation (values.yaml changes, cli --set
options, etc.
from zenko.
Hi
of course i waited mongodb to be up and running, infact the 3 istances were deployed and running.
i have a 5 workers cluster, infact all the pods/containers become fully running in a few minutes, less than 10. then all the containers based on backbeat images start to restart, i waited a whole night and i got hundreds of restarts.
i just did a plain helm install as in the installation guide without any change in yml
from zenko.
@patriziobassi This is a strange issue and typically should never happen beyond the first few minutes of initialization. Can you post or update your original post with the current status of all the pods? This will help us better understand what is going on with your install.
from zenko.
kubectl get pods
NAME READY STATUS RESTARTS AGE
zenko-backbeat-api-5566bdfd68-w6bkq 0/1 CrashLoopBackOff 2226 4d16h
zenko-backbeat-gc-consumer-574d5d69cf-gvg9m 0/1 CrashLoopBackOff 2224 4d16h
zenko-backbeat-gc-consumer-574d5d69cf-m4xhf 0/1 CrashLoopBackOff 2229 4d16h
zenko-backbeat-gc-consumer-574d5d69cf-p4ngk 0/1 CrashLoopBackOff 2227 4d16h
zenko-backbeat-ingestion-consumer-db4b76459-4hskl 0/1 CrashLoopBackOff 2221 4d16h
zenko-backbeat-ingestion-consumer-db4b76459-4tzs8 0/1 CrashLoopBackOff 2216 4d16h
zenko-backbeat-ingestion-consumer-db4b76459-ww7ww 0/1 CrashLoopBackOff 2227 4d16h
zenko-backbeat-ingestion-producer-69589dfdc6-979g6 0/1 CrashLoopBackOff 2230 4d16h
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-8b6mq 0/1 CrashLoopBackOff 2228 4d16h
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-bp5rw 0/1 CrashLoopBackOff 2233 4d16h
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-jwb8r 0/1 CrashLoopBackOff 2232 4d16h
zenko-backbeat-lifecycle-conductor-65d5ccb9f6-pr4x9 1/1 Running 19 4d16h
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-88sz9 0/1 CrashLoopBackOff 2230 4d16h
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-nrgsn 0/1 CrashLoopBackOff 2219 4d16h
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-zpg7v 0/1 CrashLoopBackOff 2225 4d16h
zenko-backbeat-replication-data-processor-86f5dbcd9-7vbt6 0/1 CrashLoopBackOff 2224 4d16h
zenko-backbeat-replication-data-processor-86f5dbcd9-hwbhw 1/1 Running 2226 4d16h
zenko-backbeat-replication-data-processor-86f5dbcd9-m4qj6 0/1 CrashLoopBackOff 2214 4d16h
zenko-backbeat-replication-populator-749cfd8d59-m27zz 1/1 Running 20 4d16h
zenko-backbeat-replication-status-processor-b8cfd5785-2x652 1/1 Running 2222 4d16h
zenko-backbeat-replication-status-processor-b8cfd5785-4chcx 1/1 Running 2218 4d16h
zenko-backbeat-replication-status-processor-b8cfd5785-tzgl2 0/1 CrashLoopBackOff 2220 4d16h
zenko-cloudserver-dd76fb887-4wvcl 1/1 Running 12 4d16h
zenko-cloudserver-dd76fb887-6cjsc 1/1 Running 6 4d16h
zenko-cloudserver-dd76fb887-6svzz 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-6t8kp 1/1 Running 6 4d16h
zenko-cloudserver-dd76fb887-6wlhq 1/1 Running 13 4d16h
zenko-cloudserver-dd76fb887-74c5h 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-74c8w 1/1 Running 9 4d16h
zenko-cloudserver-dd76fb887-7jpmc 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-88p95 1/1 Running 5 4d16h
zenko-cloudserver-dd76fb887-8fx99 1/1 Running 13 4d16h
zenko-cloudserver-dd76fb887-8shr7 1/1 Running 6 4d16h
zenko-cloudserver-dd76fb887-9c7r4 1/1 Running 6 4d16h
zenko-cloudserver-dd76fb887-9g2lg 1/1 Running 7 4d16h
zenko-cloudserver-dd76fb887-bpdl8 1/1 Running 7 4d16h
zenko-cloudserver-dd76fb887-ddvsc 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-dzpj7 1/1 Running 6 4d16h
zenko-cloudserver-dd76fb887-h79w2 1/1 Running 9 4d16h
zenko-cloudserver-dd76fb887-kg5hb 1/1 Running 7 4d16h
zenko-cloudserver-dd76fb887-lp24g 1/1 Running 14 4d16h
zenko-cloudserver-dd76fb887-mhjwb 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-mnxzb 1/1 Running 14 4d16h
zenko-cloudserver-dd76fb887-nqzz4 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-q88tv 1/1 Running 11 4d16h
zenko-cloudserver-dd76fb887-qvm5k 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-r85k6 1/1 Running 8 4d16h
zenko-cloudserver-dd76fb887-vxlg9 1/1 Running 6 4d16h
zenko-cloudserver-dd76fb887-wj4l2 1/1 Running 5 4d16h
zenko-cloudserver-dd76fb887-x6zlw 1/1 Running 5 4d16h
zenko-cloudserver-dd76fb887-z4qn2 1/1 Running 10 4d16h
zenko-cloudserver-dd76fb887-zcg74 1/1 Running 10 4d16h
zenko-cloudserver-manager-75c489df7-fkh4n 1/1 Running 8 4d16h
zenko-cosmos-operator-5b86756ff7-plpd5 1/1 Running 0 4d16h
zenko-cosmos-scheduler-5c96f7ff9d-k2jvd 0/1 CrashLoopBackOff 1310 4d16h
zenko-grafana-fd8d85b84-q6wbx 3/3 Running 4 4d16h
zenko-mongodb-replicaset-0 2/2 Running 0 4d16h
zenko-mongodb-replicaset-1 2/2 Running 0 4d16h
zenko-mongodb-replicaset-2 2/2 Running 0 4d16h
zenko-prometheus-server-0 2/2 Running 0 4d16h
zenko-prometheus-server-1 2/2 Running 0 4d16h
zenko-redis-ha-server-0 2/2 Running 0 4d16h
zenko-redis-ha-server-1 2/2 Running 0 4d16h
zenko-redis-ha-server-2 2/2 Running 0 4d16h
zenko-s3-data-78f46ccd8d-dd4g5 1/1 Running 0 4d16h
zenko-zenko-queue-0 2/2 Running 3 4d16h
zenko-zenko-queue-1 2/2 Running 0 4d16h
zenko-zenko-queue-2 2/2 Running 0 4d16h
zenko-zenko-queue-config-ab364206-5h79r 0/1 Completed 4 4d16h
zenko-zenko-queue-exporter-6b995f54fc-hcxlx 1/1 Running 5 4d16h
zenko-zenko-queue-manager-5475c68d5c-6sn8v 1/1 Running 2 4d16h
zenko-zenko-queue-manager-bootstrap-4b2b1b55-fs4kl 0/1 Completed 5 4d16h
zenko-zenko-quorum-0 2/2 Running 0 4d16h
zenko-zenko-quorum-1 2/2 Running 0 4d16h
zenko-zenko-quorum-2 2/2 Running 0 4d16h
zenko-zenko-reporting-count-items-1572854400-hgkq7 0/1 Completed 0 5m40s
from zenko.
i tried to "wake up" things by scaling mongo to a 4th instance.
mong specific commands show it's working fine
for i in 0 1 2 3; do kubectl exec zenko-mongodb-replicaset-$i -- sh -c 'mongo --eval="printjson(rs.isMaster())"'; done
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-0 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : true,
"secondary" : false,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"electionId" : ObjectId("7fffffff0000000000000003"),
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864205, 4),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:25Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864205, 4),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:25Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:26.371Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864205, 4),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864205, 4),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-1 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : false,
"secondary" : true,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:26Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:26Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:22.599Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864206, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864206, 3),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-2 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : false,
"secondary" : true,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:26Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864206, 2),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:26Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:21.870Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864206, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864206, 2),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Defaulting container name to mongodb-replicaset.
Use 'kubectl describe pod/zenko-mongodb-replicaset-3 -n default' to see all of the containers in this pod.
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.8
{
"hosts" : [
"zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-1.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-2.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017"
],
"setName" : "rs0",
"setVersion" : 4,
"ismaster" : false,
"secondary" : true,
"primary" : "zenko-mongodb-replicaset-0.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"me" : "zenko-mongodb-replicaset-3.zenko-mongodb-replicaset.default.svc.cluster.local:27017",
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1572864207, 2),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2019-11-04T10:43:27Z"),
"majorityOpTime" : {
"ts" : Timestamp(1572864207, 2),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2019-11-04T10:43:27Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-11-04T10:43:30.558Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 6,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1572864207, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1572864208, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
from zenko.
Could you post the output of kubectl -lapp=cloudserver-manager
? It seems like something occurred at startup that couldn't be reconciled and would be interested in finding the cause to prevent this from happening in the future.
However after seeing the last outputs you posted, I believe kubectl delete pod -l'app in (cloudserver,cloudserver-manager)'
would fix the transient issue. In this specific state, it will trigger a re-initialization by the cloudserver service allowing the remaining services to stabilize.
from zenko.
Hello @patriziobassi. My name is Dasha, I am in Zenko community team.
Was the issue resolved?
I would love to learn more about your project and how you are using Zenko.
This is Zenko forum where we can chat or if you prefer any other channel let me know :)
from zenko.
Related Issues (20)
- Cloudserver memory leak HOT 8
- Single Node Kubernetes instllation referece link broken HOT 5
- Redis-HA new chart location HOT 2
- Zenko Docker Swarm Testing installation reference link broken HOT 3
- [feature]: apache airflow provider HOT 1
- Why is everything broken HOT 1
- Download helm curl command does not fetch the correct tar ball
- README.md "Deploying a HA Kubernetes cluster" leads to 404 page not found link
- Install with Helm results in an error HOT 1
- Upgrade procedure from 1 to 2 version
- Make zenko-operator repo public so we can build the solution image.
- Zenko v1 API KO ? HOT 4
- Review of https://github.com/scality/Zenko/pull/1903
- Review of https://github.com/scality/Zenko/pull/1911
- Zenko error on Scality Artesca
- Cloudserver default API key credentials doesn't seem to work - Docker image
- Release 1.1.6's main Chart.yaml is tagged 1.1.5 HOT 1
- jq error (redis.port) HOT 2
- spark write to zenko - bucket does not exist HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zenko.