Comments (5)
After I have all the members upgraded, I still need to go to the master and restart it. It seems the Leadership Notification event that should be sent to the etcd cluster members, in leadership.go, function LeaderNotification, is not invoked. So, the non-leader members of the cluster has an empty leadership token, and any token passed to them is rejects, since they don't know who the leader is.
Only after restarting the leader by terminating the instance and letting the ASG create a new instance, did I see the Leadership notification come to the other members of the cluster. Then,, the upgrade to 3.4.13 proceeded.
from etcd-manager.
Yes - great debugging. I was able to reproduce with those hints - thank you so much!
#391 is intended to fix it; we're essentially going to broadcast leadership as part of every leader iteration; please take a look and see if you agree.
from etcd-manager.
Hi @olemarkus and @justinsb and @hakman,
I ran across this as well when doing the upgrade to 3.4.13 from 3.4.3 with out k8s 1.19.4 clusters. I saw this when I used kops to rolling-update the "master instance group X" that was the leader for etcd and etcd-events.
What I saw was that the other two etcd master instance groups remained leaderless for a few minutes, then one of them became the leader.
When the "master instance group X" instance came online, it became the leader again, and then told the two non-updated master instance groups to update to 3.4.13.
They don't have 3.4.13 yet, and then I see the message in the original post on the non-upgraded master instances:
I1125 15:20:08.619698 4338 hosts.go:181] skipping update of unchanged /etc/hosts
2020-11-25 15:20:10.106552 W | etcdserver: the local etcd version 3.4.3 is not up-to-date
2020-11-25 15:20:10.106571 W | etcdserver: member 38c5d2eb0a0d1cfb has a higher version 3.4.13
from etcd-manager.
I think maybe this is related to doing an etcd upgrade, in combination with a newer version of etcd-manager, at the same time? Sometimes, the etcd-manager doesn't have the newer version of etcd yet, depending upon why you are doing the upgrade.
For instance, I saw the kops 1.19.beta-2 release with the new etcd-manager that has the embedded etcd 3.4.13. So, I am trying to roll my cluster with both a newer kops image, and etcd version, at the same time.
If I had just placed the newer etcd-manager version first, done a rolling update on the masters, then updated the etcd version, and rolled again, all masters would have the newer version of etcd.
from etcd-manager.
Thanks @mmerrill3. Very nice debugging.
from etcd-manager.
Related Issues (20)
- Leader Election Causes Downtime in Kops Rolling Updates
- ETCD TLS Bad Certificate
- Backups not coming in after OpenStack token expires: Reauth bug? HOT 1
- Handle volume failures gracefully
- Recommendation for ETCD node type for a large cluster
- Etcd-manager-ctl docker image
- Kubernetes deployment file to start the Etcd manager
- How to connect/configure existing Etcd-cluster with Ectd-manager
- etcd-manager backup cleaner calls vfs storage quite heavily HOT 1
- OpenStack virtio-scsi device path can contain full cinder uuid instead of shorten one HOT 3
- Restore Results In 0.0.0.0:8081: bind: address already in use if ETCD_LISTEN_METRICS_URLS is used HOT 2
- DNS errors after upgrading rolling update of masters in cluster HOT 14
- Problems mounting EBS volumes in ETCD nodes in the same availability zone
- Could not perform operation. Error
- How to override the default backup period?
- Auth Errors on Cluster Upgrade HOT 2
- ccccccrijfjjtdlvlchnclinvvfrfvlbnfcfgdkvhbhn
- Add a second docker registry as backup HOT 1
- DockerHub Rate Limit HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from etcd-manager.