Comments (18)
Any updates on this?
from eksctl.
If this helps, sure, but i don't see why is this relevant, I am not using the cluster config to upgrade my clusters.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: my-cluster
region: eu-west-1
version: "1.29"
vpc:
id: "vpc-xxxxx"
subnets:
public:
eu-west-1a:
id: "subnet-xxxx"
eu-west-1b:
id: "subnet-xxxx"
private:
eu-west-1a:
id: "subnet-xxxx"
eu-west-1b:
id: "subnet-xxxx"
clusterEndpoints:
publicAccess: true
privateAccess: true
publicAccessCIDRs: ["x.x.x.x/32"]
iam:
withOIDC: true #enables the IAM OIDC provider as well as IRSA for the Amazon CNI plugin
managedNodeGroups:
- name: gp-2vcpu-8gb-ondemand-v2
amiFamily: Bottlerocket
minSize: 2
maxSize: 3
desiredCapacity: 2
volumeSize: 50
volumeType: gp3
volumeEncrypted: true
ssh:
allow: false
instanceTypes: ["m6a.large", "m6i.large", "m5a.large"]
labels:
lifecycle: OnDemand
privateNetworking: true
- name: mo-2vcpu-16gb-spot-v2
amiFamily: Bottlerocket
minSize: 2
maxSize: 5
desiredCapacity: 2
volumeSize: 50
volumeType: gp3
volumeEncrypted: true
ssh:
allow: false
instanceTypes: ["r5n.large", "r5b.large", "r5a.large"]
spot: true
labels:
lifecycle: Ec2Spot
privateNetworking: true
secretsEncryption:
keyARN: arn:aws:kms:eu-west-1:xxxxx
cloudWatch:
clusterLogging:
enableTypes: ["*"]
from eksctl.
Sure, you can expect a release this week!
from eksctl.
Hello BogdanRS 👋 Thank you for opening an issue in eksctl
project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl
on our website
from eksctl.
Hey @BogdanRS could you please share the cluster config you are using?
from eksctl.
Indeed it seems that upgrading Bottlerocket nodes doesn't work currently. I'm able to reproduce the issue and it's not only happening to a specific configuration.
To reproduce this issue:
eksctl create cluster
1.27
cluster with a1.27
Bottlerocket nodegroupeksctl upgrade cluster --version 1.28
eksctl upgrade nodegroup --kubernetes-version 1.28
(command succeeds but nodegroup stays on 1.27)
The support to upgrade Bottlerocket nodes seems to be added recently via #6766, but it is still unclear to me how Bottlerocket upgrades are different from AL2 nodes.
from eksctl.
Upgrade fails because the changeset only contains the following changes
[
{
"type": "Resource",
"resourceChange": {
"action": "Modify",
"logicalResourceId": "ManagedNodeGroup",
"physicalResourceId": "bot/ng1",
"resourceType": "AWS::EKS::Nodegroup",
"replacement": "False",
"scope": [
"Properties"
],
"details": [
{
"target": {
"attribute": "Properties",
"name": "ForceUpdateEnabled",
"requiresRecreation": "Never"
},
"evaluation": "Static",
"changeSource": "DirectModification"
}
]
}
}
]
and as I removed the changes in #6923, the changeset fails again, although this time it includes Version
update among other updated fields.
#4423 seems relevant here.
From #4666, it seems the idea was (when we)
upgrade non-al2 nodegroups we update the Version field in the template to the correct kubernetes versions
and MakeManagedSSMParameterName
has to return empty to let the Version
field populate instead. However, #6923 changed that behaviour allowing MakeManagedSSMParameterName
to populate the latestReleaseVersion
:
eksctl/pkg/actions/nodegroup/upgrade.go
Lines 266 to 277 in 76902cd
and as a result, Version
never gets populated. Because latestReleaseVersion
for Bottlerocket doesn't upgrade with higher Kubernetes version, Bottlerocket nodegroup upgrades do nothing right now.
from eksctl.
and as I removed the changes in #6923, the changeset fails again, although this time it includes Version update among other updated fields.
I compared the new CFN template in the changeset against the old CFN template -- there is no change except the ManagedNodeGroup.Version
field, yet the changeset still list all of the following as Changes:
- LaunchTemplate
- Tags
- ManagedNodeGroup
- Tags
- NodeRole
- Version
- NodeInstanceRole
- Tags
from eksctl.
Thank you, @yuxiang-zhang, so as i understand there should also be some changes to be made on the bottlerocket side, right? Is it possible for you guys to talk with them about this?
from eksctl.
@BogdanRS Opened a PR, I tested it myself and it works for me. Would you mind doing a review and test if the fix works for you?
from eksctl.
Is it ok if you make a patch release with this one? We have a bit of an automated process that only uses eksctl binary from releases.
from eksctl.
@BogdanRS - please find the release that contains the fix here.
from eksctl.
@yuxiang-zhang it doesn't seem to work for me, i have the latest version of eksctl 0.175 and when I upgrade my nodegroups, they still get reverted to eks 1.27...
This is what I can see in the Cloudformation Stack of one of the managed nodegroups(after upgrade):
"NodegroupName": "gp-2vcpu-8gb-ondemand-aza-v2",
"ReleaseVersion": "1.19.2-29cc92cc",
"ScalingConfig": {
"DesiredSize": 1,
"MaxSize": 3,
"MinSize": 1
},
"Subnets": [
"subnet-00002137f5bb9effa"
],
"Tags": {
"alpha.eksctl.io/nodegroup-name": "gp-2vcpu-8gb-ondemand-aza-v2",
"alpha.eksctl.io/nodegroup-type": "managed"
},
"Version": "1.27"
}
},
from eksctl.
is your cluster on 1.29? how did you upgrade your cluster?
from eksctl.
yes, my cluster is on 1.29. Using eksctl upgrade cluster(never had any issues on control plane upgrade, only with nodes)
from eksctl.
I couldn't reproduce what you have. I created a 1.27 cluster, upgraded to 1.29, and then upgraded the nodegroup from 1.27 to 1.29.
I think to mitigate the issue you are seeing, you could just manually create a changeset that changes the Version to 1.29
.
This is the command I used:
eksctl upgrade nodegroup --name ng1 --kubernetes-version 1.29 --cluster bot
from eksctl.
Well, i see it is the same command, the one that i used in the initial post, but for some reason for me it doesn't work and still has the same behavior. Maybe its the fact that the last version that the nodegroups have been created/upgraded with a version of eksctl that didn't contained your changes(the ones in 0.175)? For example, i have this version on the nodegroups of the clusters that haven't been upgraded yet, alpha.eksctl.io/eksctl-version - 0.151.0(CF stack).
You could also try, for a test, to create some cluster with that version of eksctl, then upgrade eksctl, then upgrade the nodes. That's the only thing that comes to mind.
from eksctl.
The Version field was stuck on 1.27 on the Cloudformation stack. So the solution was either to remove that manually or just recreate the nodegroups from scratch.
from eksctl.
Related Issues (20)
- AWS SDK writes mfa token challenge to STDOUT HOT 1
- Latest version of AWS SDK breaks unit tests
- [Bug] Degraded managedNodeGroups when using a pathed instanceRoleARN HOT 2
- [Bug] IAM permissions are not required for "vpc-cni" addon; any IRSA configuration or pod identity associations will be ignored HOT 2
- [Bug] Regression: eksctl 0.183.0 fails to create managed nodegroup in AWS China HOT 4
- [Feedback] Config File Schema - /usage/schema/ HOT 3
- [Bug] could not create cluster provider from options: managedNodeGroups[0].overrideBootstrapCommand - 0.183.0 HOT 3
- Fargate and IPv6 integration tests fail on `main`
- [Bug] Default addons update version is different with listed latest version. HOT 1
- [Feature] Support upgrading managed Ubuntu nodegroups
- [Help] Cant install eksctl on macOS Monterey HOT 2
- AWS throttling errors are not retried
- Allow limiting the number of nodegroups created in parallel HOT 2
- Disable default retry rate-limiting behavior in AWS SDK
- [Feature] publish eksctl snap on snapstore HOT 1
- [Bug] SIGSEGV when creating a cluster with custom AMI HOT 1
- [Bug] eksctl create (or update) iamserviceaccount does not work for an existing ServiceAccount HOT 2
- [Bug] Trouble installing cilium Network on the EKS CLUSTER without the addons HOT 1
- [Bug] Cluster creation on AWS Outpost reports failure due unknown OIDC issuer URL HOT 1
- [Help] The ASG provisioned EC2s in the same AZ.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eksctl.