High Availability (HA) for aws-eks-asg-rolling-update-handler about aws-eks-asg-rolling-update-handler

[nitinatgh]

Apologies for the late reply.

Thanks for your time on this and providing more details for my understanding. Will take on this information and let our platform team know the outcome and what we need to do.

Thanks again!

[TwiN]

Hello @nitinatgh,

If I may ask, why do you want to run this application in HA?
It's not taking any traffic from anybody, it's just replacing outdated nodes at its own pace.

In any case, it's built to survive being restarted mid-upgrade, and in theory, you can run more than one pod.

In practice, it's possible that two replicas will run at the exact same time, and two nodes from the same ASG (or several ASGs with a node selector in common, if applicable) may be drained at the same time, but if you're using PodDisruptionBudgets to protect your important apps, you shouldn't have any issues.

[nitinatgh]

Thanks for the quick reply, appreciated!

We've recently been doing fault injection in our EKS clusters to make it more resilient, and as this falls under a platform components we'd like to make sure that all these ones identified are highly available to prevent any down time or at least minimise it. There could be such scenarios like:
- pod creation issues (if they had to get recreated from a failed node)
-- CrashLoopBackOff issues i.e. failed to get docker image
-- cluster resourcing issues
-- etc...
And these could happen for various reasons.

But from your explanation we can now have a better understanding what can happen if more than 1 pod is running. So this is also valuable information that you have provided to us.

So not sure if this would be considered in any future feature request?

Thanks

Nitin

[TwiN]

Like I said though, if this application goes down, it just means there wont be any automated rolling upgrade until it goes back up.
Even if the application was down for several weeks, there shouldn't be any issues -- it just won't start the rolling upgrade 🤔

It doesn't cause any serious, severe or major collateral damage, as such, I don't think there's any advantages in making this highly available, and while you can run two of them at the same time without causing too much friction, it's not something I think is worth investing time in given how little value it has.