Stunner gateway operator `ERROR updater cannot update service` on AWS + EKS + ALB due to `"cannot upsert service \"stunner/udp-gateway\": Service \"udp-gateway\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set"` about stunner

[davidkornel]

Hi @calexandre
Thank you for reporting this edge case. The field spec.loadBalancerClass is indeed immutable. And since we do not set the field's value when the gateway operator creates the initial service based on the corresponding Gateway's config, the cloud provider's default value will be used. It seems that after deploying the Service in the cluster, AWS's controller will immediately set this immutable field based on the annotations. Although it should not be modified, they do it anyway. Could not find anything useful based on this online, if you got anything please post it in the comments.

Does this Error get raised only once?
If it works as expected maybe we should leave it as of now and create a note in the docs to raise attention that "AWS....".

@rg0now Maybe in the upcoming releases we could support an additional annotation to set the spec.loadBalancerClass field by default when creating the Service. However, it might be problematic because the user needs to know which `aws-loadbalancer-' combination comes with whatever LoadBalancer class. AND it is just AWS we do not know how other cloud providers handle this.

[calexandre]

@davidkornel for now it only affects AWS, but it will change in the future because the spec.loadBalancerClass is part of the service spec from official Kubernetes documentation.

It behaves very much like the spec.ingressClass on the ingress resource.

Now on my end, the stunner error I posted above, is happening on every reconciliation loop (about 1 or 2 times per minute). What are the implications of the error on the stunner side? Without that answer I cannot quantify the severity of the issue...

[davidkornel]

I asked whether it happens only once or every time to see if it blocks the service update totally or not. If it happens on every reconciliation it blocks the update process totally (I think). As I see from that point we lose control over the service since we cannot update it, however deletion should work. Thus a modification to the Gateway resource should not apply to the service.

[calexandre]

I asked whether it happens only once or every time to see if it blocks the service update totally or not. If it happens on every reconciliation it blocks the update process totally (I think). As I see from that point we lose control over the service since we cannot update it, however deletion should work. Thus a modification to the Gateway resource should not apply to the service.

Okay, so if we do not update the gateway, despite the error there should be no problems right?

[rg0now]

Okay, so if we do not update the gateway, despite the error there should be no problems right?

Yes, you're right. But it's still a bug though, thanks a lot for reporting it!

The solution would be to special-case the code that updates Services here:

• If the Service new Service (which we are about to DeepCopy) contains a nil spec.loadBalancerClass then we should leave the spec.loadBalancerClass on the existing Service intact. This should fix this error.
• If the new and the existing Service both contain a non-nil spec.loadBalancerClass and the two classes differ, then we should delete the existing Service and recreate it with the new spec.loadBalancerClass (this is a super-intrusive update, public IPs and NodePorts will change, but this should be a relatively rare occurrence).

I'm wondering whether changing controllerutil.CreateOrUpdate to controllerutil.CreateOrPatch would automatically fix this without special-casing. We'll need to do some testing to see this.

[calexandre]

Sounds great!
Let me know if you need anything else on my end for reproduction purposes...

[rg0now]

Fixed in l7mp/stunner-gateway-operator@81c2eaf