Audit events have node IP address as source address for requests about gardener-extension-provider-metal

[mreiger]

The apiserver SNI feature with its istio gateway moves the problem from the apiserver service (which is now internal) to the istio-ingressgateway server, with the additional "bonus" that we can not change this service from this extension provider.

We still need the original client address for the audit log eventually.

Gardener's documentation says it is possible to use a self-deployed istio instead of using the Gardener managed istio, but do we want to do this?

[Gerrit91]

Did we ask some people from the Gardener community in the past on how they deal with this?

As the problem originates from the kubeproxy, I assume that running Calico or another CNI like Cilium in "kubeproxy-free" mode resolves the issue as a whole. It's probably hard to setup and change will be disruptive, but it generally fits to our environment and also from the network / performance perspective it seems to be superior:

• https://projectcalico.docs.tigera.io/maintenance/enabling-bpf
• https://docs.cilium.io/en/stable/gettingstarted/kubeproxy-free/#direct-server-return-dsr

[mwennrich]

This has nothing to do with calico or CNI.
From https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/

.spec.externalTrafficPolicy - denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. There are two available options: Cluster (default) and Local. Cluster obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. Local preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type Services, but risks potentially imbalanced traffic spreading.

[Gerrit91]

According to the Cilium documentation it sounds to me like externalTrafficPolicy=Cluster does not obscure source IPs when running in DSR mode. But maybe I did not understand it correctly then. 🤔

externalTrafficPolicy=Cluster: For the Cluster policy which is the default upon service creation, multiple options exist for achieving client source IP preservation for external traffic, that is, operating the kube-proxy replacement in DSR or Hybrid mode if only TCP-based services are exposed to the outside world for the latter.

(source already mentioned above)

As far as I know, the local policy comes with minor nits that are probably tolerable in our case, but still not 100% cool. This is unequal traffic distribution + short traffic blackholing on service update / roll (see this KEP?).

I am gonna need to read a bit more or experiment with it first...