Optimisation for Neighbourhood discovery on scale about kubenurse

[szuecs]

I am colleague of @myaser and suggested internally the use of SWIM and hashicorp memberlist.
I think having swim as discovery of new neighbors would be great already, reducing load to kube-apiserver is always great and kubenurse by design creates a lot of load to apiserver. You need only the initial list from the kube-apiserver.

Maybe we need not to use a full Mesh of TCP/IP handshakes or http requests, but just a subset that you can define via option. For example test 10% of your neighbors and find via memberlist who is your neighbor peer, so you can have the full ring tested without having "wholes". I agree testing http would be great.

[clementnuss]

digging up this issue, as we (postfinance) are currently being impacted by scale issues on our largest cluster.

I will work on that in the coming weeks!

[djboris9]

Hi @myaser

The first point (make the interval configurable) is something that is definitely worth to implement and is not a big deal.

Your second point is very interesting and I need to have a deeper look at the SWIM paper. From what I've seen now, it should be a good mechanism to detect communication loss to other instances of the kubenurse while reducing the network traffic.

Our initial reason why we developed the kubenurse the way it is now was to understand where network issues happen. Therefore we wanted to do a real HTTP request and record the latencies between each instance to narrow down issues on routing, virtualisation, switches, firewalls, kubelet etc. This is easy to process when the check interval is static and low.

With the SWIM protocol I'm not sure if we can easily detect which network path is causing troubles because of the non uniform check interval and distribution. But it would be a good solution to find if something is fishy in the network at larger scales.

Therefore I see that the first point (make the interval configurable) can easily be implemented and is a quick win. For larger clusters we would probably have to implement a SWIM (or similar) based check additionally.

This is my impression after scratching the paper. But probably there are also other solutions to this problem. I definitely need to dig deeper into it. It looks interesting. What about you @zbindenren ?

[djboris9]

btw, I started with #45 before my holidays and need to finish it when I got time. This could probably be impacted but is no big deal

[myaser]

Therefore, we wanted to do a real HTTP request and record the latencies between each instance to narrow down issues on routing, virtualization, switches, firewalls, kubelet, etc

Maybe we can use SWIM to track membership and not rely on k8s API server discovery and still call the /alwayshappy of every neighbor. This way, each pod queries k8s API only once at the bootstrap time and joins the memberlist. SWIM will then track the members as they join/leave without further queries to the k8s server. example

However, this leaves us with the problem of the number of TCP/IP handshakes that remain unsolved

[zbindenren]

Maybe using a cached client could reduce the load on the API server?

[szuecs]

If you use informers you have a cache and a maintained WATCH, which in case you get a new node with kubenurse it will send to all kubernurse. Then to get unstuck in most cases you should also use refresh interval and then the client automatically does a LIST (get all endpoints) and this from every node all time.Duration.

[zbindenren]

Yes that is the plan.

[zbindenren]

@clementnuss thanks for doing this. Here some of my thoughts:

When using a cached client we could get rid of this nodeCache struct and the load on the api server for getting all pods here would hopefully decrease too.

I propose to switch to controller-runtime's client, then it would be pretty easy to use their cached client like in this simplified example:

// TODO: handle errors
config, _ := rest.InClusterConfig()
if err != nil {
	return nil, fmt.Errorf("kubernetes config: %w", err)
}

c, _ := cache.New(config, cache.Options{
	ByObject: map[client.Object]cache.ByObject{
		&corev1.Pod{}: {},
	},
})

_ = c.Start(ctx)

You can even use labels for the cache. Here you can find the documentation: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/cache