Comments (14)
I am colleague of @myaser and suggested internally the use of SWIM and hashicorp memberlist.
I think having swim as discovery of new neighbors would be great already, reducing load to kube-apiserver is always great and kubenurse by design creates a lot of load to apiserver. You need only the initial list from the kube-apiserver.
Maybe we need not to use a full Mesh of TCP/IP handshakes or http requests, but just a subset that you can define via option. For example test 10% of your neighbors and find via memberlist who is your neighbor peer, so you can have the full ring tested without having "wholes". I agree testing http would be great.
from kubenurse.
digging up this issue, as we (postfinance) are currently being impacted by scale issues on our largest cluster.
I will work on that in the coming weeks!
from kubenurse.
Hi @myaser
The first point (make the interval configurable) is something that is definitely worth to implement and is not a big deal.
Your second point is very interesting and I need to have a deeper look at the SWIM paper. From what I've seen now, it should be a good mechanism to detect communication loss to other instances of the kubenurse while reducing the network traffic.
Our initial reason why we developed the kubenurse the way it is now was to understand where network issues happen. Therefore we wanted to do a real HTTP request and record the latencies between each instance to narrow down issues on routing, virtualisation, switches, firewalls, kubelet etc. This is easy to process when the check interval is static and low.
With the SWIM protocol I'm not sure if we can easily detect which network path is causing troubles because of the non uniform check interval and distribution. But it would be a good solution to find if something is fishy in the network at larger scales.
Therefore I see that the first point (make the interval configurable) can easily be implemented and is a quick win. For larger clusters we would probably have to implement a SWIM (or similar) based check additionally.
This is my impression after scratching the paper. But probably there are also other solutions to this problem. I definitely need to dig deeper into it. It looks interesting. What about you @zbindenren ?
from kubenurse.
btw, I started with #45 before my holidays and need to finish it when I got time. This could probably be impacted but is no big deal
from kubenurse.
Therefore, we wanted to do a real HTTP request and record the latencies between each instance to narrow down issues on routing, virtualization, switches, firewalls, kubelet, etc
Maybe we can use SWIM to track membership and not rely on k8s API server discovery and still call the /alwayshappy
of every neighbor. This way, each pod queries k8s API only once at the bootstrap time and joins the memberlist. SWIM will then track the members as they join/leave without further queries to the k8s server. example
However, this leaves us with the problem of the number of TCP/IP handshakes that remain unsolved
from kubenurse.
Maybe using a cached client could reduce the load on the API server?
from kubenurse.
If you use informers you have a cache and a maintained WATCH, which in case you get a new node with kubenurse it will send to all kubernurse. Then to get unstuck in most cases you should also use refresh interval and then the client automatically does a LIST (get all endpoints) and this from every node all time.Duration.
from kubenurse.
Yes that is the plan.
from kubenurse.
@clementnuss thanks for doing this. Here some of my thoughts:
When using a cached client we could get rid of this nodeCache struct and the load on the api server for getting all pods here would hopefully decrease too.
I propose to switch to controller-runtime's client, then it would be pretty easy to use their cached client like in this simplified example:
// TODO: handle errors
config, _ := rest.InClusterConfig()
if err != nil {
return nil, fmt.Errorf("kubernetes config: %w", err)
}
c, _ := cache.New(config, cache.Options{
ByObject: map[client.Object]cache.ByObject{
&corev1.Pod{}: {},
},
})
_ = c.Start(ctx)
You can even use labels for the cache. Here you can find the documentation: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/cache
from kubenurse.
Related Issues (20)
- Cache /alive results HOT 1
- exclude neighbourhood pods from Nodes where scheduling is disabled HOT 3
- CI: Create CI test with kind HOT 1
- goreleaser & github actions for releases
- clusterrole permission HOT 5
- failed request for me_ingress with Get "<INGRESS_URL>/alwayshappy": unsupported protocol scheme "" HOT 2
- Question: How to contribute? HOT 5
- CI Test fails on ingress deployment
- "Moved Permanently" response from /alive endpoint. HOT 1
- Request: Create a gh-pages branch HOT 2
- 404 Not Found service endpoints HOT 9
- Grafana Dashboard HOT 7
- Failed request for me_ingress HOT 6
- Proposal: Use httptrace to get advanced metrics and error messages HOT 1
- Autocreate helm chart index
- Monitor external endpoints HOT 12
- should this be `Before`
- WIP: Proposal: Check cluster health with mutations to the state HOT 2
- Kubenurse shold not try to bind pvc on startup HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubenurse.