robusta-dev / alert-explanations Goto Github PK

I saw your posts about CPU limit. The post is good, the insight into the CPU being renewable is a great analogy to explain it and if you do not mind I will "borrow" it. I am reaching out to you with aims to hear your points agains my opinion and and I will run some further tests. Now with that said I have a bit on an issue the way the CPU limit is portrayed as somewhat evil Here and here.
I mean you might be right, but my experience is that very low limits create problems, no limits is the wild west, high enough limits are good and save hardware too. Sometimes when your program is throttled it deserves to be throttled and you should check HPA, the configs of the program and other stuff. Meaning request being let say 1 CPU and limit being 2CPUs that number is almost always too low, but you can get a good formula that does not allow hoarders to take the whole CPU of the node and allows for hoarders to take anything that is not vital to everyone else when they need to somewhat hoard one day.
For instance:
"Markus wont let Teresa drink it because her limit is 1 liter per day so she dies of thirst."
CPU is not limited to 1 litter because k8s say you say you have one litter as limit. The program Can and does still ask for CPU if it needs it; program != POD configs. It's more like limited to 1 litter if everyone else is depleting the CPU at that time, but if your program wants/needs more than your limit, you will likely have proportional to your limit compared to other deployments limits and what they need at a given time.
Also death would the the wrong analogy, death would be like OOM (hitting memory limit), what actually happens is more like a pause in living/or sleep without dreams xD, precisely because CPU does not END (renewable), it's finished at a given moment in time or not, but the next clock cycle it's distributed again to the needy.
With that said, yes, limits cause problems sometimes. I prefer some limits that do not compromise the nodes whole CPU, but allow for pods to go a lot over the request at a given time.
Now with that said, programs that need about more than 1/2 of your node regularly/often, should be analized and you should have if possible at all programs that consume much less than half your node. In real life Life is complicated TBH, but if the program regularly consumes 1/2 of a node, then it probably deserves a node for him self probably.

Balance in the force... is what I look for.

For instance I personally tend to use limits that are around these proportions to start with, let say I do not know anything about the progra, but in our company we load test (or should) before going to prod:
resources: requests: cpu: "250m" limits: cpu: "1000m"
This is 4x request vs limit, but I sometimes make it much higher depending on how the monitoring says the program behaves. This particular program uses about 80% of the request when it has some load and can consume much more sometimes, but it scales via HPA at around 80%, now it does not matter if it goes to 200% for a short time (like your example of 100m, vs 200m) while it scales and the limit allows it. This allows for certain freedom for the program and should be increased or reduced depending on the program behaviour over time, node size, etc. Of course between a low limit and no limit I would probably go with no limit to start and then adjust.