Comments (13)
Hi @Hanspagh
It definitely should. If it doesn't, I'd consider it a bug. Do you have indication that it doesn't or you just wanna make sure?
from k8s.
I do not have something concrete, just the indications that after a while (half a day), I do not seem to be getting any new events. I am using Kubernetes in azure, we have noticed in python we need to handle timeout explicitly since they would shut down our watchers. So was just wondering if the same could be happening here
from k8s.
Half a day? Very strange... But I'm gonna have a look...
from k8s.
I have enabled debug logging to see, if I can get a bit more information, I know the kubenetes api sets a semi-random timeout on all the watch requests to spread out the load
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ ---min-request-timeout.
In V1 I could see you handle the timeout explicitly here and I couldn't seem to find the corresponding code in V2, that is what got me wondering
from k8s.
Hey @Hanspagh, I've tried to reproduce this on a local cluster. See my findings:
Steps
- I have set up a local cluster using kind.
- Using
docker exec
I've updated/etc/kubernetes/manifests/kube-apiserver.yaml
, adding- --min-request-timeout=5
to the container command. - restarted the docker container
- Started a watch (with debug statements).
Behaviour
Every 6-7s, the watcher receives a BOOKMARK, followed by a :done
. :done
Is sent by the Mint adapter to signal that the request ended (in this case, probably because of a timeout). Upon receiving a :done
, the watcher resumes the watch.
Conclusion
I have to assume the watcher works as expected.
Now what...
I'm still curious however about the reason why you stop receiving events. Does the debug logging shed any light?
I do not seem to be getting any new events.
Is the process running the stream still alive at that point?
I am using Kubernetes in azure
In what version?
from k8s.
Thank you for investigating this, I will report back once this happens again, it should not be more than a day.
I am using Flow on top of the stream to capture updates over a time window, so maybe something in there is a cause of the problems.
from k8s.
Could it be the api-server going away? Because of maintenance or so... but that would not occur on a daily basis I guess...
from k8s.
Enabled debug logging, now streams have been running fine for 48 hours. This might have been an azure Kubernetes thing after all. Sorry for the inconvenience. Will comment or reopen if I at some point find out what was the cause of this
from k8s.
I'm gonna have to re-open this. It was there directly in front of me and I did not see it. But when the server goes away (in my test I can simulate this by restarting the docker container running the cluster), k8s doesn't always recognise this. Since the connection is cached it tries to make requests using the same (closed) connection. Or worse, the stream just stays open (like in your case).
from k8s.
Ahh, interesting, I guess that could have happened for my cluster.
from k8s.
I'm pretty sure about it. I'm trying to fix this, but I'm struggling and currently have limited time. But it's an important requirement for Bonny.
from k8s.
OK I think I have working code. Starting a PR soon.
from k8s.
This should be fixed in 2.0.3. @Hanspagh could you please keep me posted? Thanks!
from k8s.
Related Issues (20)
- Source links in the documentation are incorrect HOT 3
- Read kube config from string
- 2.0.0-rc.1 doesn't work with kind clusters HOT 6
- Configuration of http_provider and discovery_driver is ignored HOT 13
- Large resources unusable with 2.0.0 HOT 6
- Patch strategy should be configurable HOT 5
- hostname issues HOT 7
- Streams resetting resource version HOT 4
- K8s.Conn.Config does not exist HOT 4
- Improve documentation on in-cluster connection HOT 5
- what is different between k8s lib and swarm ? HOT 2
- Support pods/logs subresource streaming HOT 5
- Exception raised when running too many get requests HOT 6
- Mint HTTPAdapter failure cycle spams errors HOT 1
- Hex.pm Usage Guide doc is unavailable HOT 2
- Http2 Adapter causing 500 requests HOT 10
- Long-lasting shell connection example is broken HOT 4
- Connections being closed -- K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process. HOT 2
- ArgumentError in K8s.Client.Mint.ConnectionRegistry on TCP Connection HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8s.