Comments (6)
Hi Tobias
kubenurse isn't retrying DNS or HTTP requests by itself so it could be that you just hit the failing DNS resolutions.
The httptrace hooks are looking promising to get additional error details and a basic extension shouldn't be that hard to implement. I will try it when I got time but will open a proposal right now.
Best regards,
Boris
from kubenurse.
Hi Tobias
I don't have experience with the ARO service but probably this information could help you.
If only the me_ingress
is failing and no path_
or me_service
checks, I would investigate the load balancer or the ingress controller as the networking inside the cluster seems to work.
Therefore I would check the following:
- Is it possible that you have high latencies on the ingress or DNS path?
- Is the load balancer or ingress working fine for other applications?
- Is the DNS server(s) that serves your domain usually responding?
One other option to "debug" this issue would be to start a Pod with a curl -v
job that frequently fetches the /alwayshappy
endpoint.
Currently there is no debug level in kubenurse but I could imagine that something like httptrace directly or go-httpstat could be an improvement to get the error and latency details.
Best regards
Boris
from kubenurse.
Hello Boris,
Thanks for your reply.
Only me_ingress isn't working everything else works fine.
Is it possible that you have high latencies on the ingress or DNS path?
- Sometimes we got a timeout for DNS but this happens only one time and not many times one after the other.
Is the load balancer or ingress working fine for other applications? - Yes every is working fine
Is the DNS server(s) that serves your domain usually responding? - We had a problem with DNS but this was fixed through Azure
If the issue occurs I tried a curl -v in the Terminal of the same kubenurse pod and this works without any problem.
But I can setup a second debug pod with curl -v.
This would be great if this could be implemented :)
Regards
Tobias
from kubenurse.
Hi Tobias
I hacked something together for #45, which adds usable httptrace metrics and logs also lower level errors. This looks like this:
# Connection issue
2022/07/26 15:45:04 httptrace: failed connect_done for path_kind-control-plane with dial tcp 10.244.0.8:8080: connect: connection refused
2022/07/26 15:45:04 failed request for path_kind-control-plane with Get "http://10.244.0.8:8080/alwayshappy": dial tcp 10.244.0.8:8080: connect: connection refused
# DNS Issue (dns_done)
2022/07/26 15:46:32 httptrace: failed dns_done for api_server_dns with lookup kubernetes.default.svc.cluster.local: operation was canceled
2022/07/26 15:46:32 failed request for api_server_dns with Get "https://kubernetes.default.svc.cluster.local:443/version": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
And metrics are similar to this:
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.0005"} 0
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.005"} 0
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.01"} 0
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.025"} 11
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.05"} 13
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.1"} 13
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.25"} 13
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="0.5"} 13
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="1"} 13
kubenurse_httpclient_trace_request_duration_seconds_bucket{event="wrote_request",type="api_server_dns",le="+Inf"} 55
kubenurse_httpclient_trace_request_duration_seconds_sum{event="wrote_request",type="api_server_dns"} 103.054209003
kubenurse_httpclient_trace_request_duration_seconds_count{event="wrote_request",type="api_server_dns"} 55
Are you interested in trying this work-in-progress out? You can build it from branch 45-proposal-use-httptrace-to-get-advanced-metrics-and-error-messages
or let me know if I should temporarly publish an image.
from kubenurse.
Hello Boris,
Thanks for implementing this feature. if it is possible for you it would be great if you publish an image.
But no hurry i am not in the company until the end of august.
If it is not possible for you to publish an image I will build it from scratch when I'm back.
Thanks & Regards
Tobias
from kubenurse.
Related Issues (20)
- Cache /alive results HOT 1
- exclude neighbourhood pods from Nodes where scheduling is disabled HOT 3
- CI: Create CI test with kind HOT 1
- goreleaser & github actions for releases
- clusterrole permission HOT 5
- failed request for me_ingress with Get "<INGRESS_URL>/alwayshappy": unsupported protocol scheme "" HOT 2
- Question: How to contribute? HOT 5
- CI Test fails on ingress deployment
- "Moved Permanently" response from /alive endpoint. HOT 1
- Request: Create a gh-pages branch HOT 2
- 404 Not Found service endpoints HOT 9
- Grafana Dashboard HOT 7
- Proposal: Use httptrace to get advanced metrics and error messages HOT 1
- Autocreate helm chart index
- Optimisation for Neighbourhood discovery on scale HOT 14
- Monitor external endpoints HOT 12
- should this be `Before`
- WIP: Proposal: Check cluster health with mutations to the state HOT 2
- Kubenurse shold not try to bind pvc on startup HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubenurse.