Comments (6)
Did you set --node-ip and --node-external-ip to the correct values for each of the agents, or just the servers?
Based on the information you shared, it sounds like the apiserver is trying to connect to the kubelet's external IP to get logs. Normally it would connect to the public IP using the agent tunnel, so I suspect that the internal and external IPs are not being set properly.
from k3s.
Thanks for the help Brandon!
For the servers, node-ip defaults to the private IP (I believed) which would be 192.x.x..
The agents don't have these set as they only have public IP addresses which are used as the nodes IP addresses.
Should I set node-ip and node-external-ip specifically to their public addresses?
This is the current setup
Server node 1: node-ip not set, node-external-ip set to NAT gateway
Server node 2: node-ip not set, node-external-ip set to NAT gateway
Server node 3: node-ip not set, node-external-ip set to NAT gateway
Agent 1: node-ip, node-external-ip not set but only public iP
Agent 2: node-ip, node-external-ip not set but only public iP
Agent 3: node-ip, node-external-ip not set but only public iP
from k3s.
Related to #7355 I think. Base on the comment #7355 (comment), I'm still unable to get it working right.
from k3s.
Thanks for linking that issue @tdtgit !
I don't think it is the same, but it helped me identify the issue a little bit better.
I'm not entirely sure if what I'm trying to achieve is even possible mind you, concretely this is where I'm doubtful:
Since I only have one NAT gateway, I only have one public IP address. So this is my server config:
MY_EXTERNAL_IP=80.xxx.xxx.xxx
server 1: --node-ip 192.168.88.2 --node-external-ip ${MY_EXTERNAL_IP} --flannel-external-ip
server 2: --node-ip <internal-ip> --node-external-ip ${MY_EXTERNAL_IP} --flannel-external-ip
server 3: --node-ip <internal-ip> --node-external-ip ${MY_EXTERNAL_IP} --flannel-external-ip
My agent config:
agent 1: --node-ip 80.xxx.xxx.xxx --node-external-ip 80.xxx.xxx.xxx --server https://80.xxx.xxx.xxx:6443
agent 2: --node-ip 80.xxx.xxx.xxx --node-external-ip 80.xxx.xxx.xxx --server https://80.xxx.xxx.xxx:6443
agent 3: --node-ip 80.xxx.xxx.xxx --node-external-ip 80.xxx.xxx.xxx --server https://80.xxx.xxx.xxx:6443
Here is some additional information:
- Pods deployed on server1 - server 3 are not able to contact services (f.e. the kubernetes API)
- When running
wg show
I get this output (server 3):
interface: flannel-wg
public key: xxxx
private key: (hidden)
listening port: 51820
peer: xxxx
endpoint: 80.xxx.xxx.xxx:51820
allowed ips: 10.42.3.0/24
latest handshake: 41 seconds ago
transfer: 1.96 KiB received, 764 B sent
persistent keepalive: every 25 seconds
peer: xxxx
endpoint: 80.xxx.xxx.xxx:51820
allowed ips: 10.42.4.0/24
latest handshake: 1 minute, 27 seconds ago
transfer: 1.23 KiB received, 1.27 KiB sent
persistent keepalive: every 25 seconds
peer: xxxx
endpoint: 80.xxx.xxx.xxx:51820
allowed ips: 10.42.5.0/24
latest handshake: 1 minute, 38 seconds ago
transfer: 556.30 KiB received, 282.04 KiB sent
persistent keepalive: every 25 seconds
peer: xxxx
endpoint: 80.xxx.xxx.xxx:51820
allowed ips: 10.42.0.0/24
transfer: 0 B received, 17.63 KiB sent
persistent keepalive: every 25 seconds
What I can see from this is, is that server three has only peered with one other server instead of two!
We're missing a peer here and I assume that is related to the NAT gateway that forwards all traffic to one server (server 1).
from k3s.
I've indeed managed to fix this by assigning public IPs to all of my servers.
I have one last issue that persists though.
I can read the logs from all pods except the ones on server2 & server3
, there I receive a "502: bad gateway" error.
I'm sure this was spotted before in the wild, could you give me some pointers?
from k3s.
I've indeed managed to fix this by assigning public IPs to all of my servers. I have one last issue that persists though.
I can read the logs from all pods except the ones on
server2 & server3
, there I receive a "502: bad gateway" error. I'm sure this was spotted before in the wild, could you give me some pointers?
but some agent node on nat backend, not public IP.
what should i do?
from k3s.
Related Issues (20)
- Validate SUSE Liberty Linux 8.9 HOT 1
- Enhance install script to support Suse Liberty Linux HOT 2
- [v1.29.3+k3s1] Certificate rotation on k3s-agent fails to include sans addressing, causing api requests via kubectl to potentially fail if kube-vip is in use HOT 1
- Add etcd snapshot s3 config secret support
- Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 2
- [Release-1.29] - Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 1
- [Release-1.28] - Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 1
- [Release-1.27] - Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 1
- kubectl logs selectively timing out based on the host CIDR range HOT 4
- v1.30.2-rc3 Images missing HOT 4
- rpm.rancher.io RHEL 9 packages missing/removed HOT 3
- k3s crashes completely with "Observed a panic: "integer divide by zero"" HOT 10
- Allow configuration of Rootlesskit's CopyUpDirs through an environment variable
- Embedded registry mirror pulling failed
- Snapshot option discrepancy with --etcd-snapshot-name
- Support for dynamic ports all the way up to 65535.
- How can I stop k3s apiserver listening {IP}:6443/static/charts directory? HOT 2
- Extreme CPU Usage on small cluster HOT 1
- kubelet service is missing on single node cluster HOT 3
- Cannot override k3s default kubelet-args via user-provided kubelet config file HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.