Comments (23)
juanfont
commented on August 27, 2024
Hi!
I am not being able to replicate this.
Can you paste the logs of headscale serve ?
from headscale.
momaek
commented on August 27, 2024
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /key --> github.com/juanfont/headscale.(*Headscale).KeyHandler-fm (3 handlers)
[GIN-debug] GET /register --> github.com/juanfont/headscale.(*Headscale).RegisterWebAPI-fm (3 handlers)
[GIN-debug] POST /machine/:id/map --> github.com/juanfont/headscale.(*Headscale).PollNetMapHandler-fm (3 handlers)
[GIN-debug] POST /machine/:id --> github.com/juanfont/headscale.(*Headscale).RegistrationHandler-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8000
[GIN] 2021/07/13 - 14:41:13 | 200 | 10.933µs | 127.0.0.1 | GET "/key"
[GIN] 2021/07/13 - 14:52:42 | 404 | 841ns | 122.228.235.37 | GET "/"
[GIN] 2021/07/13 - 14:53:17 | 200 | 40.751µs | 122.228.235.37 | GET "/key"
2021/07/13 14:53:17 New Machine!
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 5.048844ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 899.77µs | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 833.998µs | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 1.064822ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 1.044999ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 1.122197ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:17 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:17 | 200 | 1.006174ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:18 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:18 | 200 | 990.494µs | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:18 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:18 | 200 | 1.061863ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:19 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:19 | 200 | 1.112326ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:20 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:20 | 200 | 1.049061ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:21 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:21 | 200 | 1.055216ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:22 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:22 | 200 | 1.084824ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:24 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:24 | 200 | 985.07µs | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:25 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:25 | 200 | 1.336628ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:27 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:27 | 200 | 1.106841ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:29 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:29 | 200 | 1.005188ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:30 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:30 | 200 | 1.065916ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:33 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:33 | 200 | 1.127932ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:36 [awxt-aaaw] Not registered and not NodeKey rotation. Sending a authurl to register
[GIN] 2021/07/13 - 14:53:36 | 200 | 1.164391ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:40 [awxt-aaaw] Client is registered and we have the current NodeKey. All clear to /map
[GIN] 2021/07/13 - 14:53:40 | 200 | 4.058121ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537"
2021/07/13 14:53:40 [awxt-aaaw] ReadOnly=true OmitPeers=false Stream=true
2021/07/13 14:53:40 [awxt-aaaw] Client is starting up. Asking for DERP map
[GIN] 2021/07/13 - 14:53:40 | 200 | 6.833069ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537/map"
2021/07/13 14:53:40 [awxt-aaaw] ReadOnly=true OmitPeers=false Stream=true
2021/07/13 14:53:40 [awxt-aaaw] Client is starting up. Asking for DERP map
[GIN] 2021/07/13 - 14:53:40 | 200 | 7.011296ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537/map"
2021/07/13 14:53:40 [awxt-aaaw] ReadOnly=true OmitPeers=false Stream=true
2021/07/13 14:53:40 [awxt-aaaw] Client is starting up. Asking for DERP map
[GIN] 2021/07/13 - 14:53:40 | 200 | 4.396302ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537/map"
2021/07/13 14:53:40 [awxt-aaaw] ReadOnly=false OmitPeers=false Stream=true
2021/07/13 14:53:40 [awxt-aaaw] Client is ready to access the tailnet
2021/07/13 14:53:40 [awxt-aaaw] Sending initial map
2021/07/13 14:53:40 [awxt-aaaw] Notifying peers
2021/07/13 14:53:40 [awxt-aaaw] Sending data (1598 bytes)
2021/07/13 14:53:40 [awxt-aaaw] Sending keepalive
2021/07/13 14:53:40 [awxt-aaaw] Sending data (75 bytes)
2021/07/13 14:53:41 [awxt-aaaw] The client has closed the connection
[GIN] 2021/07/13 - 14:53:41 | 200 | 27.610709ms | 122.228.235.37 | POST "/machine/6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537/map"
2021/07/13 14:53:41 [awxt-aaaw] ReadOnly=false OmitPeers=false Stream=true
2021/07/13 14:53:41 [awxt-aaaw] Client is ready to access the tailnet
2021/07/13 14:53:41 [awxt-aaaw] Sending initial map
2021/07/13 14:53:41 [awxt-aaaw] Notifying peers
2021/07/13 14:53:41 [awxt-aaaw] Sending data (1598 bytes)
2021/07/13 14:53:41 [awxt-aaaw] Sending keepalive
2021/07/13 14:53:41 [awxt-aaaw] Sending data (75 bytes)
2021/07/13 14:54:41 [awxt-aaaw] Sending keepalive
2021/07/13 14:54:41 [awxt-aaaw] Sending data (75 bytes)
2021/07/13 14:55:41 [awxt-aaaw] Sending keepalive
2021/07/13 14:55:41 [awxt-aaaw] Sending data (75 bytes)
2021/07/13 14:56:41 [awxt-aaaw] Sending keepalive
2021/07/13 14:53:41 [awxt-aaaw] The client has closed the connection this log is so weird..........
from headscale.
juanfont
commented on August 27, 2024
Ok. This is weird. Did your node remained hanged, and showing the register URL?
The client has closed the connection
Thats a perfectly normal log message. Clients poke the server multiple times as part of their initialization...
from headscale.
momaek
commented on August 27, 2024
Yes, after showing the register URL, it hangs .. Nothing output
from headscale.
juanfont
commented on August 27, 2024
Can you please send me the output of these commands?
In the client:
tailscale status
In the server:
headscale -n wentx nodes list -o json
from headscale.
momaek
commented on August 27, 2024
tailscale status:
root@awxt-aaaw:~# tailscale status
Logged out.
Log in at: http://127.0.0.1:8000/register?key=6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537headscale -n wentx nodes list -o json :
ubuntu@VM-4-11-ubuntu:~$ sudo headscale -n wentx nodes list -o json
[
{
"ID": 1,
"MachineKey": "6b81abd8292b2edfe86720f53c25f4fa3c968974df844b9b99238f800588d537",
"NodeKey": "523aec74972f1477d63712705de9db24d32f8852765170f1ac72b446fd38cb50",
"DiscoKey": "89e37ceee33501990c4becce8ecbf52f2c49a65757a4b81a7f8a3c8e63c4f450",
"IPAddress": "100.127.105.139",
"Name": "awxt-aaaw",
"NamespaceID": 1,
"Namespace": {
"ID": 0,
"CreatedAt": "0001-01-01T00:00:00Z",
"UpdatedAt": "0001-01-01T00:00:00Z",
"DeletedAt": null,
"Name": ""
},
"Registered": true,
"RegisterMethod": "cli",
"AuthKeyID": 0,
"AuthKey": null,
"LastSeen": "2021-07-14T23:26:44.203438+08:00",
"Expiry": "0001-01-01T08:05:43+08:05",
"HostInfo": {
"OS": "linux",
"GoArch": "amd64",
"NetInfo": {
"PCP": false,
"PMP": false,
"UPnP": false,
"WorkingUDP": true,
"DERPLatency": {
"1-v4": 0.214764163,
"2-v4": 0.136939201,
"3-v4": 0.330014741,
"4-v4": 0.259974053,
"5-v4": 0.315374518,
"6-v4": 0.340824564,
"7-v4": 0.248941838,
"8-v4": 0.255692985,
"9-v4": 0.176007433
},
"HairPinning": true,
"WorkingIPv6": false,
"PreferredDERP": 2,
"MappingVariesByDestIP": false
},
"Hostname": "awxt-aaaw",
"Services": [
{
"Port": 64569,
"Proto": "peerapi4"
}
],
"OSVersion": "Ubuntu 18.04.5 LTS (Bionic Beaver); kernel=4.15.0-144-generic; container",
"IPNVersion": "1.10.1-t6b6016130-g49e1dcc20",
"BackendLogID": "64b9694a57e0e228a490a1f35cc30f90e8c0ee747e4354dfc0fe283511c531e6"
},
"Endpoints": [
"122.228.235.37:41641",
"10.10.10.9:41641"
],
"EnabledRoutes": null,
"CreatedAt": "2021-07-13T14:53:17.215468+08:00",
"UpdatedAt": "2021-07-14T23:26:44.204246+08:00",
"DeletedAt": null
}
]from headscale.
juanfont
commented on August 27, 2024
I have been able to replicate it when running the tailscale client in the same machine as the headscale server. Is that your case?
from headscale.
juanfont
commented on August 27, 2024
Can you try
sudo tailscale up --login-server http://localhost:8000 --force-reauth
from headscale.
momaek
commented on August 27, 2024
I have been able to replicate it when running the tailscale client in the same machine as the headscale server. Is that your case?
Nop
Can you try
sudo tailscale up --login-server http://localhost:8000 --force-reauth
It hangs..
from headscale.
momaek
commented on August 27, 2024
Does headscale have any dependencies? I'v installed wireguard on my headscale vm. I'll try without wireguard
from headscale.
momaek
commented on August 27, 2024
It just the same.. tailscale hangs
I can grant an access to my vm. And you can try it yourself....
from headscale.
juanfont
commented on August 27, 2024
I will contact you via mail
…On Thu, 15 Jul 2021 at 05:16, wentx ***@***.***> wrote:
It just the same.. tailscale hangs
I can grant an access to my vm. And you can try it yourself....
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#50 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABMGQ5546NHCNPLKJPFFMDTXZHHHANCNFSM5AIMM52A>
.
from headscale.
faicker
commented on August 27, 2024
The same problem on different host.
But it does work though tailscale status tell me logged out.
I can ping each other OK.
Both versions are the latest main branch.
from headscale.
juanfont
commented on August 27, 2024
@faicker do you have nginx in front?
from headscale.
faicker
commented on August 27, 2024
no. @juanfont
from headscale.
maddie
commented on August 27, 2024
I've encountered the same problem, and it turns out it's because I have a tailscaled joined in the official network running. Starting headscale on a freshly spawned VM fixes the hang problem.
from headscale.
kradalby
commented on August 27, 2024
I am running into a similar issue when registering many nodes in succession. It seems to me like there is a deadlock somewhere, potentially in the HTTP server.
What I have observed is:
I try to join 5 clients at the same time, 3 succeeds while 2 hangs, if I restart headscale while they are still hanging, it looks like the request gets retried and succeed.
It does not always succeed tho, so I suspect that if you restart and already connected clients try to talk to headscale, it gets busy again and blocks.
I tried to join about 13 hosts and I think I had to restart it 8-9 times over 20 minutes to get them all in.
from headscale.
bharathmsd7
commented on August 27, 2024
I am running into a similar issue when registering many nodes in succession. It seems to me like there is a deadlock somewhere, potentially in the HTTP server.
What I have observed is:
I try to join 5 clients at the same time, 3 succeeds while 2 hangs, if I restart
headscalewhile they are still hanging, it looks like the request gets retried and succeed.It does not always succeed tho, so I suspect that if you restart and already connected clients try to talk to
headscale, it gets busy again and blocks.I tried to join about 13 hosts and I think I had to restart it 8-9 times over 20 minutes to get them all in.
+1
Facing the same issue
After restarting headscale the new peer is connected but some peers not getting update about the new peer connection,
running tailscale status doesn't shows the new peer. Logging out and logging in the old peer in solves this issue but it again hangs needs to restart headscale for everytime.
from headscale.
juanfont
commented on August 27, 2024
@bharathmsd7 @momaek @faicker Can you please try with v0.5.0? https://github.com/juanfont/headscale/releases/tag/v0.5.0
from headscale.
bharathmsd7
commented on August 27, 2024
@bharathmsd7 @momaek @faicker Can you please try with v0.5.0? https://github.com/juanfont/headscale/releases/tag/v0.5.0
This works man, Deadlock is not occurring now.
But my peers which are already connected doesn't get any update on newly connected peer.
tailscale status not updating newly joining peers.
from headscale.
kradalby
commented on August 27, 2024
Ah yes, I see what you mean, it get it on some of the first joined nodes.
I will try to do a dive into it, it would definitively benefit us with some integration tests, just need to find a clever way to wire them up.
from headscale.
juanfont
commented on August 27, 2024
This is fixed now in 0.7.
from headscale.
mbugert
commented on August 27, 2024
For the record, I ran into the same issue as OP using headscale v0.12.3 and tailscale 1.20.2 on a bunch of Debian VMs. The issues I saw were tailscale up on clients not printing "Success" and just hanging indefinitely, tailscale up --force-reauth hanging as well, and tailscale status saying state=Starting, wantRunning=true.
The problem was that my headscale server and tailscale client were running in a local network without WAN access (I had used the host-only adapter network mode in virtualbox). Giving them WAN access (NAT network mode) fixed it for me.
from headscale.
Related Issues (20)
- [Feature] Add ExecReload to systemd service HOT 1
- [Bug] ACL policy not working after update to v23.0 beta1 HOT 26
- [Bug] v0.23.0-beta1 breaks built-in DERP HOT 17
- [Bug] 0.23.0-beta1 breaks DNS custom nameservers HOT 4
- [Bug] tailscale on ios cannot connect to the postgresql server HOT 6
- [Feature] Add SafeSurfer DoH HOT 2
- [Bug] 0.23.0-beta1 wipes resolv.conf on clients regardless of dns_config HOT 4
- [Bug] API does not return tags for a node
- [Bug] API backfillips call fail
- [Bug] expiring a node does not update the online status
- [Bug] ipv4 range not used HOT 2
- [Bug] Self-built DERP server cannot obtain delay information HOT 3
- tailscale-private | 2024/08/04 15:49:08 control: NetInfo: NetInfo{varies=true hairpin= ipv6=false ipv6os=true udp=true icmpv4=false derp=#900 portmap= link="" firewallmode="ipt-default"}
- [Feature] Headscale policy set validate ACL before applying? HOT 11
- [Bug] Feature request form HOT 2
- [Bug] Tailscale Exit Node DNS Doesnt Provide DNS HOT 3
- [Bug] 0.23.0-beta client setup hangs on linux on raspberry pi HOT 2
- [Bug] gRPC API fails to resolve relative policy file path, unlike server startup behavior
- [Bug] New Config.yaml ACL section does not work - Must use old acl_policy_path HOT 4
- [Bug] Unable to Add Another Account on Windows 10 Client 1.70.0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from headscale.