[Bug] Node Connection Issues(~600 nodes) in v0.23.0-alpha12 about headscale

[kradalby]

t2.medium sounds a bit optimistic, its unclear if its too small for the headscale, or for the test clients:

The error mentioned would mean one or more of:

• The node has gone away and its not taking the update
• The node is reconnecting and the update is being sent to the "closed" version
• The node did not accept the message fast enough

The problem here might be either that the Headscale machine does not have enough resources to maintain all of the connections, or the VMs running 100s of client does not have enough resources to run them all.

The machine used in #1656 is significantly larger, its probably a bit overspecced with the new alpha.
Have you tried the same with 0.22.3 (latest stable)? It is a lot more inefficient so might struggle more on a t2.medium.

[jwischka]

Another important question is whether you are running sqlite or postgres. If sqlite try enabling wal, or switching to postgres. Sounds like it could be a concurrency issue.

[dustinblackman]

Using Postgres I'm experiencing the same issue here using alpha 12 in a network of ~30 nodes, with a handful of ephemeral nodes coming in and out through the day. I've seen both regular users on laptops, and machines in the cloud be able to connect to Headscale, but then not be able to reach any other node in the network. Headscale outputs the same errors at stated at the beginning of the issue, though while digging through the new map session logic I'm unsure if the error and the issue is related. If I were to guess something is hanging in

headscale/hscontrol/poll.go

Line 271 in 8571513

case update, ok := <-m.ch:

I had the problem with a laptop connecting to a remote machine, so I had ran tailscale down && tailscale up on the remote machine, and it then fixed the problem. I'm betting there is an issue with connection recovery in the notifier, either to the node or database. I'll dig through logs later in the evening.

[kradalby]

did you verify that there was a problem with the connections between nodes, or are you saying that you do not expect any errors?

[nadongjun]

did you verify that there was a problem with the connections between nodes, or are you saying that you do not expect any errors?

I verified that there are two issues in the latest version:

(1) When 600 users join a single Headscale server, the error "ERR update not sent, context cancelled..." occurs in Headscale.

(2) Some of the joined 600 users are in an offline status when checked with headscale node list.

There are no issues with connections between users who are in an online status.

[nadongjun]

I am currently using sqlite(without wal option). I will rerun the same tests on a higher performance instance using postgres.

[kradalby]

Please try with WAL first.

[kradalby]

WAL on by default for SQLite is coming in #1985.

I will close this issue as it is more of a performance/scaling thing than a bug. We have a couple of hidden tuning options, which together with WAL might be good content for a "performance" or "scaling" guide in the future.