Crash due to number of open files about headscale

[qbit]

I switched to a non-nginx configuration and things seem happy. I wonder if diddling nginx timeouts would resolve this?

Maybe proxy_connect_timeout 300; or something in the location block?

(I'll try to test the above, but it might be a bit before I can)

[diseq]

issue is gone. thanks!

[qbit]

I hit this as well. Mine is fronted by nginx - i meant to check and see if nginx is misbehaving or headscale but I haven't had time to track it down.

[diseq]

same here. nginx in front.
it seems to happen after ephemeral nodes are added. Not sure if this is related. Might be a coincidence.

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   10 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   10 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   10 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   10 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   14 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   15 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   16 FDs

for pid in /proc/[0-9]*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done |grep 78809
PID  78809 has   19 FDs

[juanfont]

@qbit is it also happening for you with ephemeral nodes?

[qbit]

I am not using pre-auth keys, so I don't think it is.

[diseq]

0.9.3 seems to have resolved the accumulating open fds.
Testing on the same configuration, so I can rule out other changes.

will leave it for some time running.