Nodes connected to headscale fail to see each other if headscale is fronted by nginx about headscale

[viq]

In case it's relevant, registration has been done using authkeys.

[juanfont]

Can you add the output for this:

headscale -n viqWORKS nodes list -o json

[viq]

headscale# headscale -n viqWORKS nodes list -o json                                                                                                                                                                                                                                                                         
[
        {
                "ID": 6,
                "MachineKey": "6594a17c9e61cd05571e10493228fe16277608228fa94b5f72764840333d8317",
                "NodeKey": "43363646d947038c6f6b556868e7657aac75e8031c143fdaf1ed25fc2a8f4b53",
                "DiscoKey": "4ee08d2740640e2ebed0dcd12d083ffe5c32c58a879406c2b7f8da0d010eee5a",
                "IPAddress": "100.87.15.215",
                "Name": "innernet-test",
                "NamespaceID": 1,
                "Namespace": {
                        "ID": 0,
                        "CreatedAt": "0001-01-01T00:00:00Z",
                        "UpdatedAt": "0001-01-01T00:00:00Z",
                        "DeletedAt": null,
                        "Name": ""
                },
                "Registered": true,
                "RegisterMethod": "authKey",
                "AuthKeyID": 4,
                "AuthKey": {
                        "ID": 4,
                        "Key": "ad8f0662366b2a8f25c793fa9c47ff6c2a34ebbf006c9edb",
                        "NamespaceID": 1,
                        "Namespace": {
                                "ID": 0,
                                "CreatedAt": "0001-01-01T00:00:00Z",
                                "UpdatedAt": "0001-01-01T00:00:00Z",
                                "DeletedAt": null,
                                "Name": ""
                        },
                        "Reusable": true,
                        "Ephemeral": false,
                        "CreatedAt": "2021-07-17T17:02:28.22417+02:00",
                        "Expiration": "2021-07-17T17:32:28.22072+02:00"
                },
                "LastSeen": "2021-07-17T19:16:17.82462+02:00",
                "Expiry": "0001-01-01T01:24:00+01:24",
                "HostInfo": {
                        "OS": "openbsd",
                        "GoArch": "amd64",
                        "NetInfo": {
                                "PCP": false,
                                "PMP": false,
                                "UPnP": false,
                                "WorkingUDP": true,
                                "DERPLatency": {
                                        "1-v4": 0.102038515,
                                        "2-v4": 0.17486828,
                                        "3-v4": 0.262211003,
                                        "4-v4": 0.020722566,
                                        "5-v4": 0.289097232,
                                        "6-v4": 0.166939259,
                                        "7-v4": 0.268397717,
                                        "8-v4": 0.037278815,
                                        "9-v4": 0.137691975
                                },
                                "HairPinning": false,
                                "WorkingIPv6": false,
                                "PreferredDERP": 4,
                                "MappingVariesByDestIP": true
                        },
                        "Hostname": "innernet-test",
                        "Services": [
                                {
                                        "Port": 47775,
                                        "Proto": "peerapi4"
                                }
                        ],
                        "IPNVersion": "date.20210603",
                        "BackendLogID": "ee6414f1b608db52193ac3e35f185522bc0ce6528ee16a49bab8c6a8c2060618"
                },
                "Endpoints": [
                        "51.75.32.28:57997",
                        "192.168.135.48:22735"
                ],
                "EnabledRoutes": null,
                "CreatedAt": "2021-07-17T17:04:17.881506+02:00",
                "UpdatedAt": "2021-07-17T19:16:17.832448+02:00",
                "DeletedAt": null
        },
        {
                "ID": 5,
                "MachineKey": "8c9e29df0f628d41d480e8951331f1d5d621b47d3019214e3db0c1eac661f839",
                "NodeKey": "5b081928375c477f3303eb7f35591673ff0a37dd6460017cfcc1e0bec045b93a",
                "DiscoKey": "d52bb11973f8889bbe5bbfdb0deacff1d1cdea7c22755e42a5c140f105cfdc0c",
                "IPAddress": "100.99.59.105",
                "Name": "headscale",
                "NamespaceID": 1,
                "Namespace": {
                        "ID": 0,
                        "CreatedAt": "0001-01-01T00:00:00Z",
                        "UpdatedAt": "0001-01-01T00:00:00Z",
                        "DeletedAt": null,
                        "Name": ""
                },
                "Registered": true,
                "RegisterMethod": "authKey",
                "AuthKeyID": 4,
                "AuthKey": {
                        "ID": 4,
                        "Key": "ad8f0662366b2a8f25c793fa9c47ff6c2a34ebbf006c9edb",
                        "NamespaceID": 1,
                        "Namespace": {
                                "ID": 0,
                                "CreatedAt": "0001-01-01T00:00:00Z",
                                "UpdatedAt": "0001-01-01T00:00:00Z",
                                "DeletedAt": null,
                                "Name": ""
                        },
                        "Reusable": true,
                        "Ephemeral": false,
                        "CreatedAt": "2021-07-17T17:02:28.22417+02:00",
                        "Expiration": "2021-07-17T17:32:28.22072+02:00"
                },
                "LastSeen": "2021-07-17T19:16:58.522021+02:00",
                "Expiry": "0001-01-01T01:24:00+01:24",
                "HostInfo": {
                        "OS": "openbsd",
                        "GoArch": "amd64",
                        "NetInfo": {
                                "PCP": false,
                                "PMP": false,
                                "UPnP": false,
                                "WorkingUDP": true,
                                "DERPLatency": {
                                        "1-v4": 0.1011065,
                                        "2-v4": 0.180232352,
                                        "3-v4": 0.299721761,
                                        "4-v4": 0.038514911,
                                        "5-v4": 0.299863107,
                                        "6-v4": 0.173342277,
                                        "7-v4": 0.272282242,
                                        "8-v4": 0.03836486,
                                        "9-v4": 0.161534457
                                },
                                "HairPinning": false,
                                "WorkingIPv6": false,
                                "PreferredDERP": 8,
                                "MappingVariesByDestIP": false
                        },
                        "Hostname": "headscale",
                        "Services": [
                                {
                                        "Port": 39599,
                                        "Proto": "peerapi4"
                                }
                        ],
                        "IPNVersion": "date.20210603",
                        "BackendLogID": "684e0fca0c5f487084b120b5dbe9bd2711ccffd8987b1fd88ed91205a4e2b573"
                },
                "Endpoints": [
                        "51.75.32.29:22502"
                ],
                "EnabledRoutes": null,
                "CreatedAt": "2021-07-17T17:03:35.956224+02:00",
                "UpdatedAt": "2021-07-17T19:16:58.533797+02:00",
                "DeletedAt": null
        }
]

[viq]

How does the http connection work? Looking at tcpdump, I see a bunch of Connection: close between nginx and headscale, but not when tailscaled and headscale are talking directly. Maybe nginx needs some tuning for long polling or whatnot?

[viq]

OK, so seems like some of the settings from https://help.hcltechsw.com/connections/v65/admin/install/inst_post_nginx.html may have helped...

[viq]

Currently seems to work with the below settings, I'll poke at it some more at a later time.

    server {
        listen       443 ssl;
        server_name  headscale.viq.vc;
        ssl_certificate      /etc/ssl/headscale.viq.vc.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/headscale.viq.vc.key;
        ssl_session_timeout  5m;
        ssl_session_cache    shared:SSL:1m;
        ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
        ssl_prefer_server_ciphers   on;
        client_body_timeout 5m;
        client_header_timeout 5m;
        location / {
                proxy_read_timeout 6m;
                proxy_http_version 1.1;
                proxy_set_header Connection "";
                proxy_set_header Upgrade $http_upgrade;
                proxy_buffering off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:8000;
        }
    }

[juanfont]

I have not been able to replicate the issue in my Vagrant setup, aside the longpolling connections being closed after 60 secs (nginx's default) as expected Would in be possible to have access to your environment?

…

On Sun, Jul 18, 2021, 20:16 viq ***@***.***> wrote: How does the http connection work? Looking at tcpdump, I see a bunch of Connection: close between nginx and headscale, but not when tailscaled and headscale are talking directly. Maybe nginx needs some tuning for long polling or whatnot? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABMGQ2U5V3HMGBXCUTEGGTTYMLA5ANCNFSM5ARFQ5FQ> .

[viq]

Update: currently 2 vhosts, headscale.viq.vc with "broken" config as previously, and testscale.viq.vc with I think working config:

        map $http_upgrade $connection_upgrade {
                default upgrade;
                ''      close;
        }
    server {
        listen       443 ssl http2;
        server_name  testscale.viq.vc;

        ssl_certificate      /etc/ssl/headscale.viq.vc.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/headscale.viq.vc.key;

        ssl_session_timeout  1d;
        ssl_session_cache    shared:MozSSL:10m;
        ssl_session_tickets off;

        # modern configuration
        ssl_protocols TLSv1.3;
        ssl_prefer_server_ciphers off;

        # HSTS (ngx_http_headers_module is required) (63072000 seconds)
        add_header Strict-Transport-Security "max-age=63072000" always;

        # OCSP stapling
        ssl_stapling on;
        ssl_stapling_verify on;

        client_body_timeout 5m;
        client_header_timeout 5m;
        location / {
                proxy_read_timeout 6m;
                proxy_http_version 1.1;
                #proxy_set_header Connection "";
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
                proxy_buffering off;
                proxy_no_cache "always";
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:8000;
        }
    }

You still have access, feel free to poke around.

[viq]

Hm, looks like with those settings I need to restart headscale for nodes (either new, or after being restarted) to see each other.

[juanfont]

I have been finally able to replicate this 😄

I will check nginx configs now...

[juanfont]

@viq @bharathmsd7

Can you check with this config in nginx?

https://github.com/juanfont/headscale/wiki/nginx-configuration

[qbit]

Switched my config over to nginx, this seems to resolve it for me - I am able to ping hosts and ssh to them!

[qbit]

It seems that after a while though things break down. I was adding hosts to a namespace and now tailscale up --login-server .... just hangs.

I see Client is registered and we have the current NodeKey. All clear to /map and then a POST to /machine/ID, but nothing after that.

[viq]

With following config

    server {
        listen       80;
        server_name  testscale.viq.vc;

        client_body_timeout 3m;
        client_header_timeout 3m;
        location / {
                proxy_read_timeout 3m;
                #proxy_http_version 1.1;
                #proxy_set_header Connection "";
                #proxy_set_header Upgrade $http_upgrade;
                proxy_ignore_client_abort off;
                #proxy_set_header Connection $connection_upgrade;
                #proxy_set_header Connection upgrade;
                proxy_buffering off;
                proxy_cache off;
                proxy_cache_bypass "always";
                proxy_no_cache "always";
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:8000;
        }
    }

(disabling websocket related things one by one; finally proxy_http_version seemed to make a difference) after a couple restarts of various things I was still able to register clients, without the issues @qbit is describing (I would see them sometimes otherwise; might still as it's not necessarily 100% repeatable). But adding a new (ephemeral) node to headscale, none of the two so far connected nodes see it without restarting things. Which I guess is the same state as we started with...

[viq]

I'm now retrying with exactly your set of options.

[viq]

Specifically:

    server {
        listen       80;
        server_name  testscale.viq.vc;

        client_body_timeout 3m;
        client_header_timeout 3m;
        location / {
                proxy_read_timeout 3m;
                proxy_ignore_client_abort off;
                proxy_request_buffering off;
                proxy_buffering off;
                proxy_no_cache "always";
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

and I'm apparently seeing what you described, i.e. headscale seeing clients closting the connections, but nginx and tailscaled believing they are still open.

[juanfont]

It seems that after a while though things break down. I was adding hosts to a namespace and now tailscale up --login-server .... just hangs.

I see Client is registered and we have the current NodeKey. All clear to /map and then a POST to /machine/ID, but nothing after that.

Can you send us the logs from the tailscaled daemons failing to connect?

[qbit]

Ya, i'll try and get some more details today - It looks like the same behavior as #50 though.

[juanfont]

@qbit @viq Can you please try with v0.5.0? https://github.com/juanfont/headscale/releases/tag/v0.5.0

[Juliaria08]

It fixed the issue i had and it is now working, thanks for the work of this project

[juanfont]

@viq @qbit can you check 0.7? This issue should be solved now.

[qbit]

Can confirm! Currently running just fine!