patte / fly-tailscale-exit Goto Github PK
View Code? Open in Web Editor NEWRun a VPN with global exit nodes with fly.io, tailscale and github!
Home Page: https://news.ycombinator.com/item?id=36064305
Run a VPN with global exit nodes with fly.io, tailscale and github!
Home Page: https://news.ycombinator.com/item?id=36064305
Thank you for your project, it has been of great use to me.
But I have a confusion, I am not sure if it is a problem with my operation or the script itself.
When I follow the process to step 10 flyctl deploy
, the platform shows two machines at the same time. This is shown in the image below:
I made sure I didn't perform any additional operations. I checked the project's scripts and didn't find the problem either. Is there a way to start only one machine? Two machines in the same area seems a bit redundant.
how i can fix this
Recent Events
TIMESTAMP TYPE MESSAGE
2023-02-04T11:49:58Z Received Task received by client
2023-02-04T11:49:59Z Task Setup Building Task Directory
2023-02-04T11:50:16Z Driver Failure rpc error: code = Unknown desc = error waiting for vsock readiness: vsock connection attempts exhausted
2023-02-04T11:50:16Z Not Restarting Error was unrecoverable
2023-02-04T11:50:16Z Alloc Unhealthy Unhealthy because of failed task
2023-02-04T11:50:16Z Killing Sent interrupt. Waiting 5s before force killing
2023-02-04T11:50:00Z [info]Configuring virtual machine
2023-02-04T11:50:00Z [info]Pulling container image
2023-02-04T11:50:06Z [info]Unpacking image
2023-02-04T11:50:08Z [info]Preparing kernel init
2023-02-04T11:50:09Z [info]Configuring firecracker
2023-02-04T11:50:09Z [info]Starting virtual machine
2023-02-04T11:50:09Z [info]Starting init (commit: e3cff9e)...
2023-02-04T11:50:09Z [info]Preparing to run: /app/start.sh
as root
2023-02-04T11:50:09Z [info]Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })
2023-02-04T11:50:09Z [info][ 0.150709] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
2023-02-04T11:50:09Z [info][ 0.152751] CPU: 0 PID: 1 Comm: init Not tainted 5.12.2 #1
2023-02-04T11:50:09Z [info][ 0.152715] Call Trace:
2023-02-04T11:50:09Z [info][ 0.153150] show_stack+0x52/0x58
2023-02-04T11:50:09Z [info][ 0.153548] dump_stack+0x6b/0x86
2023-02-04T11:50:09Z [info][ 0.153997] panic+0xfb/0x2bc
2023-02-04T11:50:09Z [info][ 0.154410] do_exit.cold+0x60/0xb0
2023-02-04T11:50:09Z [info][ 0.154927] do_group_exit+0x3b/0xb0
2023-02-04T11:50:09Z [info][ 0.155760] __x64_sys_exit_group+0x18/0x20
2023-02-04T11:50:09Z [info][ 0.156397] do_syscall_64+0x38/0x50
2023-02-04T11:50:09Z [info][ 0.156867] entry_SYSCALL_64_after_hwframe+0x44/0xae
2023-02-04T11:50:09Z [info][ 0.157472] RIP: 0033:0x7f7f661f16c9
2023-02-04T11:50:09Z [info][ 0.157863] Code: eb ef 48 8b 76 28 e9 a5 03 00 00 64 48 8b 04 25 00 00 00 00 48 8b b0 b0 00 00 00 e9 af ff ff ff 48 63 ff b8 e7 00 00 00 0f 05 3c 00 00 00 48 89 d0 0f 05 eb f9 66 2e 0f 1f 84 00 00 00 00 00
2023-02-04T11:50:09Z [info][ 0.160939] RSP: 002b:00007ffc13bbb8b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
2023-02-04T11:50:09Z [info][ 0.162255] RAX: ffffffffffffffda RBX: 00007f7f65f6c340 RCX: 00007f7f661f16c9
2023-02-04T11:50:09Z [info][ 0.163163] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
2023-02-04T11:50:09Z [info][ 0.164177] RBP: 0000000000000001 R08: 00007f7f662cba58 R09: 0000000000000000
2023-02-04T11:50:09Z [info][ 0.165102] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc13bbb918
2023-02-04T11:50:09Z [info][ 0.166010] R13: 00007ffc13bbb928 R14: 0000000000000000 R15: 0000000000000000
2023-02-04T11:50:09Z [info][ 0.166978] Kernel Offset: disabled
2023-02-04T11:50:09Z [info][ 0.167434] Rebooting in 1 seconds..
--> v7 failed - Failed due to unhealthy allocations and deploying as v8
-- deleted
I encountered the following error while using fly deploy
.
$ fly deploy
Update available 0.0.353 -> v0.0.499.
Run "fly version update" to upgrade.
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded
After diagnosing with the fly doctor
command, it was found that the problem was caused by the local network firewall, which prevented the normal use of WireGuard to connect to fly.io's servers.
$ fly doctor
Update available 0.0.353 -> v0.0.499.
Run "fly version update" to upgrade.
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)
We can't establish connectivity with WireGuard for your personal organization.
WireGuard runs on 51820/udp, which your local network may block.
If this is the first time you've ever used 'flyctl' on this machine, you
can try running 'flyctl doctor' again.
Switching to another server and network without similar restriction rules, the problem was solved smoothly. I hope this can be helpful for anyone who may encounter similar problems.
Is there a way to debug when after deploying everything and approving everything and then enabling:
sudo tailscale up --exit-node=fly-syd --exit-node-allow-lan-access=true
It's not possible to access the internet. All packets get dropped. ping 8.8.8.8
doesn't work.
My tailscale network is up and running and I'm able to connect to other tailscale machines, but nothing gets routed over the exit node.
Furthermore I notice that the fly-syd
doesn't respond to ICMP ping. I guess you have to use tailscale ping fly-syd
, which does work, and the logs on the fly dashboard seem to indicate things are fine.
please tell me how to fix this
You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
1 desired, 1 placed, 0 healthy, 1 unhealthy
--> v0 failed - Failed due to unhealthy allocations and deploying as v1
Error logs
2022-02-10T03:39:24.854 app[f483df71] maa [info] Starting init (commit: 0c50bff)...
2022-02-10T03:39:24.871 app[f483df71] maa [info] Preparing to run: /app/start.sh
as root
2022-02-10T03:39:24.874 app[f483df71] maa [info] Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })
Hi @patte,
I've noticed that the new flyctl
looks for Fly Templates now named fly.toml
which causes the current fly-template.toml
to not be found.
My suggestion would be to update all documentation and files to now refer to fly.toml
instead of fly-template.toml
I have set up an exit node using this guide, but traffic doesn't pass through this node. If anyone else runs into this issue, I have asked about it here:
So based on the readme I have dedicated an IPv4 on the fly.io
flyctl ips list
VERSION IP TYPE REGION CREATED AT
v4 1.2.3.4 public global 1h44m ago
But when I tried to do ping from the flyctl vm to my machine it always routed via the DERP relay
3d8d909f754089:/# ./app/tailscale ping my-machine
pong from my-machine (100.86.107.48) via DERP(sin) in 21ms
This is my current fly.toml
# fly.toml app configuration file generated for kkk-net on 2023-10-03T14:50:04+07:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = "my-net"
primary_region = "sin"
kill_signal = "SIGINT"
kill_timeout = "5s"
[experimental]
auto_rollback = false
private_network = true
[build]
[env]
PORT = "443"
[[services]]
protocol = "udp"
internal_port = 443
processes = ["app"]
[[services.ports]]
port = 443
[services.concurrency]
type = "connections"
hard_limit = 100
soft_limit = 75
Thanks a lot in advance!
I encountered the following warning while using fly deploy
.
$ flyctl deploy
...
==> Creating release
Error Services defined at indexes: 0 require a dedicated IP address. You currently have no dedicated IPs allocated. Please allocate at least one dedicated IP before deploying (`fly ips allocate-v4` and/or `fly ips allocate-v6`). Affected services:
[0] udp/41641 => 8080
$ fly ips allocate-v4
? Looks like you're accessing a paid feature. Dedicated IPv4 addresses now costs $2/mo. Are you ok with this?
After investigation, it was found that Fly.io no longer provides dedicated IPv4 addresses for free.
I wonder if there's a way to lift the restrictions mentioned above by modifying the tailscale configuration? Thank you for your help.
I tried with flyctl deploy after cloning to local, and it returned the error message:
==> Validating app configuration
--> Validating app configuration done
Services
UDP 41641 ⇢ 41641
Remote builder fly-builder-spring-bird-602 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon 75.2kB
[+] Building 435.9s (4/14)
=> [internal] load remote build context 0.0s
=> copy /context / 0.1s
=> [internal] load metadata for docker.io/library/alpine:latest 0.7s
=> ERROR [tailscale 1/4] FROM docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad 435.2s
=> => resolve docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad 0.0s
=> => sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad 1.64kB / 1.64kB 0.0s
=> => sha256:1304f174557314a7ed9eddb4eab12fed12cb0cd9809e4c28f29af86979a3c870 528B / 528B 0.0s
=> => sha256:9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5 1.47kB / 1.47kB 0.0s
=> => sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 0B / 2.81MB 435.2s
------
> [tailscale 1/4] FROM docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad:
------
Error error building: failed commit on ref "layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49": "layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49" failed size validation: 0 != 2806054: failed precondition
Is that a issue with the dockerfile, or with the remote worker?
I have been keeping my own images and noticed that now the V2 API has become mandatory. I will update my instructions according to the following issue: spotsnel#7. if you appreciate, I can port this back after I finished.
Currently dealing with some related issues with scaling, as flyctl
has some unexpected behaviour:
Fly.io's free tier (160GB + 140GB in other regions) isn't meant for use by proxies.
I followed the instructions and even set flyctl scale count 1
and I see fly-syd
and fly-syd-1
in my tailscale admin. Why are there 2 machines? There's even only 1 app on fly dashboard. Is there a way to know which machine is actually connected?
I face this issue when deploying. How can I fix?
1 desired, 1 placed, 0 healthy, 1 unhealthy
v2 failed - Failed due to unhealthy allocations
***v2 failed - Failed due to unhealthy allocations and deploying as v3
Checking log there is an error
nrt [info] Running: `/app/start.sh` as root
nrt [info] Error: UnhandledIoError(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })
After using the exit node for like a 30mins-1hr it always goes offline, and the log says
Out of memory: Killed process 526 (tailscaled) total-vm:932140kB, anon-rss:185320kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:608kB oom_score_adj:0
Is there a way to prevent this?
Hey @patte,
It's been a while 😜
I have discovered an error in Wget while deploying with Fly.io's remote builder - the deployment will error out ([HOST](wget: can't connect to remote host: Host is unreachable))
while trying to download the latest package from pkgs.tailscale.com
I've created a workaround with cURL which also deploys successfully to Fly.io and will make a pull request if that's fine with you
Here are the latest deployment logs in case you can find a problem with it
*[main][~/fly-tailscale-exit]$ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-long-glitter-301 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon 79.54kB
[+] Building 7.1s (7/14)
=> [internal] load remote build context 0.0s
=> copy /context / 0.1s
=> [internal] load metadata for docker.io/library/alpine:latest 3.6s
=> [stage-1 1/8] FROM docker.io/library/alpine:latest@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b799 0.9s
=> => resolve docker.io/library/alpine:latest@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc00 0.0s
=> => sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517 1.64kB / 1.64kB 0.0s
=> => sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501 528B / 528B 0.0s
=> => sha256:b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d 1.47kB / 1.47kB 0.0s
=> => sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.37MB / 3.37MB 0.8s
=> => extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 0.1s
=> [tailscale 2/4] WORKDIR /app 0.0s
=> CANCELED [stage-1 2/8] RUN apk update && apk add ca-certificates iptables ip6tables iproute2 && rm -rf /var 2.5s
=> ERROR [tailscale 3/4] RUN wget https://pkgs.tailscale.com/stable/tailscale_1.38.2_amd64.tgz && tar xzf ta 2.4s
------
> [tailscale 3/4] RUN wget https://pkgs.tailscale.com/stable/tailscale_1.38.2_amd64.tgz && tar xzf tailscale_1.38.2_amd64.tgz --strip-components=1:
#7 2.429 Connecting to pkgs.tailscale.com ([2604:a880:2:d0::61c:d001]:443)
#7 2.430 wget: can't connect to remote host: Host is unreachable
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c wget https://pkgs.tailscale.com/stable/${TSFILE} && tar xzf ${TSFILE} --strip-components=1]: exit code: 1
Thanks for this guide!
However, I'm not sure what I'm doing wrong here, but I don't seem to be able to get a direct connections to the nodes on Fly.
i.e. tailscale status
and tailscale ping
shows that connections are going through a DERP relay:
$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
…
100.125.56.76 fly-fra chris@ linux active; exit node; relay "fra", tx 3185 rx 5303
$ /Applications/Tailscale.app/Contents/MacOS/Tailscale ping fly-fra
pong from fly-fra (100.125.56.76) via DERP(fra) in 65ms
pong from fly-fra (100.125.56.76) via DERP(fra) in 57ms
…
2022/03/07 22:09:24 direct connection not established
The same happened when launching the Fly app in fra
, ams
, or lhr
.
Which seems unexpected, as the Fly config asks for UDP port 41641 to be open, which is normally all that's needed to establish a direct connection. For example, on another exit node I have:
$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
…
100.78.27.79 xyz chris@ linux active; offers exit node; direct 89.14.247.123:41641, tx 9001 rx 9002
Are you folks using this setup seeing Tailscale able to make direct connections to the Fly app? Am I missing something?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.