bekriebel / livekit-flydotio

An example on how to deploy LiveKit on Fly.io

Install and configure a single-node LiveKit server on Fly.io

⚠️ These instructions may no longer be valid

Unfortunately, I began having several connectivity issues with my LiveKit cluster on Fly. I have since migrated off of the service and no longer maintain these instructions.

This guide is still available in case it proves helpful to anyone else, but you may have difficulty hosting a LiveKit server or cluster using these steps.

LiveKit is an open source platform for real-time communication (WebRTC) based on the Pion Go WebRTC stack. This example allows you to easily run a single-node LiveKit server on Fly.io hosting. This can be used to run a personal server for my LiveKit AVClient Module for FoundryVTT.

To provide a better reconnection mechanism with LiveKit, redis is also deployed to Fly. The redis steps can optionally be skipped, however users may not automatically reconnect if there is an interruption to the WebSocket connection during a session.

Note that this implementation does not allow you to scale to multiple LiveKit servers due to Fly only providing a single BGP Anycast ipv4 address per application. There are ways to accomplish this with Fly, but it is not the topic of this example.

Prerequisites

A functioning Docker installation on the machine you will be deploying from.

Install and configure flyctl

1. Follow Fly's QuickStart guide: Installing flyctl
2. Follow Fly's QuickStart guide: Login to Fly

Launch redis on Fly

1. Switch to the redis folder and review the configuration files. No changes should be needed.
2. Init a new Fly app for redis
   flyctl apps create
3. Create an app name, or let one be auto-generated. Remember this name for later steps, indicated with <REDIS_APP_NAME>
4. Determine your primary region, listed as Region Pool. Remember this for later steps, indicated with <REGION>
   flyctl -a <REDIS_APP_NAME> regions list
5. Create a persistent volume for your redis app
   flyctl -a <REDIS_APP_NAME> volumes create livekit_redis --region <REGION>
6. Add a secure password as an application secret to secure redis. I recommend randomly generating a string for this. Remember this for later steps, indicated with <YOUR_REDIS_PASSWORD>
   flyctl -a <REDIS_APP_NAME> secrets set REDIS_PASSWORD=<YOUR_REDIS_PASSWORD>
7. Deploy the redis app from from the redis path
   flyctl -a <REDIS_APP_NAME> deploy

Launch LiveKit on Fly

1. Switch to the livekit folder and review the configuration files. No changes should be needed.
2. Init a new fly app for LiveKit
   flyctl apps create
3. Create an app name, or let one be auto-generated. Remember this name for later steps, indicated with <LIVEKIT_APP_NAME>
4. Set the region to match your redis app
   flyctl -a <LIVEKIT_APP_NAME> regions set <REGION>
5. Generate an API key/secret pair (for more information, see LiveKit's guide: Generate API key and secret)
   docker run --rm livekit/livekit-server generate-keys
6. Note the generated key/secret pair for use in later steps, indicated with <API_KEY> and <SECRET_KEY>
7. Add the LiveKit key/secret pair as an application secret on LiveKit
   flyctl -a <LIVEKIT_APP_NAME> secrets set LIVEKIT_KEYS="<API_KEY>: <SECRET_KEY>"
8. Add the redis host as an application secret on LiveKit
   flyctl -a <LIVEKIT_APP_NAME> secrets set REDIS_HOST=<REDIS_APP_NAME>.internal:6379
9. Add the redis password as an application secret on LiveKit
   flyctl -a <LIVEKIT_APP_NAME> secrets set REDIS_PASSWORD=<YOUR_REDIS_PASSWORD>
10. Deploy the LiveKit app from the livekit path
    flyctl -a <LIVEKIT_APP_NAME> deploy
11. Congratulations! LiveKit is now running at <LIVEKIT_APP_NAME>.fly.dev with yor API key/secret pair. Note that since we are using the Fly load balancer, which provides SSL termination, the sever is running under secure websockets (wss://) on port 443

From here, you can continue to configure the app however you would like. For example, you can add a custom domain by following Fly's documentation Custom Domains for SaaS.

Support my work

livekit-flydotio's Issues

Running without redis

Thanks for this!

Is redis necessary if we're only running 1 instance?

Link to Fly Login page needs updating

link needs to go to "https://fly.io/docs/getting-started/log-in-to-fly/"

current goes to "https://fly.io/docs/getting-started/login-to-fly/"

Difference is the hyphen added to "log-in"

Docker

Sorry, Im newba here.. What I have to do about this?

A functioning Docker installation on the machine you will be deploying from.

Many thanks

Could not deploy redis

Hello and thank you for your work, I've been looking for an alternative to Jitsi in FVTT for quite some time now.

I tried to use your Fly.io guide but I'm stuck at the first deployment.

It's my first time using Fly.io so it could be something obvious, sorry about that.

PS C:\Git\livekit-flydotio\redis> flyctl -a livekitredis deploy .
Deploying livekitredis
==> Validating app configuration
--> Validating app configuration done
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.8 linux x86_64
Sending build context to Docker daemon  4.096kB
[+] Building 1.0s (6/6) FINISHED
 => CACHED [internal] load remote build context                                                                    0.0s
 => CACHED copy /context /                                                                                         0.0s
 => [internal] load metadata for docker.io/library/redis:alpine                                                    0.9s
 => [1/2] FROM docker.io/library/redis:alpine@sha256:58132ff3162cf9ecc8e2042c77b2ec46f6024c35e83bda3cabde76437406  0.0s
 => CACHED [2/2] COPY start-redis-server.sh /usr/bin/                                                              0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:f7239b1a66f539479237e372d1182b91ae94a455560544cc31078c123611e3f7                       0.0s
 => => naming to registry.fly.io/livekitredis:deployment-1635953675                                                0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/livekitredis]
fd68d64f94fc: Pushed
29b13f0def62: Pushed
2926f8581b1d: Pushed
ba79e026e79d: Pushed
346615b02a36: Pushed
512970cfaf24: Pushed
e2eb06d8af82: Pushed
deployment-1635953675: digest: sha256:43ac96619161395c797feae36c9831f08c561bbb178355153086a606f3e8cd25 size: 1778
--> Pushing image done
Image: registry.fly.io/livekitredis:deployment-1635953675
Image size: 32 MB
==> Creating release
Release v2 created

You can detach the terminal anytime without stopping the deployment
Monitoring Deployment

1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2]
v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to
***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/

When I check the logs, here is what I get:

2021-11-03T15:35:37.623 runner[2e1f128b] cdg [info] Starting instance
2021-11-03T15:35:37.671 runner[2e1f128b] cdg [info] Configuring virtual machine
2021-11-03T15:35:37.678 runner[2e1f128b] cdg [info] Pulling container image
2021-11-03T15:35:38.219 runner[2e1f128b] cdg [info] Unpacking image
2021-11-03T15:35:38.236 runner[2e1f128b] cdg [info] Preparing kernel init
2021-11-03T15:35:38.489 runner[2e1f128b] cdg [info] Setting up volume 'livekit_redis'
2021-11-03T15:35:38.495 runner[2e1f128b] cdg [info] Opening encrypted volume
2021-11-03T15:35:38.897 runner[2e1f128b] cdg [info] Configuring firecracker
2021-11-03T15:35:39.049 runner[2e1f128b] cdg [info] Starting virtual machine
2021-11-03T15:35:39.343 app[2e1f128b] cdg [info] Starting init (commit: 7943db6)...
2021-11-03T15:35:39.375 app[2e1f128b] cdg [info] Mounting /dev/vdc at /data
2021-11-03T15:35:39.390 app[2e1f128b] cdg [info] Preparing to run: `docker-entrypoint.sh start-redis-server.sh` as root
2021-11-03T15:35:39.409 app[2e1f128b] cdg [info] 2021/11/03 15:35:39 listening on [fdaa:0:3878:a7b:abd:0:5244:2]:22 (DNS: [fdaa::3]:53)
2021-11-03T15:35:40.402 app[2e1f128b] cdg [info] Main child exited normally with code: 127
2021-11-03T15:35:40.403 app[2e1f128b] cdg [info] Starting clean up.
2021-11-03T15:35:40.420 app[2e1f128b] cdg [info] Umounting /dev/vdc from /data

Thanks for your help!

Unable to deploy livekit (redis was ok)

Hi. I'm on windows and tried to deploy my livekit instance today but I encounter the following output when I run the deploy command. I followed the guide step by step twice but still no joy. I have docker running and have confirmed Alpine is on there and running

c:\lk\livekit>flyctl -a MYLIVEKITNAME deploy
==> Verifying app config
--> Verified app config
==> Building image
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon 10.75kB
[+] Building 1.0s (5/6)
=> CACHED [internal] load remote build context 0.0s
=> CACHED copy /context / 0.0s
=> [internal] load metadata for docker.io/livekit/livekit-server:latest 0.0s
=> CACHED [1/3] FROM docker.io/livekit/livekit-server:latest 0.0s
=> ERROR [2/3] RUN apk add --no-cache bind-tools 0.9s

[2/3] RUN apk add --no-cache bind-tools:
#5 0.440 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
#5 0.668 140309929212744:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
#5 0.670 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.15/main: Permission denied
#5 0.671 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
#5 0.831 140309929212744:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
#5 0.833 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.15/community: Permission denied
#5 0.835 bind-tools (no such package):
#5 0.835 required by: world[bind-tools]
#5 0.836 ERROR: unable to select packages:

---

Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c apk add --no-cache bind-tools]: exit code: 1