jitsi-contrib / jitsi-kubernetes-operator

Autoscaling Kubernetes Operator

A https://k8s.libre.sh project

The issue

With covid universities and schools have to provide large scale infrastructure for visionconferences. From an environmental standpoint, as much as technical and fincancial, running jitsi at scale can be challenging!

Our magic solution

Kubernetes is becoming THE cloud API, it is beautiful but hard! This cloud infrastructure does provides the building blocks to autscale workload.

That's why we decided to pick this as the base of our solution.

Operator pattern is a way to extend easily the kubernetes API, and describe high level resources like jitsi cluster that translate into low levels resources like linux containers and network configuration.

How it works

Requiremenets

Depending on your region of the world, or your taste.

For the hackathon, we decided to use scaleway, as they provide autoscaling kubernetes cluster as a service

Install our jitsi kubernetes operator

kubectl apply -f https://raw.githubusercontent.com/jitsi-contrib/jitsi-kubernetes-operator/master/deploy/jitsi-operator.yaml

custom jitsi web interface

cf Custom jitsi Web interface

Profit

Now, as the critical path of a jitsi cluster are the JVBs, it will scale based on load.

Challenges:

One JVB is deployed per node for network facilities, we need to know the JVB port Firewall needs to allow JVB ports A new replica of a JVB instance is a equivalent to new node in the kubernetes cluster

Single shard deployments. Multishard can be implemented later. 1 shard = 1 signaling server - prosody and jicofo instance - and multiple JVBs and Web instances 3 Topologies:

Static

If you wan to determistacaly define your deployements and replicas.

Daemonset:

If all your cluster nodes are dedicated to your jitsi cluster, you can use this strategy. JVB processes will be deployed on each nodes.

Autoscalable:

JVB will be autoscaled according to stress level.

We had to tune how to read metrics for the jvb using:

Autoscalable kubernetes cluster Kube-metrics enabled on your cluster with zalendo kube-metrics adapter provisioned

jitsi-kubernetes-operator's Issues

Documentation or prerequisites needed

Dear devs.
It was an awesome thing that you have done so far on jitsi. I really appreciate and want to adopt this jitsi-kubernetes-operator.
I have some hands on with kubernetes. the guide you provided made me few basic things unclear like

1. how to install or setup on GKE,
2. what are the prerequisites needed before installing on GKE like domain name for jitsi as it is mandatory for any kind of installation.
3. how can i customise the container like JVB, web, Prosody, Jicofo etc.

i change the type to NodePort, Then the error happen after access it on my browser

i change the type to NodePort, Then the error "Unauthorized" happen after access it on my browser

Make maxUnavailable more reliable to all use cases.

Currently, if you have 2 nodes with 2 jvbs on each node, it is impossible for the operator to update jvbs.

It is due to the strategy:

  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

In this case, it has to be changed to 50% so that kubernetes will be able to kill one pod after another.

How to intall in a custom Kubernetes Cluster?

Hi,
I did not understand the instructions in the README.md file. what does the following command do??

kubectl apply -f https://raw.githubusercontent.com/jitsi-contrib/jitsi-kubernetes-operator/master/deploy/jitsi-operator.yaml

How can I configure my own Web Address for the Jitsi server? I guess I need somehow a Ingres configuration? Also the files in the /config/samples folder are not clear to me? How should I customize these files an how are these files applied during the deployment?

Can someone give me a hint how to setup my own jitsi server using this kubernetes-obperator?

Annotations

Hi guys! Thanks for the project.

Can I ask you to add possibility to manage annotations/labels for components (svc, deploys, daemonsets)? F.e. they are needed for prometheus SD.

Thanks in advance ;-)

No ws/udp service definition for JVB ?

Hey all,

Great work on the operator so far, I like where this is going and it is almost in a place where I can use it in my cluster.
However, there seems to be an omission of JVB service definition to allow it to reach the UDP and websocket ports on that service.

For example: https://github.com/jitsi-contrib/jitsi-helm/blob/main/templates/jvb/service.yaml

I have set my ingress up with nginx, and have added the server-snippet into the jitsi CRD ingress annotations like you used to have previously:

jitsi-kubernetes-operator/controllers/ingress.go

Lines 30 to 48 in eb013e7

	obj.Annotations["nginx.ingress.kubernetes.io/server-snippet"] = fmt.Sprintf(`add_header X-Jitsi-Shard shard;
	location = /xmpp-websocket {
	proxy_pass http://%s-prosody.%s:5280/xmpp-websocket;
	proxy_http_version 1.1;
	proxy_set_header Connection "upgrade";
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Host %s;
	proxy_set_header X-Forwarded-For $remote_addr;
	tcp_nodelay on;
	}
	location ~ ^/colibri-ws/([a-zA-Z0-9-\.]+)/(.*) {
	proxy_pass http://$1:9090/colibri-ws/$1/$2$is_args$args;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	tcp_nodelay on;
	}`, jitsi.Name, jitsi.Namespace, jitsi.Spec.Domain)

My CRD in my Rancher cluster so far :

apiVersion: apps.jit.si/v1alpha1
kind: Jitsi
metadata:
  name: gnz-jitsi
spec:
  domain: example.com
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: cert-manager-webhook-dnsimple-production
      external-dns.alpha.kubernetes.io/hostname: example.com
      kubernetes.io/ingress.class: nginx-jitsi
      nginx.org/proxy-read-timeout: "3600"
      nginx.org/proxy-send-timeout: "3600"
      nginx.org/server-snippets: |
        add_header X-Jitsi-Shard shard;
        location = /xmpp-websocket {
            proxy_pass http://gnz-jitsi-prosody.jitsi.svc.cluster.local:5280/xmpp-websocket;
            proxy_http_version 1.1;

            proxy_set_header Connection "upgrade";
            proxy_set_header Upgrade $http_upgrade;

            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            tcp_nodelay on;
        }
        location ~ ^/colibri-ws/([a-zA-Z0-9-\.]+)/(.*) {
          proxy_pass http://$1:9090/colibri-ws/$1/$2$is_args$args;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
          tcp_nodelay on;
        }
    enabled: true
    tls: true
  jibri:
    enabled: true
    replicas: 1
  jvb:
    ports:
      udp: 30100
    gracefulShutdown: true
    strategy:
      replicas: 1
      type: static
  region: europe
  timezone: Europe/London
  variables:
    ENABLE_BREAKOUT_ROOMS: "1"
    ENABLE_XMPP_WEBSOCKET: "1"
    NGINX_RESOLVER: rke2-coredns-rke2-coredns.kube-system.svc.cluster.local

I can now reach the xmpp-websocket ok, but the colibri-ws one is currently not reaching as port 9090 is not exposed in the JVB service, plus also the UDP port is not either.

I'm interested in finding out how you have you achieved this so far?

Thanks, Ryan