[Inquiry] general progress and roadmap of this project about stunner

[rg0now]

Sorry, I somehow missed this message.

Anyway, if I understand your question then there might be a slight misunderstanding here. The part "STUNner does the connection (session) mapping by exposing a single UDP port for the TURN relay connection (client facing)" is absolutely correct, that's the idea here: use a single Kubernetes LoadBalaner service to expose the entire media plane to the outside world over a single TURN URI. So far so good. The second part, however, is not quite right: "...and manage multiple media channels with media servers behind respectively, meaning the client will always get the same port for a relay candidate".

Well, that's not entirely the story here. In fact, STUNner creates a separate relay candidate for all client connections, that is, the media servers will always perceive a different sender port when receiving messages through STUNner. That's why it is very important to let clients go through the entire ICE process: clients obtain the ICE relay candidate from STUNner and send it along to the media server (usually, this is mediated by the application server), which will then know where to expect UDP/RTP packets from (from the relay candidate obtained via STUNner).

Here's an turncat output excerpt from an actual GKE deployment, where we create two turncat tunnels via STUNner:

turncat INFO: 2022/06/10 11:00:33 new connection: client-address=127.0.0.1:54077, relayed-address=10.116.0.10:19129
...
turncat INFO: 2022/06/10 11:00:49 new connection: client-address=127.0.0.1:33969, relayed-address=10.116.0.10:9601

Observe how both client connections (the one with client-address=127.0.0.1:54077 and the other one with client-address=127.0.0.1:33969) receive a separate relay candidate from STUNner (the first client gets relayed-address=10.116.0.10:19129 and the second one gets relayed-address=10.116.0.10:9601). The IP happens to be the same (the IP address of the pod where stunnerd runs) but the relay transport port is different for both clients.

Note that when STUNner is scaled out (i.e., runs multiple pods) then the transport relay IP will most probably differ too: Kubernetes will load-balance both clients to different stunnerd pods and then the relay transport address will be the pod IPs of the pod that happens to be hit by each client request.

This means that there is no practical limit on how many client connections and media servers STUNner can support, apart from the UDP port range available to the stunnerd pods (by default, this is limited to the port range [10000:20000] but you can easily override this as we did above). So if you scale STUNner to, say, 100 pods, then you will have the theoretical limit of 100x65536 client connections.

Performance is another question though. So far, we haven't been able to max out the performance a single STUNner pod yet, but if a single pod cannot keep up with the load then just kubectl scale deployment stunner 100 and then you'll get 100x the performance. That's the beauty of Kubernetes...:-)

If this is still not enough, we plan to release a Linux/eBPF STUN/TURN accelerator for STUNner, which would essentially bring per-pod performance to the range of millions of packets per sec. This should be more than enough for even the largest deployments, so bear with us until we get there (after v1.0.0).

Hope this helps!

[rg0now]

Last time we checked we got >1.2 mpps packet rates with a fairly naive UDP proxy attached to the eBPF/tc hook, and around 1.7 mpps with eBPF/XDP. We hope to get similar numbers with the STUNner accelerator too, but note that handling the specifics of the TURN wire protocol in eBPF will not be trivial and it may have its performance penalty. I guess at these speeds it will be the Kubernetes CNI or the cloud infra that become a bottleneck, and not STUNner. But it's quite fast already: you can easily get 1--2 kpps without any acceleration, with vanilla STUNner. We hope to release some benchmark results soon!

As per Cilium: as much as we are massive fans of Cilium, unfortunately we cannot use it right away: the Cilium data-plane does not know how to handle the STUN/TURN protocol formats. But we at least hope to avoid any interference with it, which will be a challenge in itself given how Cilium uses the same eBPF kernel hooks we also intend to use. We'll see!

[rg0now]

1-2 kpps for one RTP stream right?

It's actually multiple iperf UDP streams encapsulated in separate TURN streams; details will follow soon.

I close this issue now, but feel free to reopen anytime.

[rg0now]

Hi, thanks for the inquiry.

The milestones are not, well, set in stone: we're constantly changing the roadmap as we experiment with STUNner and see the missing features. Currently the major painpoint is the Kubernetes operator to wrap the low-level STUNner configs in high-level Kubernetes-like YAMLs/CRDs. After we have that in place (1-2 releases from now) we will look into scalabilty and observability a bit more. Note that scaling up STUNner and the media servers behind it is already super-simple, but scaling down may remove active sessions so that's not optimal. We have a couple of ideas on how to handle that, we expect a working PoC in 1-2 months. Best would be to check back then, but feel free to join the Discord, most discussions and planning happens there.

[imcom]

Hi @rg0now , glad to hear the plan! We will follow up accordingly.

Just a quick question, maybe, what's the practical number of media servers for a single STUNner to handle? If I understand the topology correctly, STUNner does the connection (session) mapping by exposing a single UDP port for the TURN relay connection (client facing) and manage multiple media channels with media servers behind respectively, meaning the client will always get the same port for a relay candidate and then STUNner relay the stream on an established 4-tuple to one of the media servers, am I correct?

[imcom]

Hi @rg0now, your kind answer is such a great reading, really appreciated! Things got busy lately so could read it through earlier. Without eBPF tuning, a single pod should still be able to handle around ~500Kpps I assume? Probably STUNner can also take advantage of the Cilium network in k8s directly?

[imcom]

1-2 kpps for one RTP stream right? I am not familiar with the TURN wire protocol, just have one month experience setting up coturn ... Looking forward for the benchmark!

the Cilium data-plane does not know how to handle the STUN/TURN protocol formats.

I see I see, I thought you wanted to use eBPF just for forwarding. Yup, the coexistence matters. Apart from Cilium, the Calico project also installs eBPF hooks, getting common in k8s data-plane