[META] WebTransport or Full WebSocket support / Investigate external libraries over GStreamer for transport protocols about selkies-gstreamer

[ehfd]

I am starting to believe that for both WebTransport and WebRTC, using custom Python solutions outside of GStreamer such as https://github.com/aiortc/aioquic or https://github.com/aiortc/aiortc is becoming plausible, since as long as an RTP stream is generated with GStreamer, WebRTC/WebTransport operations can definitely be separated.

This can be done soon instead of waiting for GStreamer and with much less funding.

This is because using GStreamer plugins for transport is slowly showing less flexibility and performance issues, because they are mainly focused on media processing rather than protocols, and browser issues are fixed much slower. Using a dedicated library for WebRTC or WebTransport QUIC will solve transport specific issues more versatilely.

While the core developers ourselves are short of time to do this in the meantime, and we will focus maintaining the GStreamer native webrtcbin/webrtcsink pipelines, people are welcome to set bounties or help out in this way instead of implementing a GStreamer plugin. A substantial bounty may allow us to implement this ourselves.

This will likely solve #7 for once and for all.

[ehfd]

https://www.w3.org/TR/webcodecs-hevc-codec-registration/
HEVC is slowly coming.

[ehfd]

Dependent on:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1571

[ehfd]

https://www.phoronix.com/news/Mozilla-Firefox-114

Firefox now supports WebTransport as of 114. WebGPU is also recently implemented and prototyped as of 116.

Firefox still requires the implementation of the WebCodecs API, but that makes Safari the only major browser engine without WebTransport.

[ehfd]

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1261#note_2053233

A hypothesis I have is that Issue #7 might indeed be something from the web browser client.

If true, we need our own audio stack going through MSE using RTCDataChannel or an equivalent WebTransport interface, it would be sensible to use external WebRTC libraries as well as a new signalling server. Since the only thing we can use from GStreamer's webrtcbin would be the video sink.
Solved differently in #7. Left for historical information.

[ehfd]

#7 is fixed. But WebTransport support in web browsers made things very hard for serving in localhost or for self-signed certificates. Good technology but needs waiting.

[ehfd]

There are restricted environments where only Connection: Upgrade to WebSocket from plain HTTP is available (such as banks).
This might require aiohttp as well as python-websockets.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism#upgrading_to_a_websocket_connection

It might be a good idea to pair WebSocket + WebCodecs, perhaps instead of the widely intended WebTransport + WebCodecs pairing.

WebCodecs is better than MSE or WASM in terms of latency, and it can be used to be paired with WebTransport as well. That way, both WebSocket and WebTransport will use the same media decoder if we implement both.

In conclusion, I think implementing good old WebSockets has its point.

[ehfd]

Things that are relevant for using WebRTC without exposing various ports for the TURN server, perhaps.
(Continuously updated)

https://github.com/amitv87/turn_ws_proxy
https://groups.google.com/g/turn-server-project-rfc5766-turn-server/c/un2u2xsrjLA

[ehfd]

Archive of relevant discussions from #7

@ehfd

Opus + MP4? https://chromestatus.com/feature/5100845653819392

https://web.archive.org/web/20211003152946/https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc

• https://github.com/parsec-cloud/parsec-sdk/blob/master/sdk/parsec.h
• https://github.com/evshiron/parsec-web-client
• https://github.com/robterrell/web-client

@ehfd

https://www.w3.org/2018/12/games-workshop/slides/21-webtransport-webcodecs.pdf

Looks like WebTransport + WebCodecs (in implementation within Firefox) is the modern alternative to WebAssembly or MSE. In progress to be implemented in Firefox.

I’ve been tracking WebTransport closely and prototyping it locally off and on the past year.

@danisla

It’s interesting but not ready for the Selkies Project yet. Right now, it works well in a tightly controlled production environment. However, today it requires a lot of configuration, valid certificates, browser command line arguments and OS exceptions that make working it locally very challenging.

@ehfd

Self-signed certificates look possible but complicated. By the way, is mainly using WebSocket that bad (for a fall-back protocol) compared to WebRTC? - Since it's what Guacamole, NoVNC, etc uses.

https://docs.libp2p.io/concepts/transports/webtransport/

When connecting to a WebSocket server, browsers require the server to present a TLS certificate signed by a trusted CA (certificate authority). Few nodes have such a certificate, which is the reason that WebSocket never saw widespread adoption in the libp2p network. libp2p WebTransport offers a browser API that includes a way to accept the server’s certificate by checking the (SHA-256) hash of the certificate (using the serverCertificateHashes option), even if the certificate is “just” a self-signed certificate. This allows us to connect any browser node to any server node, as long as the browser knows the certificate hash in advance (see WebTransport in libp2p for how WebTransport addresses achieve this).

WebTransport in libp2p
WebTransport multiaddresses are composed of a QUIC multiaddr, followed by /webtransport and a list of multihashes of the node certificates that the server uses.

For instance, for multiaddress /ip4/192.0.2.0/udp/123/quic/webtransport/certhash/, a standard local QUIC connection is defined up until and including /quic. Then, /webtransport/ runs over QUIC. The self-signed certificate hash that the server will use to verify the connection.

The WebTransport CONNECT request is sent to an HTTPS endpoint. libp2p WebTransport server use /.well-known/libp2p-webtransport. For instance, the WebTransport URL of a WebTransport server advertising /ip4/192.0.2.0/udp/1234/quic/webtransport/ would be https://192.0.2.0:1234/.well-known/libp2p-webtransport?type=noise (the ?type=noise refers to the authentication scheme using Noise).

@wanjohiryan

Yeah, i can second what @danisla is saying... local development for webtransport is hectic at the moment especially for anything working on with the browser (server-side quic development is the easiest).

For example, for warp we must generate self-signed certs every 14 days, as well as a fingerprint hash as a workaround to having to execute bash scripts each time we need to refresh our site in order to force Chrome to allow QUIC connections.

Also, there is currently no way to load balance QUIC connections, UDP load balancers are great and all but they take away the QUIC's ability for roaming support (you can change internet connections, e.g mobile data to Wifi, without having to reinitiate the handshake to the server)

QUIC is shiny, promising and all... however we are not there yet