Support remote controlling retro console devices about selkies-gstreamer

[ehfd]

Hmm... We are focused on providing a remote desktop that offers a way to control a remote screen as well as send the remote screen to the client.

Streaming only videos of such devices is pretty trivial and there are WebRTC gateways that would provide a Web UI as well. I am unsure what the gain would be by supporting to send a unidirectional stream like the description.

Could you elaborate on how this request aligns with the Selkies project?

[Wanimatrix]

I understand sending over a unidirectional stream is trivial and can be done with other WebRTC gateways. Maybe not all described use cases are that relevant.

However, my personal use case is remote play on the original retro consoles (eg. NES, SNES, ...). When attaching the HDMI to my server using a capture card (represented as UVC device) and streaming it over the local network I would be able to see the gameplay but there is no way to control the game. Therefore, I also need the controller input to be streamed back to the server and then (via custom implementation not related to this project) pass it to the controller input of the console.

Without this enhancement the video from the capture card should first go through the display stack to be shown on the screen before selkies-gstreamer can stream it to remote devices. This would definitely add to latency in a scenario where low input latency is key.

To provide more flexibility for other people's use cases in general supporting appsrc can be very useful. This allows the src of the video can come from anywhere and can have any gstreamer filter be applied to it before passing it on to selkies-gstreamer.

This implementation can help all use cases where video comes from an external source and user input is needed.

[ehfd]

Okay. Let's say that we can use appsrc to stream arbitrary screens to Selkies. However, the interface to relay the control input is a wholly different thing to do. This is processed independently of GStreamer.
We have a sophisticated framework of a few thousand lines to relay input to the Linux X server. Is there a unified API for all those console devices so that we don't have to implement the input pipeline every time there is a new device?

Because you still have to go through a Linux host in whichever case, I don't believe that the latency benefits will be large enough.

[ehfd]

We will consider this proposal, IF there is a unified API to process and pass inputs for such retro consoles and video devices.

Otherwise, retro game emulators like RetroArch also are meant to run on an X server.

[Wanimatrix]

There is a standard for passing input info over USB/Bluetooth/...: HID format. But unfortunately in reality it is not as simple as that: eg. for sending inputs over USB the host running selkies-gstreamer should have a USB 'OTG' port so it can act as a USB slave. This is a port most PC's don't have as it was originally created for handheld devices like smartphones.

For my personal situation, I plan to abuse the UART debug port of a Blueretro device so I can pass inputs over a wired connection directly to the retro console. This device supports several retro consoles and input can be transmitted over UART in the HID format.

Because the way of handling input is quite scattered I don't think it is useful to implement such custom input forwarding: anyone who wants to use this can catch input events and forward them in a way that is suitable for their situation. But I still think it is useful to provide support for another source for video than the X server/wayland, even if it is only for applying custom gstreamer transformations to the video capture of the display.

Further, I should note that I am relatively new to gstreamer and after reading in a bit more I think providing support of something like shmsrc + gdpdepay should be used to allow connection with a gstreamer pipeline from another process. The appsrc plugin seems to be used only for getting data from the same process that is not a gstreamer pipeline into a gstreamer pipeline.

[ehfd]

I'll leave this issue open, but supporting external sources and external input devices should not be our scope, as the current implementations perform everything inside the operating system and don't even rely on a connected physical monitor.
Way too complex to keep external devices in mind for little benefit.

There are a lot of different requirements for "custom" input forwarding. If you learn GStreamer a bit more, you should find that replacing ximagesrc with a different source is not that hard. However, input relaying will be very tough.
Right now, 3/4 of the code is about passing the input to the X server, not obtaining the screen as a stream and encoding them. Screen capturing is not a big concern.

The X server is universal to everyone using a Linux operating system. What you are proposing, however, is to handle a set of fragmented devices with different ways to obtain its stream and different ways to process input.
It will take around a couple dozen lines to add appsrc or shmsrc + gdpdepay (with the only purpose of relaying the video/audio to the web interface without the ability to control the source device - There are WebRTC gateways for cameras that can do this better).
However, it will take over 2,000 lines (if even possible) to be able to control one of such external devices. A few hundred to thousand lines will be added for each other device.
I repeat, adding a different appsrc or shmsrc + gdpdepay video source will not solve anything.

https://github.com/darthcloud/BlueRetro looks interesting, though.

I suggest forking the project, changing ximagesrc to appsrc or any other source plugin like you said, and trying to apply BlueRetro into webrtc_input.py. If this is done to the main repository, this will add an overwhelming amount of code for a feature only a fraction of the people will use.
(But you will have to basically rewrite this repository if you go this path.)

[ehfd]

Plus, the latency added from the X server is miniscule. I am pretty sure the largest portion of the latency comes from the internet network, not the X server. If there is an application on Linux that lets you control the retro devices, you should use it instead of going through all this effort.

Even if you try to implement a method to directly control the retro device, you still pass through the computer that runs Selkies, which relays the video stream that you obtain and pass the input commands.

I assure you that latency differences will be minimal even if you go through the X server virtual display.

[ehfd]

https://github.com/LizardByte/PyStream

@Wanimatrix Perhaps this is on your direction.