selkies-project / selkies-gstreamer Goto Github PK

Currently, selkies-gstreamer does support gamepads only when the uinput device is provisioned. This is very cumbersome and risky in Kubernetes.

Instead of using that, it may be possible to directly translate them to Xorg operations (and later Wayland #46 using virtual keyboard).

Relevant files: gst-web module and webrtc_input.py.

https://docs.conda.io/en/latest/

This is possible. Conda enables installing packages without regard to ABI or OS compatibility, and GStreamer/GST-Python are usually updated to most new stable and development versions. The plugins are also in the repository. Even if the plugins are old, we can request Anaconda to update them to the recent versions.

The CUDA Toolkit is also always available from NVIDIA. Most dependencies including the X related ones will also be ready in an OS agnostic way.

If we create a conda package for Selkies-GStreamer, this will lead to effortless installations of both gst-web and the Python components absolutely without root whatsoever, because it creates its own dependency environment. This is also the most academia way to install a package (lots of support from ML and other related projects), and it will help usage of Selkies in HPC clusters that use batch schedulers such as SLURM.

We have a choice; adding this package to the conda-forge channel and let them compile the project, or to build the package with GitHub Actions and push it to our own channel. In the latter case, we may provide custom GStreamer builds as well.

Contribution from interested parties are welcome, and this is also a good first issue if you are accustomed to conda.

Pressing either < or > results in > being pressed on the remote.

This breaks tasks like writing HTML in the session.

May need to re-write the python keyboard input handler to use the Xlib directly rather than relying on pynput to do the right thing.
Example

When not used full-screen, local scaling interface does not fit into the window of the web browser in HiDPI clients. Reproducible by setting PC DPI settings above 100% (96 DPI for Linux). 125% (120 DPI for Linux) requires a slight scroll to see the taskbar, and 150% (144 DPI for Linux) requires more scroll to see the taskbar.

Explore how noVNC and Guacamole implements local scaling.
Starting points:
https://github.com/novnc/noVNC/blob/master/app/ui.js
https://issues.apache.org/jira/browse/GUACAMOLE-164
https://issues.apache.org/jira/browse/GUACAMOLE-165

Rather than setting a static cursor size from the cursor_size argument, cursor size should match the client screen DPI.

In my tests, the cursor appears very large.

It doesn't look like there is a way to disable the cursor resize accessibility feature and revert back to the standard cursor sizes, only a fixed forced resize is possible now.

In general, things like remote DPI and cursor sizes should match the client per the Javascript window.devicePixelRatio variable.

The remote screen DPI should also match the client DPI, but that is another issue.

Per conversation in #7 (comment):

• Figure out how to build the nvimage element without the NVIDIA Video Codec SDK requirement.
• Update gst pipeline to support nvfbch264enc option.

• Build and add .deb and .rpm packages (help for Arch or Alpine distros also welcome)

We should definitely do this, but we need help. This requires build procedures for every relevant distro.

• Testing workflows

In every PR, we should test using Selenium and Firefox an end-to-end procedure, starting from container builds to testing that GStreamer starts, and we see that the Start button of the web interface is visible.

• Formatting, automatic code assessment tools, and a single syntax for code documentation

We should incorporate format checking such as pylint or yapf and code testing. Similarly for JavaScript. Code checking and assessments should automatically be done when a Pull Request starts.
People with relevant experience are welcome to reach out.

Code documentation should also be added to the code itself. There are well-documented locations and those that aren't really so.

These should be done before the codebase becomes substantially larger than now.

https://web.dev/i18n/en/webtransport/
https://www.w3.org/TR/webtransport/
https://web.dev/webcodecs/
https://www.w3.org/TR/webaudio/
https://web.dev/media-mse-basics/

WebTransport is a promising new web protocol based on HTTP3. This combined with Media Source Extensions, WebCodecs, and Web Audio API, will enable Selkies to provide equal performance compared to the native UDP protocols used by many high-performance remote desktop protocols such as Parsec. A TURN server will no longer be required to be used, as WebTransport establishes a client-server connection.

This will enable more codecs not restrained to WebRTC specifications, such as H.265 HEVC and AV1, as well as bringing the performance equal to existing desktop client implementations such as Sunshine or Parsec.

However, a WebTransport plugin for GStreamer is not yet ready. Look below.

This feature requires funding for their developers to add. Please reach out if you are interested in funding.

Hi, I am using one of the desktops and am experiencing keymap issues. If I start firefox, I am unable to type anything in the search bar. Restarting the container fixes that, but I am still unable to use the caps lock key.

Thanks in advance.

It seems browsers on mobile phones won't show a keyboard automatically when focusing on an input box on remote.

Maybe selkies could show a virtual keyboard on touch-screens.

Yesterday I checked another git repo that shown a lot of configuration pipelines https://gist.github.com/hum4n0id/2760d987a5a4b68c24256edd9db6b42b#display-test-video, Can we use gst-launcher to create pipeline and run selkies -gstreamer ?. Is it possible add feature session recording and JWT auth with secret key?

It seems that selkies-gstreamer will stuck on Waiting for video stream on Firefox 96.

The status message only shows
``
[19:0:0] [signalling] Connecting to server.
[19:0:2] [signalling] Registering with server, peer ID: 1
[19:0:4] [signalling] Registered with server.
[19:0:4] [signalling] Waiting for video stream.
[19:0:6] [webrtc] [ERROR] attempt to send data channel message before channel was open.
[19:0:6] [webrtc] Received incoming video stream from peer
[19:0:6] [webrtc] Received incoming audio stream from peer
[19:0:7] [webrtc] [ERROR] attempt to send data channel message before channel was open.

``

Apache Guacamole and noVNC provide access to multiple users simultaneously to one screen. They also have functionalities which allow users to set view-only passwords for view-only screen sharing.
It is required to investigate the feasibility of this function for the Selkies project to be used more widely as a stable remote desktop solution.

Also: what about isolated sessions for multiple users, where throwaway desktops are created and scrapped for different users?
Such environment probably requires a lot more things than just Selkies-GStreamer, but the scope for the remote desktop system is addressed here.

Partially unresolved aspect of #28 and #97.

Mouse movement is slower than normal when the client resolution is higher than the remote resolution, and faster than normal when the client resolution is lower than the remote resolution.

The cursor movement should be scaled in the web client then sent to the remote server.

Gstreamer pipeline is not starting because in main.py lambda preventing to starting pipeline loop.run_in_executor(None, lambda: app.start_pipeline())

after deleting lambda pipeline started

and there is another issue If we change from

    while self.bus is not None and self.pipeline is not None:
        while self.bus.have_pending():
            msg = self.bus.pop()
            if not self.bus_call(msg):
                self.bus = None
                break
        time.sleep(0.1)

to print("started") we see

and that means time.sleep(0.1) triggering infinite sleep and preventing to start the session
If we change time.sleep(0.1) to "break" everything is ok.

Python app crashes when pressing any key in a non-English layout:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/selkies_gstreamer/gstwebrtc_app.py", line 1007, in <lambda>
    'on-message-string', lambda _, msg: self.on_data_message(msg))
  File "/usr/local/lib/python3.10/dist-packages/selkies_gstreamer/webrtc_input.py", line 533, in on_message
    self.send_x11_keypress(int(toks[1]), down=True)
  File "/usr/local/lib/python3.10/dist-packages/selkies_gstreamer/webrtc_input.py", line 321, in send_x11_keypress
    self.keyboard.press(keycode)
  File "/usr/local/lib/python3.10/dist-packages/pynput/keyboard/_base.py", line 392, in press
    self._handle(resolved, True)
  File "/usr/local/lib/python3.10/dist-packages/pynput/keyboard/_xorg.py", line 242, in _handle
    with display_manager(self._display) as dm:
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/pynput/_util/xorg.py", line 78, in display_manager
    display.sync()
  File "/usr/local/lib/python3.10/dist-packages/Xlib/display.py", line 182, in sync
    self.get_pointer_control()
  File "/usr/local/lib/python3.10/dist-packages/Xlib/display.py", line 833, in get_pointer_control
    return request.GetPointerControl(display = self.display)
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/rq.py", line 1368, in __init__
    self.reply()
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/rq.py", line 1380, in reply
    self._display.send_and_recv(request = self._serial)
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/display.py", line 612, in send_and_recv
    gotreq = self.parse_response(request)
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/display.py", line 719, in parse_response
    gotreq = self.parse_error_response(request) or gotreq
  File "/usr/local/lib/python3.10/dist-packages/Xlib/protocol/display.py", line 745, in parse_error_response
    req = self.get_waiting_request(e.sequence_number)
AttributeError: 'BadRRModeError' object has no attribute 'sequence_number'

After that, I was unable to access the webrtc app:

0:03:15.931713640   106 0x7fd9bc0b0520 WARN         sctpassociation sctpassociation.c:899:handle_association_changed:<GstSctpAssociation@0x7fd980002590> SCTP event SCTP_COMM_LOST received
0:03:15.931733758   106 0x7fd9bc0b0520 WARN                 sctpenc gstsctpenc.c:872:on_sctp_association_state_changed:<sctpenc1> error: SCTP association went into error state
0:03:16.518575545   106 0x55dac020ff00 ERROR        sctpassociation sctpassociation.c:483:gst_sctp_association_send_data:<GstSctpAssociation@0x7fd980002590> Association not connected yet
0:03:16.518588352   106 0x55dac020ff00 WARN                 sctpenc gstsctpenc.c:646:gst_sctp_enc_sink_chain:<sctpenc1> error: Failed to send data
0:03:16.518606074   106 0x55dac020ff00 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<appsrc1> error: Internal data stream error.
0:03:16.518608505   106 0x55dac020ff00 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<appsrc1> error: streaming stopped, reason error (-5)
INFO:signaling:Connection to peer ('10.0.0.39', 59252) closed, exiting handler
INFO:signaling:Cleaned up 1 session
INFO:signaling:Also cleaned up 0 session
INFO:signaling:Closing connection to 0
INFO:gstwebrtc_app:stopping pipeline
INFO:gstwebrtc_app:data channel closed
INFO:gstwebrtc_app:setting pipeline state to NULL
0:03:17.327469669   106 0x55dac0210d20 WARN                audiosrc gstaudiosrc.c:227:audioringbuffer_thread_func:<pulsesrc> error reading data -1 (reason: Success), skipping segment

Looking for contributors to help out.

Basically, this is the opposite direction to the audio capabilities available now.

1. Fetch microphone permissions from the web browser.
2. Receive the microphone stream and encode to opus within the browser.
3. Send the audio stream using RTP wrapped with WebRTC and the host/container Python component receives the stream.
4. The host/container Python component first decodes the opus codec, and then creates a microphone device and sends audio to the host/container with pulsesink.

The gst-web components should be overhauled to accomodate new improvements such as webcams, printing, and file sharing in the future as well as to finish this job.

Would be convenient to investigate the following issues along the way for better usability if we are indeed overhauling gst-web for this task (Unfortunately, I don't know JavaScript too well compared to Python): #7 (try splitting the audio and video streams and create a total of three webrtcbins as @Xosrov said before), #9 (AZERTY keyboard) #22 (Touch keyboard capabilities), #25 (URL parameters for authentication), #55 (HTML5 Gamepad), #75 (Firefox issue).

Additional issues (intertwined with gst-web and Python): #28 (Erratic mouse), #39 (Multiple users), #48 (WebTransport), #74 (WebRTC chunked DataChannel)

This is when the connection is established on FireFox 109, Windows or Linux. Cursor doesn't work. Only a few metrics work. Waiting for video stream stays there even when the connection is established.

Self-explanatory. Just like VS Code Server and noVNC, a button click in Jupyter should lead to a window with Selkies.
This will help greatly in robotics, simulations, and other kinds of research.

External contribution dearly expected.

It can be a separate project, a PR, or any other form of contribution.
Integration with Jupyter Docker containers should also be possible.

Creating a template for Coder would also be of interest.

Just tried the example from the webpage on a centos8 stream system s/docker/podman/. While connecting with a browser, it just sits there "Waiting for video stream" and the following is printed in the logs:
0:04:47.907950623 47 0x3088f00 ERROR webrtcice gstwebrtcice.c:289:_resolve_host: Error resolving “7125f11a-e853-4553-b81c-ecf214892303.local”: Name or service not known

Reproducible both with nvh264enc and x264enc. Possibly issues with GStreamer configurations or the X11 capturer.

Happens every few dozen minutes or hours.

For now, changing the bitrate then changing it back fixes it. Disabling the compositor also might help.

Screenshots will come up.

Sorry for the complete newbie question but I'm really struggling to get the demonstration container to run...

I have a Ubuntu 20.04 env with latest docker installed... I am running Ubuntu with no X11 operating on the host - so hopefully no port conflicts etc.

When I run the docker run command I just get a repeating error:

screen 0 shmid 0
/entrypoint.sh: 2: [[: not found
error repeats....

I have also tried running the container with --user root but again to no avail...

I'm guessing I'm being a complete idiot but any pointers would be gratefully appreciated... Thanks

INFO:webrtc_input:sending clipboard content, length: 49194
0:05:03.202935604   197 0x7f0f90002b80 WARN       webrtcdatachannel webrtcdatachannel.c:264:_channel_store_error:<WebRTCDataChannel@0x1e04db0> Error: Requested to send a string that is too large
0:05:03.203222887   197 0x7f0f980022a0 WARN       webrtcdatachannel datachannel.c:449:gst_webrtc_data_channel_on_error:<WebRTCDataChannel@0x1e04db0> Error: Requested to send a string that is too large
TypeError: <lambda>() takes 1 positional argument but 2 were given
WARNING:gstwebrtc_app:unhandled on_data_close

To reproduce: Ctrl+C anything that is sufficiently long. All input becomes impossible until session is reloaded.

gstwebrtc_app.py is approaching 1000 lines. It can be a good idea to create a pipeline-based workflow ourselves too.

If this is implemented, optimal low-latency settings for protocol interfaces (for example WebTransport in addition to WebRTC), screen capturing interfaces, video/audio encoders and color converters can be compartmentalized into individual objects which includes the most optimal parameters in separate files instead of housing them all in gstwebrtc_app.py, connected by if conditionals.

This will allow easier accessibility to allow new contributors to add more encoders, color converters, screen capturing interfaces, and protocol interfaces without necessarily needing to understand the rest of the codebase.

Relevant code: requires a rewrite of https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/gstwebrtc_app.py but most code can be recycled. Probably a good idea to create one Python file for each type of components.

I have two issues with mouse pointer:

1. if pointer is not locked, mainly games cannot be controlled as mouse jumps randomly over screen. Could be related to resolution change? It does not happen e.g., with tuxracer. But it does happen in witcher 2.

2. if pointer is locked and remote and local resolution is significantly different, e.g., 4k vs. 1920x1080, pointer motion is very slow, acceleration should be adjusted.

Windows support is in progress with the contribution from the community with @callTx mainly working on the project. The pipeline will add d3d11screencapturesrc, d3d11colorconvert, and wasapi or wasapi2.

Removal of patches to the GStreamer build process in #51 means that the binary build available from the project for Windows will work without additional alterations.

In addition, it would be worthy to consider adding the mfh264enc and mfvp9enc pipelines for Windows native encoding capabilities, on top of the existing x264enc, nvh264enc, vp8enc, and vp9enc pipelines, which also may be used for Windows.

Relevant to #34.

Spun out from #34.

This is a placeholder issue for all discussions associated with supporting Wayland capture.
NVIDIA has introduced GBM support to its 510.xx drivers on Linux, and GStreamer's Pipewire plugin to capture Wayland would be mature enough by now. This also means that pulseaudio and Xorg no longer has to be matched with a timer.

Implementation of capturing Wayland using GStreamer's Pipewire plugin is accessible with https://github.com/H-M-H/Weylus or https://github.com/pavlobu/deskreen and some more Wayland capturing references are available in https://github.com/SunshineStream/Sunshine.
How to deploy a headless Sway DE on Wayland: https://github.com/bbusse/swayvnc-firefox.

When switching to a resolution not found in xrandr, then to another resolution, then back to the newly added xrandr resolution, errors like this are seen:

WARNING:main:stopping ximagesrc
WARNING:main:resizing display from 1920x1080 to 1156x918
INFO:gstwebrtc_app_resize:resizing display to 1156x918
INFO:gstwebrtc_app_resize:adding mode 1156x918 to xrandr screen 'DVI-D-0'
INFO:gstwebrtc_app_resize:creating new xrandr mode: 1160x918_60.00 87.25  1160 1224 1344 1528  918 921 931 953 -hsync +vsync
INFO:gstwebrtc_app_resize:adding xrandr mode '1160x918_60.00' to screen 'DVI-D-0'
INFO:gstwebrtc_app_resize:applying xrandr screen 'DVI-D-0' mode: 1160x918_60.00
WARNING:main:stopping ximagesrc
WARNING:main:resizing display from 1160x918 to 1902x952
INFO:gstwebrtc_app_resize:resizing display to 1902x952
INFO:gstwebrtc_app_resize:adding mode 1902x952 to xrandr screen 'DVI-D-0'
INFO:gstwebrtc_app_resize:creating new xrandr mode: 1904x952_60.00 149.75  1904 2016 2216 2528  952 955 965 988 -hsync +vsync
INFO:gstwebrtc_app_resize:adding xrandr mode '1904x952_60.00' to screen 'DVI-D-0'
INFO:gstwebrtc_app_resize:applying xrandr screen 'DVI-D-0' mode: 1904x952_60.00
WARNING:main:stopping ximagesrc
WARNING:main:resizing display from 1904x952 to 1156x918
INFO:gstwebrtc_app_resize:resizing display to 1156x918
INFO:gstwebrtc_app_resize:adding mode 1156x918 to xrandr screen 'DVI-D-0'
INFO:gstwebrtc_app_resize:creating new xrandr mode: 1160x918_60.00 87.25  1160 1224 1344 1528  918 921 931 953 -hsync +vsync
ERROR:gstwebrtc_app_resize:failed to create new xrandr mode: '1160x918_60.00 87.25  1160 1224 1344 1528  918 921 931 953 -hsync +vsync': b''b'X Error of failed request:  BadName (named color or font does not exist)\n  Major opcode of failed request:  140 (RANDR)\n  Minor opcode of failed request:  16 (RRCreateMode)\n  Serial number of failed request:  35\n  Current serial number in output stream:  35\n'

It looks like the xrandr output parser does not detect that a new mode has already been added and tries to create it again.

This is probably because the modes are name like such: 1160x918_60.00

Need to make the newly created modes canonical, to remove any postfixes from the gtf output.

Hello, I've seen that part of the implementation for "nvfbchevcenc" is done but currently can not be part of the pipeline:

https://github.com/CERIT-SC/webrtc/blob/3ae73f4c7ffc5d08e5957e7aec60f03a1e6cf774/streamer/gstwebrtc_app.py#L176

elif self.encoder in ["nvfbch264enc", "nvfbchevcenc"]:
pass
else:
raise GSTWebRTCAppError("Unsupported encoder for pipeline: %s" % self.encoder)

how can I help to make it work?

When browsing this project as a potential option to replace apache guacamole in my setup, I noticed a lot of the set up and complexity comes from configuring the various WebRTC components to work effectively together. I've recently worked on some implementations of the WebRTC WHIP standard, and I think it could be an effective option to simplify things. There's options for paid hosted services, like Cloudflare Stream, and self-hosted server instances like Broadcast Box

WHIP standard: https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html
WHEP standard: https://www.ietf.org/archive/id/draft-murillo-whep-02.html

Gstreamer recently implemented WHIP support via their rust plugins, which may possibly be used to reduce implementation difficulty. Otherwise, the code from an OBS Studio PR I've worked on may be a useful reference: obsproject/obs-studio#7926

Enabling audio results in high latency (700-1000ms) after first connection. It eventually drops down to something more reasonable (100-250ms).

GStreamer has encoders and screen capturing interfaces from various embedded operating systems, including Raspberry Pi and NVIDIA Jetson. gst-omx is an example of an encoder usable for Raspberry Pi and other ARM SoCs. v4l2codecs in gst-bad enables encoding support for Video4Linux as well.
While support for Android or iOS is probably out of our scope, ARM Linux support should be useful.

Relevant files: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/gstwebrtc_app.py and to add omxh264enc and/or v4l2h264enc, then to add aarch64 capabilities to GStreamer build workflows.

There doesn't seem to be a github discussion area for this project so I'm putting this here.

I'm trying to find the perfect multiple-monitor, low-latency, high-resolution, hardware-accelerated remote desktop solution. I just want to have a thin client with two 4k monitors remotely stream my home computer that has two 4k monitors and a 1660ti.

I'm not any kind of coder or technologist but it seems like all of the components exist for this to work.

Can this project stream my multiple-monitor extended desktop to a thin client with two monitors?

Thanks!

Also, is there a website or anything with tutorials and stuff for this?

@xhejtman Thank you for sharing your improvements in #7! Are you interested in meeting with @danisla and me over audio/video in the Selkies Discord server sometime? We'd like to learn more about your use case and exchange knowledge about gstreamer (and any other shared dependencies). Your teammates are also welcome! 🙂

The issue #43 has been patched by #73, but the core issue is that there is no graceful handling for a DataChannel message which is above the size of 65535 bytes.

Therefore, it would be VERY useful to implement chunked messages for DataChannels, as the benefit can be applicable to any workload which can use DataChannel messages between the client and server using WebRTC.

If implemented, remove the restrictions placed by:

I've deployed the example docker file provided by the repo, locally. But the scroll speed is too fast when using touchpad of the mac(I'm using mac 2020 model), but if I try with physical mouse the scrolling speed is fine.

Is something need to be done from gst-web/frontend or selkies_gstreamerhttps://github.com/selkies-project/selkies-gstreamer/tree/main/src/selkies_gstreamer, or any configuration is required after desktop has started?

Any kind of help is appreciated.

The uinput interface is currently undocumented for anything other than uinput-device-plugin (for Kubernetes). Docs/adequate code should be added for normal OS or Docker interfaces.

Add support for AZERTY (French) keyboard layout.

1. Add the respective environment variables to enable TURN over TLS or TURN over TCP for argparse.
   https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/__main__.py#L267
2. For forcing TURN over TCP and TURN over TLS: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/__main__.py#L154
   Could try changing turn:%s:%s?transport=udp to %s:%s:%s?transport=%s and use format() to prioritize TCP/UDP transport and use turns: over turn: if TURN over TLS is enabled, and change =udp to =tcp to prioritize TCP over UDP.
3. For applying TURN over TLS additionally https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/__main__.py#L183
   Similarly, change turn://%s:%s@%s:%s to %s://%s:%s@%s:%s and switch from turn: to turns: if TURN over TLS is enabled, or add a new block with elif url.startswith("turns:"):. Change format() respectively.
4. Change https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/signalling_web.py#L71 similarly
5. Test whether this works.

Good first issue.

These are the milestones before general availability. I can do some, some I need help.

• Adequete documentation about setup on bare metal (not just containers), all the way from installing to startup. Done.
• An activate file that can be sourced to get the GStreamer environment up regardless of which path it is unpacked to Done.
• How to add selkies-gstreamer in a container and also to optionally compile in a container. Done.
• Make the entrypoint or startup less cumbersome, a quick startup script maybe (with and without an option to start a new Xvfb and also the ability to set which DISPLAY)? Documentation added.
• Include VP8 and VP9 encoding capabilities, although they currently only support software acceleration. The code is already there, but the vpx plugin is not included during GStreamer compilation. Done.
• @danisla Find a way to integrate #6 to Selkies and not leave it to the user/OS or fix the issue altogether. Fixed by #60.
• Check that all of the documentation instructions are valid and none fails. Done.
• Governance event for MPLv2 license. Pull request #62.
• Check fix for python-xlib/python-xlib#233. Fixed by #68.
• Address #43. Patched by #73.
• GStreamer version upgrade. Depends on https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1605. Upgraded to 1.22.5.
• Regardless, potentially convert docker-nvidia-egl-desktop and docker-nvidia-glx-desktop to selkies-project. Done.
• Because of the warning message rtpsession gstrtpsession.c:2435:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Can't determine running time for this packet without knowing configured latency, add a message handler after it does Gst.Pipeline.new(). Something that's going to regularly pop messages from the bus and process them, or a GLib mainloop and a bus signal handler like in that helloworld example.
• Update outdated GitHub Actions pipeline.
• Pull Request and Issues Template.
• Fix existing CI processes, CI for building a documentation website; diet on README.md, clean/update the docs

Plans after the milestone:

• Manual compilation (all of gstreamer, web, py wheel) docs on bare metal on RHEL/SUSE, etc as well as Debian. Nobody uses RHEL-based repositories regularly here, so we should wait for contributions from people who actually do.
• Attempt a resolution to #7 (but not a critical issue). After the milestone.
• Do #15. Try to preserve the single Dockerfile.example file, but use ARGS to install components in conditional. After the milestone.

noVNC:

selkies-gstreamer x264enc:

selkies-gstreamer nvh264enc:

Screenshot with 12 mbps video, 60 fps, 64 kbps audio.

When observed very carefully, the color in "Software Updater" seems a bit blurred for nvh264enc. The eye becomes a bit tired like looking at stereoscopic display. This artifact persists through various media across the screen. While not critical, finding why this happens can be helpful for quality control.

Will need to investigate the NVENC API linked with selkies-gstreamer.

Use XFixes extension to listen to mouse cursor change events and send the updated cursor to the browser over the Data Channel to be rendered locally.

Currently using python-xlib which doesn't appear to support XFixesCursorNotifyEvent.

Another example showing how to use the C Libs directly.

Work left to go

• Capture XFixesCursorNotify event with python-xlib library.
• Convert cursor image to PNG.
• Send cursor image over Data Channel
• Decode and render cursor with CSS.

To make this project easier to build on top of, create some base images for various use cases. The Dockerfile.example has many of the dependencies codified, but more purpose based images would be ideal.

Images Proposal

Base Images:

• images/ubuntu/base: image with most of the system dependencies needed to use the software encoder with xvfb. Includes gstreamer, gst-web, python app, and entrypoint script. Dockerfile should be parameterized for both Ubuntu 18.04 and 20.04.
• images/ubuntu/gpu: starts from base image and adds cuda libraries to support GPU accelerated encoding with NVENC.
• images/ubuntu/glx: starts from gpu image and adds support for hardware accelerated GLX with integrated X11 server.
• images/ubuntu/vgl: starts from gpu image and add support for off-screen rendering with VGL using EGL DRI method.

Example Images:

• images/examples/minimal-desktop: example image similar to the current Dockerfile.example, installs basic window manager and desktop experience. Parameterize Dockerfile with variants for the various base images.
• images/examples/gnome-desktop: example image with gnome desktop and HiDPI support.
• images/examples/xfce-desktop: example image with xfce desktop and HiDPI support.

Entrypoint

It should be relatively simple for users to integrate the base image in their own projects.

The python app and all of it's dependencies (like pulseaudio) should run in the background.

Some options:

• entrypoint wrapper script (like the selkies-vdi app streaming images do today)
• supervisord
• systemd (only supported on sysbox runtime)

Has anyone looked into using the Rust element webrtcsink ?

https://mathieuduponchelle.github.io/2021-12-14-webrtcsink.html

One nice feature is that is handles congestion control.

xclip apparently works on Xorg from the client to the host, but not from the host to the client. Both directions work using Xvfb.

This is reproducible on https://github.com/selkies-project/docker-nvidia-glx-desktop (where Xorg is used) and works correctly both directions on https://github.com/selkies-project/docker-nvidia-egl-desktop.

This might be limited to https://github.com/selkies-project/docker-nvidia-glx-desktop and not on bare metal remote hosts.

Note: Works and does not work for me now randomly with an unknown reason.

11006e3

As specified here, the upstream python-uinput package is not maintained for 6-7 years.

Might need a change in the uinput interface to a project that continues to update, or maintain a selkies-project fork.

Moreover, why can't we completely replace pynput.mouse.Controller() and pynput.keyboard.Controller() with Xlib.xtest?

Also, we need to update all the dependencies for JavaScript dependencies.

A situation where an embedded browser such as Android WebView or VR headsets may cause selkies-gstreamer to be unable to be authenticated with the basic authentication option. As a solution, URL parameters ?username=myusername&password=mypassword can also be provided as an alternative in web browsers in embedded systems. Ideally, selkies-gstreamer can prioritize URL parameters if provided and then show the dialog for basic authentication if both parameters aren't provided.

Additional:

Allow basic authentication passwords to be passed in hashed form, not plain text.

This is related to #39.