Audio randomly gets disabled after latest changes to webrtcbin about selkies-gstreamer

[ehfd]

This issue has been tagged as Urgent.

@danisla

[ehfd]

I believe that this issue is because the audio and video WebRTC elements do not cross-check that both protocols have been connected successfully.

[danisla]

How do you reproduce this?

[ehfd]

@danisla Try to reload the web interface multiple times. Especially from a non-local connection passing through the internet. It's not guaranteed to happen every time, but I have at least one other user reproducing this.

The core problem is that the current code seemingly does not ensure that both WebRTC streams are established before exposing the web interface to the user.

[ehfd]

Audio Stats
Latency: 0 ms
Codec: NA
Bit rate: 0.00 kbps

It seems like this is shown even when audio is active. Stats are not shown.

[ehfd]

@Xosrov Could I ask how you solved this problem?

[Xosrov]

Hey, I don't recall such an issue, but I did have a look at the new source code and I notice something:

if audio_only:
    self.build_audio_pipeline()
else:
    self.build_video_pipeline()

This implies the start_pipeline function is called twice; once for video and once for audio. Am I getting this right? If so, the self.pipeline object is being modified and that could cause the signaling process to be disturbed. There needs to be more logging to know for sure, though.

The way I had implemented it was to first preroll the capturing + encoding + packetizing pipeline, then attach and sync a specific webrtcbin to the already running pipeline, which In my experience made initializing the stream a lot faster.

Some other things (unrelated to this topic,) that you could consider adding to POSSIBLY improve the whole stream experience:

• Activating all available header extensions for audio and video. these do show up in the resulting SDP, but I haven't tested if they have noticeable effect.
• Since there are two webrtcbins now, I would enable NACK on both. I'm not sure why you're doing it here; as the if-statement implies you only use it for one and not both?
• I also used to enable FEC specifically in audio streams, but again I haven't tested to see if they have an effect on audio quality.

I hope this helps.

[ehfd]

Thanks for the massive help. @Xosrov
While I'm busy dealing with other things, I'll make sure to bake your suggestions into the code.