[BUG] iOS doesn't play audio about neko

[mbattista]

I found something interessting.

neko/client/src/neko/base.ts

Line 313 in f59128c

private onTrack(event: RTCTrackEvent) {

At this point the event.stream[0] from safari has only one track, while on chrome it has two already. But even if I skip the Video track instead of the audio track in the In the index.ts the outcome stays the same. Only video, no audio.

So I think it could be one of 3 problems:

1. Malformed SDP. Safari parses something on the SDP different then the other browsers, which results in only one track on the stream.
2. Safari ignores audio tracks in webrtc if not specifically asked for it. Perhaps we need to have a special offer or remoteDescription while on an Safari device.
3. Safari removes the audio track as soon as possible, because the audio on a recvonly webrtc session should be unmuted be the user, which could means, adding the track after unmuting again on server side via signaling.

I will try what happens if neko askes for camera / mic permissions before login. In the Apple documentation they write, that giving those permissions should result in a webrtc session with sound. (Via Apple documentation )

[m1k1o]

Spent almost 3 hours on this, even implemented automatic negotation using onnegotiationneeded handlers, even refactored whole webrtc (what was a good move, but not needed for this). Finally, the problem was different.

As @mbattista said:

Safari parses something on the SDP different then the other browsers, which results in only one track on the stream.

Yes, Safari needs explicitly set id for video and audio in SDP. Adding those fixed the issue.

Also deployed simple overly to click to unmute, so that it can be used.

I tried it with iOS15.0.1. If you can give me more test results, that it works even for different versions, I can finally close this isssue.

[alectrocute]

@m1k1o You are a fantastic developer.

[Adastras]

@m1k1o No problem, just wanted it known so people wouldn't say "but it still doesn't work for me". 😄

[m1k1o]

I'd suggest adding simply mute/unmute option to settings. Or just try to find releated event for touch devices to start sound, because atm it is only click:

neko/client/src/main.ts

Lines 33 to 39 in 3dbb265

	this.$accessor.setActive()
	if (this.$accessor.settings.autoplay && this.$accessor.video.playing) {
	this.$accessor.video.setMuted(false)
	}
	window.removeEventListener('click', click, false)
	}
	window.addEventListener('click', click, false)

Fullscreen issue: Although I tried to fix it already in 2782a1d, it does not seem to work and needs more analysis.

[m1k1o]

Fulllscreen is now working even for iOS but that did not help to mitigate the original issue for me. Audio still does not work, even after unmutung.

[joaofrenor]

I'm not sure if it gonna work but seams to be a issue on webrtc runing at mobile pion/webrtc#716

[joaofrenor]

What if you set autoplay by default as false on iOS and when I click to play on it unmutes the player as well

[mbattista]

I'm not sure if it gonna work but seams to be a issue on webrtc runing at mobile pion/webrtc#716

I think this is a different issue, since the video is playing.
It could either be that the video is not properly unmuted via javascript in iOS similar to videojs/video.js#4478 or that the webrtc audioTrack itself gets muted on iOS devices and has also to be unmuted.

I will look into it.

[joaofrenor]

I tried to force unmute using this.$accessor.video.setMuted(false) on iOS safari console, but didn't work.

[alectrocute]

IMHO this is the "biggest bug" currently present on neko.

[DomiiBunn]

No updates on this?

[m1k1o]

I could not fix it, tried but it simply does not work. That's why this is marked as help wanted.

[DomiiBunn]

Not digging in too deep into the code but afaik the audio is muted by default. Also the audio controls are not visible on a mobile device. So a simple check like

if(/(android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|ipad|iris|kindle|Android|Silk|lge |maemo|midp|mmp|netfront|opera m(ob|in)i|palm( os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows (ce|phone)|xda|xiino/i.test(navigator.userAgent) 
    || /1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r |s )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1 u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp( i|ip)|hs\-c|ht(c(\-| |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac( |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt( |\/)|klon|kpt |kwc\-|kyo(c|k)|le(no|xi)|lg( g|\/(k|l|u)|50|54|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-| |o|v)|zz)|mt(50|p1|v )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-| )|webc|whit|wi(g |nc|nw)|wmlb|wonu|x700|yas\-|your|zeto|zte\-/i.test(navigator.userAgent.substr(0,4))) { 
    unmute ()
    setVolume(100)
}

Might be the solution
At least on Android i needed to change it to a desktop site and unmute

[alectrocute]

Perhaps the unmute method needs to be executed within a valid user gesture handler? iOS is notoriously annoying with this.

[DomiiBunn]

Would be a good option too. I mean if you join a session i think you'd want to hear the audio

[m1k1o]

You can fullscreen video, and then you see native player controls where you can unmute it. And it does not work even then, so i don't think it has to do with muted state.

[DomiiBunn]

It's in Vue so i can poke about in a while. Currently busy on a few projects

[mbattista]

I could also test it with an iOS device and sound worked!

That bug came with my pion v3 updates.

Yes, Safari needs explicitly set id for video and audio in SDP

Did you find any documentation stating this or was this just the conclusion you draw after the tests?
If there is a documentation I would be thankful if you could post it here, for future references.

[m1k1o]

Unfortunately no, I did not find documentation. That was just my trial and error conclusion. But I am trying to find some docs about it, it should be somewhere explained.

[Adastras]

I tried it with iOS15.0.1. If you can give me more test results, that it works even for different versions, I can finally close this isssue.

@m1k1o Unfortunately audio still seems dead on 14.8.1. ☹️ Great that it works on 15+ though.

[m1k1o]

@Adastras thanks for testing. I don't have iOS 14 device, so this cannot be fixed by me. I should put it to readme, that iOS it works for 15 only.