Sound support about babylonnative

[bghgary]

Unfortunately, sound support is not actively being worked on right now. We attempted to start this a couple of times, but they both didn't work out for various reasons. cc @docEdub

[markgrossnickle]

No ability to play audio seems like a deal breaker for many game projects. It's why were are going to go with another platform fwiw.

[rootVIII]

Would love to see audio functionality added to this project.. even if only for a few select targets/OSs.

[FranticOwl]

@bghgary, more than a year later and this is still an open issue?

Is there any actual plan to build support for Babylon Native, so that frameworks such as React-Native can simply use the sound API from Babylon to add audio to scenes?

[docEdub]

Love the idea of having community help getting this implemented! @bghgary thoughts on this? Could @matthargett pick up where Nick left off?

I was able to hack together a BabylonNative audio implementation last year based on Nick's work. Here's the blog post I wrote afterward: https://medium.com/@babylonjs/moving-the-ball-forward-on-babylonnative-audio-44d54de019bc.

[Drigax]

We may also want to consider:
https://www.videolan.org/vlc/libvlc.html

It may be overkill for our needs, (we also may want to consider the implications of its license) but seems to have robust support for 3d sound, media streaming, and filter effects if we choose to implement a subset of WebAudio.

[Drigax]

Currently we are still investigating the feasibility of incorporating Chromium build artifacts that contain WebAudio, and the Chrome media system libraries and dependencies. Ashu Tatake has taken the first steps in this approach.

@ryantrem had a very interesting concern regarding this first option, Apple has been known to inspect and remove apps that contain Chromium source. We should look into which apps were known to be removed for this reason, if there was any justification provided, and whether Apple has a clear policy on whether it is a blanket ban on any chromium-derived binaries, or specific parts of Chromium which are not allowed.

I'll follow up by investigating this as well.

EDIT: I'll leave this comment up, in case this becomes a future concern again, so far I can only see this issue with Electron as related:
electron/electron#20027

since Electron was using Apple internal APIs, violating the TOS. Though we should make sure there are no similar problems with leveraging the Chromium source as well.

[bghgary]

As noted here (https://forum.babylonjs.com/t/source-for-n-api-implemented-bindings-of-web-audio-found/22132), might be interesting to check out this also:

https://github.com/node-3d/webaudio-raub

[matthargett]

As we approach the next phase of our product, which includes bringing Babylon React Native to visionOS, Spatial Audio is a need that we have. What can we do at Rebecker Specialties to help close this key gap that also affects using Babylon React Native on AOSP-based headsets (Pico 4, HTC Vive XR/Focus, etc)?

[bghgary]

Let's check with @docEdub. He was looking at this at some point. Any thoughts @docEdub?

[CedricGuillemet]

Valve Steam audio supports major platforms and is apache 2.0
https://github.com/ValveSoftware/steam-audio

[bghgary]

https://github.com/resonance-audio/resonance-audio

[CedricGuillemet]

Some more tiny libraries here
https://github.com/nothings/single_file_libs#audio

[bghgary]

From forum post: https://forum.babylonjs.com/t/babylonnative-questions/9716

https://github.com/LabSound/LabSound

[chrisfromwork]

some real time filter libraries:

https://github.com/vinniefalco/DSPFilters
https://github.com/berndporr/iir1

[chrisfromwork]

https://github.com/andrewrk/libsoundio
https://github.com/PortAudio/portaudio

[chrisfromwork]

https://github.com/cycfi/q

[CedricGuillemet]

@Drigax Looks interesting for video playback support as well. I'm curious about the footprint on mobile.

[Drigax]

<style> </style>

Project Name	Link To Project	Does it implement an audio backend or directly interact with OS-included backends?	Does it implement DSP Effects?	Does it use a modular or node-based processing pipeline?	What does it do?	Windows/Win32	Windows/UWP	Android	MacOS	iOS	Linux	License	Notes
Chromium	https://source.chromium.org/chromium/chromium/src	✅	✅	✅	Full browser implementation. Includes implementations for WebAudio API with platform audio backends	✅	✅	✅	✅	✅	✅		This is our gold standard (because it pretty much is the standard.) Supports all the funtionality of WebAudio without additional dependencies, and works as a full multimedia backend.
SoLoud	https://github.com/jarikomppa/soloud	❌	✅	❌	Portable audio processing library with extensible backend support	❔	❔	❔	❔	❔	❔	Zlib	This isn't necessarily an all-in-one solution, as it relies on pulling in an audio backend as a dependency. However it has a number of audio effects implemented to support the subset of WebAudio that we need to implement.
libSDL	https://github.com/libsdl-org/SDL	✅	❌	❌	Low level cross-platform I/O interface	✅	❔	✅	✅	✅	✅	Zlib	This is more than just a cross platform audio backend (and interface to other desired backends), this library aims to solve multiple cross platform I/O issues such as input/haptics, graphics, sensors, file system access, etc…
Resonance Audio	https://github.com/resonance-audio/resonance-audio	❌	✅	✅	Audio processing library with node graph and a wide library of built-in DSP effects	❔	❔	❔	❔	❔	❔	Apache 2	This is a DSP middleware package. While it should support the functionality of WebAudio, it requires pulling in an audio backend implementation
JACK	https://github.com/jackaudio/jack2	❌	❌	❌	Audio server that allows for inter-process audio routing.	❔	❔	❔	❔	❔	❔		This isn't an audio processing engine, or backend. JACK is an audio server that allows audio to be sent between processes ex: sending audio from one DAW to another that doesn't explicitly support each other. This doesn't seem useful for this usecase, but this is useful to list so its not misconstrued.
MiniAudio	https://github.com/mackron/miniaudio#supported-platforms	✅	⚠️	❌	Single-file crossplatform audio I/O	✅	❔	✅	✅	✅	✅	Public Domain OR MIT	This is a minimalist, single file audio engine with built in support for OS-shipped backends (DirectSound, WASAPI, CoreAudio, ALSA, Aaudio, etc…) There are some DSP effects included as well.
libsoundio	https://github.com/andrewrk/libsoundio	✅	❌	❌	Cross platform audio I/O library	✅	❔	❌	✅	❔	✅	MIT	This is a cross platform audio I/O library that interacts with a number of platform and 3rd party audio backends.
portaudio	https://github.com/PortAudio/portaudio	✅	❌	❌	Cross platform audio I/O library	✅	✅	❌	✅	❔	✅	Custom, Permissive License	This is a cross platform audio I/O library that interacts with a number of platform and 3rd party audio backends.
libvlc	https://wiki.videolan.org/LibVLC/	✅	✅	✅	Cross platform media I/O library	✅	✅	✅	✅	✅	✅	LGPLv2	This is a cross platform media player, with support for network streaming of media, as well as local playback. Support for simultaneous playing of one-shot sounds are not known, given the nature of the full application. This does have directly plug into platform audio and video backends.
LabSound	https://github.com/LabSound/LabSound	✅	✅	✅	WebAudio Port that uses libnyquist for DSP and miniaudio as a backend	✅	❔	✅	✅	✅	✅	BSD	This project began as a port of WebKit's WebAudio implementation. Some 3rd party solutions are used for audio processing and backend, namely libnyquist and miniaudio.
Key:
❔	Not conclusive, need more information
✅	supported /included
❌	Not supported / not included
⚠️	Some support/some inclusion

[Drigax]

I've spent the past few days researching various popular audio solutions, and have tabulated some of their qualities into an easy to digest chart. This should allow us to compare solutions/a set of solutions to see what direction we want to go with regarding incorporating existing audio projects. The chart above will continue to be populated as I look at more projects.

[Drigax]

I've written a blog post here, outlining the plan moving forward for this feature:
https://babylonjs.medium.com/audio-in-babylon-native-charting-the-way-c6bc6d5c0d74

Put shortly, it seems best to start with the most capable single solution and explore its feasibility before looking at combinations+custom solutions.

Starting with existing implementations in browsers, Chromium and Firefox's implementation of WebAudio come with too much additional work to get the codebases in a shape where integration would be trivial, as well as questionable build processes.

I am now investigating the feasibility of LabSound, another WebAudio implementation that appears to be "standalone" in its build process and consumption. A large plus is that the project was created to use CMake as its build system, which suggest that it should not be too difficult to integrate into BabylonNative's build system

[Drigax]

Update 2021-05-17

The project and samples were successfully built on Windows, it may be worth taking the next step and getting the project to build as a part of BabylonNative's build.

I'll have to look into how to isolate building the engine+dependencies as a consumable lib without building the additional samples

[Drigax]

Update 2021-06-07

My work is currently staged on:
https://github.com/Drigax/BabylonNative/tree/add-labsound (BabylonNative)
https://github.com/Drigax/Babylon.js/tree/add-native-audio (Babylon.js)

The native changes currently implement a Plugin component for NativeAudio, and modifies the Win32 playground to initialize NativeAudio. Currently the NativeAudio plugin intializes N-Api wrappers for the WebAudio classes needed to support Babylon.js's Sound and AudioEngine classes.

In Babylon.js some changes were made to engine instantiation so that the NativeEngine can have its AudioEngine initialized using the Engine constructor, also the AudioEngine was copied and modified to be usable by the NativeEngine (removing all dom-related functionality) as the NativeAudioEngine implementation.

[Drigax]

Moving forward there are some outstanding work to be done until I believe this work is ready for merging:

• Remove debug logging from NativeAudio.cpp
• Ideally, refactor existing logging to use a preprocessor macro/AppRuntime level cross-platform debug logging component.
• Move NativeAudio.cpp class/member declarations to a header, and refactor method implementations in respect of these changes.
• Implement some code reuse regarding AudioNode derived object wrappers. Currently, N-API doesn't implement javascript inheritance for ObjectWrappers. Also, directly inheriting from the ObjectWrapper-derived base class in C++ seems to be a dead end. I was able to work around this through explicitly implementing the AudioNode contract in GainNode (with the intention of using this for the other node implementations, however this may cause some unexpected side effects as we implement more of the contract....
• Begin mapping N-API object wrappers to LabSound objects. So far it seems that we can 1:1 map an object wrapper to LabSounds equivalent, initializing the LabSound object on ObjectWrapper constructor, and having the ObjectWrapper manage the native object's lifecycle.
• Finish stubbing the rest of the WebAudio contract. While not needed to fulfill this task, it would lay the groundwork for expanding future audio functionality easily (and preventing unexpected exceptions when trying to use WebAudio functionality outside of the Babylon.js contract)
• Modify UWP build to use miniaudio backend for LabSound instead of RtAudio.
• Refactor Babylon.js Sound components (Sound, Analyser) to only use the canvas, if possible.
• We want to support offloading of audio processing on low-performance systems. for windows-based platforms we want to figure out how to modify LabSound's audio graph (and mini-audio's rendering) to offload spatial audio processing via ISpatialAudioClient on Windows platforms

[ryantrem]

Love the idea of having community help getting this implemented! @bghgary thoughts on this? Could @matthargett pick up where Nick left off?