strummachine / sm-audiokit Goto Github PK

Here is what Strum Machine's CPU usage is like during playback; I happened to catch it at 0% although it's usually between 1% and 5%. "Other Processes" jumps around a lot, but it doesn't seem to be much higher during playback, if at all.

Memory usage is also very reasonable:

I wouldn't be surprised if these numbers are inaccurate; perhaps Cordova's browser execution time isn't being captured by the profiler. Still, I can crank up the BPM to crazy amounts and there's no impact on the responsiveness of the app.

On the other hand, here's what I get in our XCode project with 80 AudioPlayers:

All the CPU usage is basically coming from AURemoteIO::IOThread, which is steady whether players are going or not. Memory is at 175 MB, with no players buffered.

If I bring the player count down to 8, the CPU usage hovers around 10%, with memory at 39 MB.

In the Cordova testbed project, with 80 AudioPlayers, CPU usage seems to be around 30%, rising up to 40+%, with memory rising up to 150 MB:

Memory and CPU use does not decrease when playback is stopped.

I tried calling .bypass() on nodes that weren't in use; that made some difference, but not a ton, and it led to random errors.

Since resource usage does not go down when playback is stopped, I don't think this has anything to do with loading/decoding audio data.

With #6 fixed, the situation is improved, so it may not be the end of the world if this isn't improved... but it seems like quite a waste to be using so much CPU (and battery) when not actually doing that much...

Do any ideas for reducing resource usage come to mind?

I noticed that if I put mono MP3 files in the audio-package, the CPU bogs down (to the point of unresponsiveness) with lots of the following error in the console:

AudioPlayer+Buffering.swift:updateBuffer(force:):68:Copying mono data to 2 channel buffer... <AVAudioFormat 0x6000022354f0:  1 ch,  44100 Hz, Float32> (AudioPlayer+Buffering.swift:updateBuffer(force:):68)

This happens every time a file is loaded scheduled for playback, even if it's already loaded and buffered! AudioPlayer+Scheduling.swift:16 calls AudioPlayer.updateBuffer(), I guess to reload the file from disk in case it's changed? I checked, and indeed the buffer is available just before the call to AudioPlayer.play()...

This is a bummer because my samples are mono, and switching to stereo MP3 files means a big increase in file sizes – not quite 2x but still painful. Can you think of any way around this?

All that annoying production-ready stuff we need to handle:

• Changes in speakers/headphones/Bluetooth connections
• Sample rate changes?
• Playing nice with other apps / being backgrounded
  • I don't play audio in the background, because currently WebViews aren't able to run in the background
• Passing errors back through Cordova to WebCode

You mentioned making certain calls in AppDelegate. Keep in mind that we don't have access to that file from our Cordova plugin. See below.

Cordova does have a few events it sends to plugins, which we might want to listen and respond to (stubs in CordovaAudioKitPlugin.swift):

• onAppTerminate
• onMemoryWarning
• onReset - called when WebView refreshes - we should reset/destroy everything here, as we don't know if it'll be wanted after refreshing

About that last point. I'm planning to expose a setting in the app to choose an audio backend: AudioKit or Web Audio API (obviously with more user-friendly names). AudioKit will be opt-in at first, until I gather enough data to support making it opt-out. When the setting is changed, the WebView will be refreshed and the web page reloaded with whatever their preference is. Therefore, we don't want to instantiate any AudioKit objects until after the plugin calls initialize, and we want to be able to shut all the AudioKit-related stuff down if they refresh the page with Web Audio selected.

See #5 for context. I'm having a different issue now, although it's also a mutex deadlock situation. But this time it's one hung call to AudioPlayer.stop(), and a bunch of calls hung on isInManualRenderingMode:

(Thread 2 is expanded but they're all identical.)

I've only had this happen when running in the testbed Cordova app, with the BPM cranked way up, and letting it run for a minute. Still, unacceptable for production...

Important preface: I went back to using a pool of AudioPlayer objects that are all instantiated before the engine is started; that fixed random engine-related errors I was getting, and also eliminated the overhead of creating new AudioPlayer objects all the time. As a bonus, I can now easily reuse buffered players for repeated samples! In master, each Channel has its own pool of AudioPlayer/SamplePlayer objects; I am experimenting with using a shared pool of AudioPlayers (in the dev/unified-pool branch) but that has no bearing on this issue...

If you hit "Play Drum Loop" and wait awhile (10-30 seconds usually), eventually it will stop responding. If then I pause execution, it's always inside __psynch_mutexwait:

libsystem_kernel.dylib`__psynch_mutexwait:
    0x1bd7bea3c <+0>:  mov    x16, #0x12d
    0x1bd7bea40 <+4>:  svc    #0x80
->  0x1bd7bea44 <+8>:  b.lo   0x1bd7bea60               ; <+36>
    0x1bd7bea48 <+12>: stp    x29, x30, [sp, #-0x10]!
    0x1bd7bea4c <+16>: mov    x29, sp
    0x1bd7bea50 <+20>: bl     0x1bd7bc9f4               ; cerror_nocancel
    0x1bd7bea54 <+24>: mov    sp, x29
    0x1bd7bea58 <+28>: ldp    x29, x30, [sp], #0x10
    0x1bd7bea5c <+32>: ret    
    0x1bd7bea60 <+36>: ret    

This is originating from [AVAudioPlayerNode stop], which is being called by AudioPlayer.stop() – sometimes because it's called directly, sometimes as part of a call to .load() or .play(). It seems like there's a small chance of that function deadlocking anytime it's called... which is obviously far from ideal!

Happens in both iOS 15 and iOS 14.4 simulators as well as my real iPhone.

I got the occasional EXC_BAD_ACCESS when I was using DispatchQueue.global(qos: .utility).async to interact with AudioManager in the Cordova plugin/bridge. So I changed them all to use DQ.main.async, and no errors.

The question remains: is there a way to offload stuff from the main thread? Should I even worry about it?

When I don't use any DQ.main.async calls at all, I get warnings from Cordova saying the calls are taking 10-20ms to return, which will affect the responsiveness/smoothness of the app. DQ.main.async gets rid of the warnings, but I think it's still taking up time on that same thread, right?