sm-audiokit's People
sm-audiokit's Issues
Constant CPU burn from AURemoteIO::IOThread
Here is what Strum Machine's CPU usage is like during playback; I happened to catch it at 0% although it's usually between 1% and 5%. "Other Processes" jumps around a lot, but it doesn't seem to be much higher during playback, if at all.
Memory usage is also very reasonable:
I wouldn't be surprised if these numbers are inaccurate; perhaps Cordova's browser execution time isn't being captured by the profiler. Still, I can crank up the BPM to crazy amounts and there's no impact on the responsiveness of the app.
On the other hand, here's what I get in our XCode project with 80 AudioPlayers:
All the CPU usage is basically coming from AURemoteIO::IOThread
, which is steady whether players are going or not. Memory is at 175 MB, with no players buffered.
If I bring the player count down to 8, the CPU usage hovers around 10%, with memory at 39 MB.
In the Cordova testbed project, with 80 AudioPlayers, CPU usage seems to be around 30%, rising up to 40+%, with memory rising up to 150 MB:
Memory and CPU use does not decrease when playback is stopped.
I tried calling .bypass()
on nodes that weren't in use; that made some difference, but not a ton, and it led to random errors.
Since resource usage does not go down when playback is stopped, I don't think this has anything to do with loading/decoding audio data.
With #6 fixed, the situation is improved, so it may not be the end of the world if this isn't improved... but it seems like quite a waste to be using so much CPU (and battery) when not actually doing that much...
Do any ideas for reducing resource usage come to mind?
High CPU use with mono audio files
I noticed that if I put mono MP3 files in the audio-package, the CPU bogs down (to the point of unresponsiveness) with lots of the following error in the console:
AudioPlayer+Buffering.swift:updateBuffer(force:):68:Copying mono data to 2 channel buffer... <AVAudioFormat 0x6000022354f0: 1 ch, 44100 Hz, Float32> (AudioPlayer+Buffering.swift:updateBuffer(force:):68)
This happens every time a file is loaded scheduled for playback, even if it's already loaded and buffered! AudioPlayer+Scheduling.swift:16
calls AudioPlayer.updateBuffer()
, I guess to reload the file from disk in case it's changed? I checked, and indeed the buffer is available just before the call to AudioPlayer.play()
...
This is a bummer because my samples are mono, and switching to stereo MP3 files means a big increase in file sizes โ not quite 2x but still painful. Can you think of any way around this?
Engine management/lifecycle and so forth
All that annoying production-ready stuff we need to handle:
- Changes in speakers/headphones/Bluetooth connections
- Sample rate changes?
- Playing nice with other apps / being backgrounded
- I don't play audio in the background, because currently WebViews aren't able to run in the background
- Passing errors back through Cordova to WebCode
You mentioned making certain calls in AppDelegate
. Keep in mind that we don't have access to that file from our Cordova plugin. See below.
Cordova does have a few events it sends to plugins, which we might want to listen and respond to (stubs in CordovaAudioKitPlugin.swift):
- onAppTerminate
- onMemoryWarning
- onReset - called when WebView refreshes - we should reset/destroy everything here, as we don't know if it'll be wanted after refreshing
About that last point. I'm planning to expose a setting in the app to choose an audio backend: AudioKit or Web Audio API (obviously with more user-friendly names). AudioKit will be opt-in at first, until I gather enough data to support making it opt-out. When the setting is changed, the WebView will be refreshed and the web page reloaded with whatever their preference is. Therefore, we don't want to instantiate any AudioKit objects until after the plugin calls initialize
, and we want to be able to shut all the AudioKit-related stuff down if they refresh the page with Web Audio selected.
More problems with completionHandler
See #5 for context. I'm having a different issue now, although it's also a mutex deadlock situation. But this time it's one hung call to AudioPlayer.stop()
, and a bunch of calls hung on isInManualRenderingMode
:
(Thread 2 is expanded but they're all identical.)
I've only had this happen when running in the testbed Cordova app, with the BPM cranked way up, and letting it run for a minute. Still, unacceptable for production...
Deadlock in __psynch_mutexwait on calling player.stop()
Important preface: I went back to using a pool of AudioPlayer objects that are all instantiated before the engine is started; that fixed random engine-related errors I was getting, and also eliminated the overhead of creating new AudioPlayer objects all the time. As a bonus, I can now easily reuse buffered players for repeated samples! In master
, each Channel has its own pool of AudioPlayer/SamplePlayer objects; I am experimenting with using a shared pool of AudioPlayers (in the dev/unified-pool
branch) but that has no bearing on this issue...
If you hit "Play Drum Loop" and wait awhile (10-30 seconds usually), eventually it will stop responding. If then I pause execution, it's always inside __psynch_mutexwait
:
libsystem_kernel.dylib`__psynch_mutexwait:
0x1bd7bea3c <+0>: mov x16, #0x12d
0x1bd7bea40 <+4>: svc #0x80
-> 0x1bd7bea44 <+8>: b.lo 0x1bd7bea60 ; <+36>
0x1bd7bea48 <+12>: stp x29, x30, [sp, #-0x10]!
0x1bd7bea4c <+16>: mov x29, sp
0x1bd7bea50 <+20>: bl 0x1bd7bc9f4 ; cerror_nocancel
0x1bd7bea54 <+24>: mov sp, x29
0x1bd7bea58 <+28>: ldp x29, x30, [sp], #0x10
0x1bd7bea5c <+32>: ret
0x1bd7bea60 <+36>: ret
This is originating from [AVAudioPlayerNode stop]
, which is being called by AudioPlayer.stop()
โ sometimes because it's called directly, sometimes as part of a call to .load()
or .play()
. It seems like there's a small chance of that function deadlocking anytime it's called... which is obviously far from ideal!
Happens in both iOS 15 and iOS 14.4 simulators as well as my real iPhone.
Offloading CPU from the main thread
I got the occasional EXC_BAD_ACCESS
when I was using DispatchQueue.global(qos: .utility).async
to interact with AudioManager
in the Cordova plugin/bridge. So I changed them all to use DQ.main.async
, and no errors.
The question remains: is there a way to offload stuff from the main thread? Should I even worry about it?
When I don't use any DQ.main.async
calls at all, I get warnings from Cordova saying the calls are taking 10-20ms to return, which will affect the responsiveness/smoothness of the app. DQ.main.async
gets rid of the warnings, but I think it's still taking up time on that same thread, right?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.