yctung / libacousticsensing

Lib acoustic sensing is a utility library easing the process to design apps based on acoustic information

LibAS

LibAcousticSensing (LibAS) is a cross-platform framework to facilitate the development of acoustic sensing applications. Some interesting demo video of LibAS can be found at here and here.

Notice

I will make a major update soon. This update is currently in the separate branch called mobisys. For people wanna know how to convert their project from the Remote mode to Standalone mode, please check the mobisys branch. Since there are some API updates that might be inconsistent to the current branch and we have some active developers are using the master branch, I will keep the master branch intact until the mobisys branch ready to be released.

Feature

LibAS is designed to let its users focus on the design of acoustic sensing algorithms without learning much the tedious platform-dependent programming details.

• Sensing algorithms can be executed on real devices like Android/iOS/Tizen/Linux devices.
• Its remote mode allows you to design and test sensing algorithms purely by Matlab.
• Its standalone mode can help transfer the Matlab sensing algorithm into cross-platform supported libraries.

What is Acoustic Sensing?

In short, acoustic sensing can turn your device (such as a smartphone) into a sonar-like system. In this way, your device can sense the environments by generating sensing sounds and analyzing the received sounds. Various interesting applications (like this and this) can be implemented and deployed with acoustic sensing.

Why We Need LibAS?

We notice there are various platform-dependent challenges that might prevent new developers from implementing their exciting acoustic sensing ideas. For example, it is non-trivial to learn a new programming language (like Java for Android and Obj-C for iOS), especially to deal with the real-time audio recording. Moreover, it is also challenging to prototype/validate your sensing algorithm directly by these system programming languages. So we design LibAS to hide most technical programming details from the developers. The following figure shows the difference of designing acoustic sensing algorithms with or without LibAS:

Remote Mode (Easy for New Users)

In LibAS's remote mode, you don't need to know anything about real-time audio recording/playing on devices. What you need to do is download one of our prebuilt DevApp to your device and this app will automatically stream whatever signal needed for acoustic sensing to a remote Matlab server. You can easily design your sensing algorithm in Matlab and control the connected device to sense. Please read the LibAS's Matlab README for knowing the API details and its usage.

Install

Prerequisites

• Matlab 2014b or later (see uicontrol compatibility for an earlier version)
• LibAS DevApp: Google Play Store, Apple App Store, Tizen Store or build it from the DevApp source code
• Network connections between your Matlab machine and your DevApp-installed device

Setup

Let's use our Utility/FreqRespAnalysis as an example. In this example, Matlab server is programmed to ask the connected device to play several frequency sweeps for analyzing the frequency response between the device speakers and microphones. To set up this Matlab server, let's navigate to the example directory in your Matlab command window:

>> cd Utility/FreqRespAnalysis/Matlab

And then call the Setup.m function

>> Setup

WARN: For the first using this app, you will see the following message after calling the Setup. Please type Y and then restart your Matlab. (It is necessary for Matlab to load our customized java classes):

>> [WARNING]: JavaClassPath is not added yet!
>> Add this path will restart the Matlab.
>> Do you want to do this now?
>> (Y/N): Y

You are all set and can start running your fancy sensing algorithms with LibAS :)

Usage

You now can use Matlab to control the connected device to sense by the following four steps as shown in the figure:

• (1) Execute Setup.m and FreqRespAnalysisMain.m
• (2) Edit your server's address and port in the DevApp and click the connect button (you should see the red texts showing --- ACTION_CONNECT --- in Matlab when the connection is successful)
• (3) Click the Start Sensing button in the Matlab GUI
• (4) Now you hear screaming sound being played and see the the frequency response of your device being plotted

You can also change the sound to play easily by editing the AudioSource class in the FreqRespAnalysisMain.m. Please refer to LibAS's Matlab README for knowing how the code works.

Troubleshooting

• If you can't build the connection in DevApp, try to ping your Matlab machine (e.g., adb shell ping 10.0.0.1) and ensure there is no firewall between your Matlab server and device
• If you keep seeing this Matlab error message 'SensingServer' not found you might need to manually call import edu.umich.cse.yctung.* before using any SensingServer class
• If you can't hear any sound being played (the sound should be audible for this FreqRespAnalysis example), please check if your app has the permission to play/record sounds or if your device is muted.
• If you see some errors related to preamble is not synced, which menas LibAS's current setting to find the start of sensing audio fails. Please use the other microphone/speaker to and test again. Feel free to file an issue if you can't solve it.

Standalone Mode (Build Your App for Product)

NOT COMPLETE YET

Troubleshooting

• If Android tell you Error:Execution failed for task ':libacousticsensing:compileReleaseNdk'. NDK not configured. you need to update your local.properties file to include your ndk path. For example, mine is: sdk.dir=/Users/yctung/Library/Android/ndk

Compatible devices

In theory, LibAS should be compatible with all Android, iOS, Tizen devices with microphone/speaker installed. However, it might need to customized settings for different devices. Followings include the devices we have used and tested in LibAS:

• Android: Samsung Galaxy S4/5/6/7/8, Note 4, Nexus 6P, Nexus 5X, HTC One, ASUS Zen Watch 3
• iOS: iPhone 5c, iPhone 6s
• Tizen: Samsung Gear S3

License

MIT License

libacousticsensing's Issues

JavaSensingServer import error in High Sierra

The problem is caused by the latency analyzer.

I might try to build latency analyzer as a separate class

quiz

How can I use LibAS to develop my own project？Does LibAS have app on Android or Windows?

Add import to every main script

Matlab 2017 or later (or due to the OS version) can't find the JavaSensingServer class

TODO: add import to every main script

List of previously used IPs

I often move the devices around to different networks. Every time I have to retype the IP address in the settings. It would be useful if the system somehow remembered a list of IPs we previously used.

When WiFi disconnects on the client, it loses connection with server

This is obvious and probably the correct behavior. Just something that seems a little user un-friendly.

My Galaxy S5 sometimes disconnects from Wifi. When it does, it loses connection with server (which makes sense). It would be nice if it detects that this happened and maybe show some error message or notification on the phone.

IP input keyboard missing "."

In Galaxy S5 running Android 4.4.2, the keyboard doesn't allow you to enter periods. Without this, we can't enter IP addresses. The only way around this now is to write it in a different app and copy/paste it into DevApp.

No figure or app has no reaction

If the ch1 doesn’t show the figure or the app doesn’t work, please check if your matlab machine and devices are in the same network. Then run the main code again and reset IP address on your phone. If it still doesn’t work, please change port in matlab main code.

Can not run app on your devices

When you open the code with Andriod Studio, there are two choices with wear and mobile app. Please choose the right option before running the code on your devices.

Pressing "Stop" on Android doesn't stop system

The Android app should be able to stop sensing on the server side too.

ACTION_CLOSE: socket is closed remotely

Is this a problem about mobile phone or matlab server?

Matlab error about stop sensing

If matlab tells you Error: unable to find valid pilot at chIdx=2 Error:unable to sync preamble, offsets=-1 Error: going to stop sensing because preamble is not detected, which means LibAS’s current setup fails. Please restart your matlab to run Setup again.

Windows needs "/" instead of "\" to add the static java path

Might need to also check if the bin and java version consistency (e.g., Matlab version -java)

Changing mics hangs the UI

To reproduce the problem:

1. Start server and app.
2. Connect app to server
3. Start sensing on the server.
4. Stop sensing on the server.
5. Click on settings on the app and change mic source
6. Repeat steps 3 and 4 (it works fine and does change the source)
7. Then, after stopping the sensing from the server, try to press on the settings icon again and it hangs and crashes. It gives the error "DevApp isn't responding"

It doesn't seem to matter which source we change it to. This problem happened in Google Pixel phone.

Closing Matlab before pressing "Stop sensing" causes a weird state

If we close the Matlab app by pressing the close button before we press "stop sensing", the phone keeps playing the sound and doesn't stop. If this happens, we should send the "stop sensing" command to the phone. We can do this by adding a callback to the close-figure function in Matlab (added in T-Watch branch).

Displaying sound on the phone

When using multiple devices as the client (e.g. T-Watch), it might be useful to show the spectrogram or time-series of the audio signal that is loaded inside each phone. That way, we can tell which one is playing which sound.

Very low priority though.

quiz2

In (/Example/ObjectDetector/Matlab/ObjectDetectorMain.m) ，how can I solve this problem?

QR-code based IP acquisition

We can use a tool to convert server IP into a QR code (e.g. qrencode) and display that on the Matlab code. The app can use camera to capture QR code and save the server IP in shared preferences. We don't need any Internet support to do this. It can be entirely on the laptop server and the mobile client.

If we can't find that, we can at least write a command line utility which people can use (in place of ifconfig) which shows LAN IP as QR code. For example:

qrencode -t ASCII -o - `ipconfig getifaddr en0`

What is the preamble file ? I can't find it in the mobisys branch.

I admire your advance arrangements very much. But currently encountering a problem. My dear compatriots.

The class of DemoForceMonitorActivity not commit in mobisys branch

Auto connecting keeps retrying even after connection

When running in master/slave mode, sometimes the auto connection keeps trying to reconnect even after getting the message "Server is connected, now wait server to send audio". This doesn't always happen, though.

Start "remote" sensing by the device will cause the matlab SensingServer errors

How to reproduce:

• Just click the "start sensing" button by the device

Reason

• obj.traceParser = TraceParser(obj.audioSource, obj.traceChannelCnt,obj);
• is called only in the startSensingSelf() funciton

How to solve

• either don't allow the mobile device to start sensing or init these parse setting in other places
• or we can send a event by the device to notify it is going to start sensing

Note

• This will relavent to the "trace save" function in the Matlab, don't change it without considering the users might lose their traces

Samsung S8 can't find the first peak

Reason: AGC is on even the VOICE_RECOGNITION flag is set. AutomaticGainControll.isAvaialble() also returns false, so there is no way to fix it.

Two devices mode might have confusion on the preamble

TODO: mute the slave and turn the volume back solve this problem

Stop/Start Sensing button on the server side stops working after 2 times

Sometimes I have to modify the callback code so I press "Stop Sensing" on the server, change the code and press "Start sensing" again. This works the first time and the second time. But, it doesn't play any sound the third time and just says that the preamble could not be detected.

It might have to do with how I change the callback code, or maybe some internal error state caused by Matlab. I will try to reproduce it.

Good UI to know phone system volume is 0 (muted)

Some users face the issue that the phone is not playing the sound. Turns out to be the issue that the phone is in the mute mode. Providing a better UI(warning?) in DevApp might help

"Real-time" status

Sometimes the callback function is too slow and ends up delaying the real-time system. When this happens, it's not clear if we are seeing the delayed callback or if we are up-to-date. It would be helpful if we had some way to tell this -- maybe just a number to say how far behind we are (which callback we are currently evaluating vs. how many chirps were made according to our configuration) or a simple visualization like on YouTube which shows the buffering status.