mozilla / mozstumbler

Android Stumbler for Mozilla

MozStumbler

Note: Mozilla Stumbler was retired on February 8, 2021. This code works on Android 9, but not Android 10 or later.

Please refer to the wiki for detailed documentation.

Building a debug version from command line

The build system is smart enough to automatically download and install all the parts of the Android SDK for you. If you cannot build, you can either try to fix your Android dev enviroment to fit the android/build.gradle requirements - or you can simply remove ANDROID_HOME, and all traces of your Android SDK from your PATH.

make

Building a debug version from Android Studio

Setup the Android Application to use two gradle aware make targets. You must set 'Installation Options' to "Deploy Nothing".

The tricky part is to set the build tasks. You will need two tasks of type 'Gradle-Aware Make'. Android Studio will autocomplete the names below when you start typing them in.

1. :android:assembleGithubDebug
2. :android:installGithubDebug

mozstumbler's Issues

Use legalier text

after merging dougt/notice_dialog, we have a notice dialog. We should update the text to sound more legally.

Beta 2

Avoid sending "empty" locations

I've seen a couple dozen validation errors on the server, with this message:

error_handler[{'location': 'body', 'name': 'body', 'description': 'You need to provide a mapping with least one cell or wifi entry.'}]

This indicates a missing client side check. We should make sure we only include batch items, which have either cell or wifi entries or both. Probably some simple length check in the right place.

Display map of scanned APs

Depends on AP database issue #3.

It would be cool if the app displayed a (real-time?) map of APs the user has scanned.

Display notification icon when ScannerService is running in the background

Touching the notification should reopen MozStumbler so the user can see their scanning stats or stop scanning.

https://developer.android.com/guide/topics/ui/notifiers/notifications.html

Stumbler crash when clicking the scan button

stacktrace:

V/LocationManagerService(   93): Location listener died                                                                                     [1189/1617]
V/LocationManagerService(   93): _removeUpdates: listener = Receiver{46303ee0 Listener android.os.BinderPr
oxy@46499a60}
D/WifiService(   93): releaseWifiLockLocked: WifiLock{MozStumbler type=2 binder=android.os.BinderProxy@465
25160}
D/WifiService(   93): enable and start wifi due to updateWifiState
E/AndroidRuntime(  964): FATAL EXCEPTION: main
E/AndroidRuntime(  964): java.lang.IllegalStateException: Could not execute method of the activity
E/AndroidRuntime(  964):        at android.view.View$1.onClick(View.java:2072)
E/AndroidRuntime(  964):        at android.view.View.performClick(View.java:2408)
E/AndroidRuntime(  964):        at android.view.View$PerformClick.run(View.java:8817)
E/AndroidRuntime(  964):        at android.os.Handler.handleCallback(Handler.java:587)
E/AndroidRuntime(  964):        at android.os.Handler.dispatchMessage(Handler.java:92)
E/AndroidRuntime(  964):        at android.os.Looper.loop(Looper.java:144)
E/AndroidRuntime(  964):        at android.app.ActivityThread.main(ActivityThread.java:4937)
E/AndroidRuntime(  964):        at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime(  964):        at java.lang.reflect.Method.invoke(Method.java:521)
E/AndroidRuntime(  964):        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.j
ava:868)
E/AndroidRuntime(  964):        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:626)
E/AndroidRuntime(  964):        at dalvik.system.NativeStart.main(Native Method)
E/AndroidRuntime(  964): Caused by: java.lang.reflect.InvocationTargetException
E/AndroidRuntime(  964):        at org.mozilla.mozstumbler.MainActivity.onClick_ToggleScanning(MainActivit
y.java:204)
E/AndroidRuntime(  964):        at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime(  964):        at java.lang.reflect.Method.invoke(Method.java:521)
E/AndroidRuntime(  964):        at android.view.View$1.onClick(View.java:2067)
E/AndroidRuntime(  964):        ... 11 more
E/AndroidRuntime(  964): Caused by: java.lang.NullPointerException
E/AndroidRuntime(  964):        ... 15 more
W/ActivityManager(   93):   Force finishing activity org.mozilla.mozstumbler/.MainActivity
W/ActivityManager(   93): Activity pause timeout for HistoryRecord{46221930 org.mozilla.mozstumbler/.MainA
ctivity}
I/WindowManager(   93): WIN DEATH: Window{4644e6e0 org.mozilla.mozstumbler/org.mozilla.mozstumbler.MainAct
ivity paused=false}
I/ActivityManager(   93): Process org.mozilla.mozstumbler (pid 964) has died.
I/WindowManager(   93): WIN DEATH: Window{46322fa0 org.mozilla.mozstumbler/org.mozilla.mozstumbler.MainAct
ivity paused=false}
I/UsageStats(   93): Unexpected resume of com.htc.launcher while already resumed in org.mozilla.mozstumble

Stumbler crash because android.telephony.gsm.GsmCellLocation.getPsc doesn't exist

stack trace (from adb logcat):

E/AndroidRuntime(  973): FATAL EXCEPTION: Thread-9
E/AndroidRuntime(  973): java.lang.NoSuchMethodError: android.telephony.gsm.GsmCellLocation.getPsc
E/AndroidRuntime(  973):        at org.mozilla.mozstumbler.Scanner.getCellInfo(Scanner.java:264)
E/AndroidRuntime(  973):        at org.mozilla.mozstumbler.Scanner.onLocationChanged(Scanner.java:227)
E/AndroidRuntime(  973):        at android.location.LocationManager$ListenerTransport._handleMessage(Locat
ionManager.java:191)
E/AndroidRuntime(  973):        at android.location.LocationManager$ListenerTransport.access$000(LocationM
anager.java:124)
E/AndroidRuntime(  973):        at android.location.LocationManager$ListenerTransport$1.handleMessage(Loca
tionManager.java:140)
E/AndroidRuntime(  973):        at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime(  973):        at android.os.Looper.loop(Looper.java:144)
E/AndroidRuntime(  973):        at org.mozilla.mozstumbler.ScannerService$LooperThread.run(ScannerService.
java:110)

Investigate undocumented WifiManager.startScanActive() API

This issue is forked from issue #38.

WifiManager.startScan() is passive: it only reports APs that happen to be broadcasting beacons. There is a hidden startScanActive() API that can initiate an active scan that probes APs to respond:

http://www.androidjavadoc.com/2.3/android/net/wifi/WifiManager.html#startScanActive%28%29

No LICENSE

Add support for CdmaCellLocation

At https://github.com/dougt/MozStumbler/blob/master/src/org/mozilla/mozstumbler/Scanner.java#L251 we can get back either a GSMCellLocation or CdmaCellLocation object (http://developer.android.com/reference/android/telephony/cdma/CdmaCellLocation.html).

On line 256 we are dealing with the Gsm case, but not with Cdma. The protocol defines how to send Cdma data at https://mozilla-ichnaea.readthedocs.org/en/latest/cell.html#cdma

Pseudo-code for the cdma case:

if (cl instanceof CdmaCellLocation) {
    JSONObject obj = new JSONObject();
    CdmaCellLocation ccl = (CdmaCellLocation) cl;

    obj.put("lac", ccl.getNetworkId());
    obj.put("cid", ccl.getBaseStationId());
    obj.put("asu", mSignalStrength);

    switch (tm.getNetworkType()) {
        case TelephonyManager.NETWORK_TYPE_CDMA:
        case TelephonyManager.NETWORK_TYPE_EVDO_0:
        case TelephonyManager.NETWORK_TYPE_EVDO_A:
        case TelephonyManager.NETWORK_TYPE_EVDO_B:
            obj.put("radio", "cdma");
            break;
        default:
            Log.w(LOGTAG, "", new IllegalStateException("Unexpected NetworkType: " + tm.getNetworkType()));
            break;
    }
    String mcc_mnc = tm.getNetworkOperator();
    if (mcc_mnc.length() > 3) {
        mcc = mcc_mnc.substring(0, 3);
        mnc = mcc_mnc.substring(3);
        obj.put("mcc", mcc);
        obj.put("mnc", mnc);
    }
    cellInfo.put(obj);
}

in addition we need to change the code setting mSignalStrength on line 201:

public void onSignalStrengthsChanged(SignalStrength ss) {
    if (ss.isGsm()) {
        mSignalStrength = ss.getGsmSignalStrength();
    } else {
        mSignalStrength = ss.getCdmaDbm();
        // how to detect evdo?
        // mSignalStrength = ss.getEvdoDbm()
    }
}

Add on-screen counters

Related to issue 3.

When you measure something, you get more of it. Add a counter to our stumbler and geeks will naturally try to get the high score. This would be the first step towards gamification of stumbling. :)

Stumbler does not stop scanning after opening "Locate Me" map view

Add buttons that open leaderboard and map web page

We could implement a quick and dirty client leaderboard by just having a button that opens the leaderboard web page in the user's default browser.

• Leaderboard URL: https://location.services.mozilla.com/stats

We could add a button that opens the measurement map too:

• Map URL: https://location.services.mozilla.com/map

Acquire WakeLock and WifiLock to prevent sleeping when stumbling

Do we need to acquire a WakeLock and a WifiLock to avoid sleeping when stumbling?

https://developer.android.com/reference/android/os/PowerManager.html

https://developer.android.com/reference/android/net/wifi/WifiManager.WifiLock.html

Keep local database of scanned APs so users can export their data

With a local database, we can add enable users to export the data they've scanned. We also need a local database so the stumbler can show the user a map of their scanning history.

We might consider filtering our reports for already-reported APs unless:

• the latest scan has a stronger signal strength (i.e. has a more accurate location)
• the AP has moved since the most recent scan
• the most recent scan is older than X days (to remind the server that the AP is still exists)

Add "Only upload measurements over Wi-Fi" option

Stumbler should default to scanning when launched

This would streamline the user experience by removing an extra step (pressing the "Start Scanning" button). The WiGLE stumbler does this, too.

App icon

We need a fun Mozilla app icon for MozStumbler. :)

Generate random user token for database submission stats

Client generates random user token to be included in submissions. There is no opt-out yet; the user implicitly opts-in by running the stumbler.

Batch upload measurements

Depends on AP database issue #3.

We should not upload measurements so frequently. We should save the scans to a local DB and batch upload them after X unique new APs (1000?) or Y hours (24?), whichever comes first.

When batch uploading, we should preserve the groups of APs seen at the same time (to aid server gridding), not just a big flat list of APs.

The submit API docs:
https://mozilla-ichnaea.readthedocs.org/en/latest/#service-1

Display number of APs and locations reported and time of most recent upload

Also remove GPS fixes from UI. The different between fixes and locations is a common question on IRC.

Displaying the time of the most report upload will also answer questions about "is my stumbler reporting data?" we get on IRC, too.

Display user score

Client should display the user's score on screen. This would require a server API to request score for a given user token or the server could include the user's score in the submission response.

Directly point to new leaderboard page

I separated the leaderboard from the global stats and gave it its own page at /leaders. The app should probably point at that (https://location.services.mozilla.com/leaders) instead of /stats

Separating the two gives us more screen real estate to display other kinds of data.

Possible data skew?

So, what I understand that we are doing is:

1. detect a location change
2. when a location change is seen, kick off a wifi scan
3. when we get the list is returned by the scan, bundle and send to server.

If I am traveling fast, the space time between 1 and 2 might be significant. That is, I might detect a location change at point A and do a wifi/cell scan at point B. If the distance between point A and point B is significant, using this data for location determination might be troublesome.

I'd like to hear solutions to this -- or some verification that it isn't a problem.

Scanning toast messages are blank

Toast messages are blank and I see the following exception stack trace in logcat because R.string.start_scanning and stop_scanning are integers, not strings.

src/org/mozilla/mozstumbler/ScannerService.java
133:                        i.putExtra(Intent.EXTRA_TEXT, R.string.start_scanning);
158:                        i.putExtra(Intent.EXTRA_TEXT, R.string.stop_scanning);

W/Bundle  (31242): Key android.intent.extra.TEXT expected String but value was a java.lang.Integer.  The default value <null> was returned.
W/Bundle  (31242): Attempt to cast generated internal exception:
W/Bundle  (31242): java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
W/Bundle  (31242):  at android.os.Bundle.getString(Bundle.java:1069)
W/Bundle  (31242):  at android.content.Intent.getStringExtra(Intent.java:4350)
W/Bundle  (31242):  at org.mozilla.mozstumbler.c.onReceive(Unknown Source)
W/Bundle  (31242):  at android.app.LoadedApk$ReceiverDispatcher$Args.run(LoadedApk.java:758)
W/Bundle  (31242):  at android.os.Handler.handleCallback(Handler.java:725)
W/Bundle  (31242):  at android.os.Handler.dispatchMessage(Handler.java:92)
W/Bundle  (31242):  at android.os.Looper.loop(Looper.java:137)
W/Bundle  (31242):  at android.app.ActivityThread.main(ActivityThread.java:5227)
W/Bundle  (31242):  at java.lang.reflect.Method.invokeNative(Native Method)
W/Bundle  (31242):  at java.lang.reflect.Method.invoke(Method.java:511)
W/Bundle  (31242):  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:795)
W/Bundle  (31242):  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:562)
W/Bundle  (31242):  at dalvik.system.NativeStart.main(Native Method)
D/org.mozilla.mozstumbler.MainActivity(31242): Received a notification intent and showing: null

Copy uncaught exception stack traces to the Android clipboard

So users can email the stack traces to us. This is some low-tech crash reporting.

ScannerService breaks Froyo and Gingerbread compatibility by using Notification.Builder

From https://github.com/dougt/MozStumbler/pull/26#commitcomment-3689527

Notification.Builder was added in Android API Level 11 (Honeycomb). We should maintain Gingerbread support because it's still 34% of devices. I don't care too much about Froyo because it's only 3%, but it's a nice-to-have if it's not much work.

We will need to configure Notifications manually like Fennec does here: https://hg.mozilla.org/mozilla-central/file/bda9723bdccc/mobile/android/base/UpdateService.java#l367

Uploads fail on some pre-ICS devices because some CA root certificates are missing

Forced uploads (those initiated on Stop Stumbling & Upload button-press) doesn't always seem to work and should be investigated further.

Starting Stumbling first time after app launch and then stopping for the first time (with no uploaded data yet) works as it should, and stumbling data is uploaded just fine, but restarting stumbling and trying to stop & upload then (at least, sometimes) fails to upload anything.

It would seem as a good idea to implement some kind of "debug" log in the app, so dogfooders/beta testers would see how much of data were gathered & uploaded in the current session.

Also, some kind of persistent storage/caching would be great for not-yet-uploaded stumbling data, so even if upload was unsuccessful or simply did not take place, users wouldn't lose the gathered data if they put the app in background and was garbage-collected/killed by the system (this would also contribute to fixing #102, too).

StrictMode$InstanceCountViolation when device is rotated

Every time a device is rotated, the screen configuration changes and the main activity is destroyed and recreated. A side effect is that rotating MozStumbler logs these StrictMode error messages:

E/StrictMode(2914): class org.mozilla.mozstumbler.MainActivity; instances=2; limit=1
E/StrictMode(2914): android.os.StrictMode$InstanceCountViolation: class org.mozilla.mozstumbler.MainActivity; instances=2; limit=1
E/StrictMode(2914):     at android.os.StrictMode.setClassInstanceLimit(StrictMode.java:1)

Remove "token" from JSON report

Now that @hannosch has implemented server support for the X-Token and X-Nickname headers (from issue #36), we should remove the token property from the JSON reports. To ensure a seamless transition for older clients, we should probably wait a week before removing the property.

Wait for SCAN_RESULTS_AVAILABLE_ACTION intent before calling WifiManager.getScanResults()

This issue is forked from issue #38.

WifiManager.startScan() is asynchronous, but we call getScanResults() immediately, so our scan results are probably old. We should listen for the SCAN_RESULTS_AVAILABLE_ACTION intent and then call getScanResults().

Pause scanning on ACTION_BATTERY_LOW intent

We should politely stop scanning when we receive an ACTION_BATTERY_LOW intent or ComponentCallbacks.onLowMemory() callback. We will definitely need to implement this if we add the always-on background mode described in issue #21.

Should we resume scanning if we later receive an ACTION_BATTERY_OKAY intent? Or wait for the user to manually resume scanning?

Allow user to enter a nickname

Client should allow a user to enter a nickname. Client should only include the user token in submissions iff nickname is defined. This makes the leaderboard tracking opt-in because the user must manually enter a nickname to be listed on the leaderboard.

Reporter loses measurement reports if upload fails with a network error

We should probably retry later, but we also don't want to keep retrying the same report if the data itself is some causing the network problem. @hannosch says some web servers, like nginx, limit uploads to 1 MB by default.

Add stumbler version to the server requests

In #87 @cpeterso mentioned that we might be seeing a bug from older clients. It would be good if we'd actually be able to tell that for debugging purposes.

Currently the server sees a general "Android" user-agent string, for example:

"POST /v1/submit HTTP/1.1" 204 0 "-" "Dalvik/1.6.0 (Linux; U; Android 4.0.4; ZTE V790 Build/IMM76D)"
"POST /v1/submit HTTP/1.1" 204 0 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1.2; GT-I8190 Build/JZO54K)"

Maybe we could add a MozStumbler 0.2.1 token to the user-agent.

Investigate LocationManager.sendExtraCommand("gps", "delete_aiding_data") for GPS cold start

Apparently, we can use LocationManager.sendExtraCommand("gps", "delete_aiding_data") to delete cached A-GPS data to get a truer GPS fix.

Sources:

• http://developer.samsung.com/forum/board/thread/view.do?boardName=GeneralB&messageId=137014
• "Professional Android Sensor Programming"
• http://stackoverflow.com/a/13165414/61352

Investigate "Wi-Fi scan-only mode" (REQUEST_SCAN_ALWAYS_AVAILABLE) in JB 4.3

REQUEST_SCAN_ALWAYS_AVAILABLE is a new API exposed in JB 4.3:

Wi-Fi scan-only mode is a new platform optimization that lets users keep Wi-Fi scan on without connecting to a Wi-Fi network, to improve location accuracy while conserving battery. Apps that depend on Wi-Fi for location services can now ask users to enable scan-only mode from Wi-Fi advanced settings. Wi-Fi scan-only mode is not dependent on device hardware and is available as part of the Android 4.3 platform.

https://developer.android.com/reference/android/net/wifi/WifiManager.html#ACTION_REQUEST_SCAN_ALWAYS_AVAILABLE

Display map view showing both the GPS location and our server's predicted location

This map will help us validate our server search algorithms and see if we are getting useful data.

This bug is related to but different than issue #22.

API level 11 required?

Is Context.MODE_MULTI_PROCESS only in 11?

Scanning MozStumbler: ..........
src/org/mozilla/mozstumbler/Prefs.java:128: Warning: Field requires API level 11 (current min is 8): android.content.Context#MODE_MULTI_PROCESS [InlinedApi]
return mContext.getSharedPreferences(PREFS_FILE, Context.MODE_MULTI_PROCESS | Context.MODE_PRIVATE);

Filter known mobile device OUIs in MAC addresses

Wireshark maintains a list of known OUIs for hardware manufacturers: https://anonsvn.wireshark.org/wireshark/trunk/manuf

Implement Android ScannerService independent of MainActivity lifetime

Related to issue #7: every time a device is rotated, the screen configuration changes and the main activity is destroyed and recreated. A side effect is that the recreated MozStumbler activity does not know it had been scanning.

The scanner should probably be moved to an Android ScannerService that outlives MainActivity.

Auto-start scanning server when Android boots

after a reboot, we probably want to start up again.

Can we *please* use spaces instead of tabs?

:)

Changing indentation will break all blame history, but the project is still young. I can rebase my pending pull requests if you want to fix the whitespace first.

MainActivity.onStop() closes ScannerService mConnection, but onStart() does not reinitialize it

From https://github.com/dougt/MozStumbler/pull/26#commitcomment-3689366

Android calls onStop() when the activity is no longer visible. If the user navigates back to the activity, Android will call onRestart() and then onStart(). mConnection was initialized at its declaration. How will mConnection get reinitialized after onStop() sets it to null? We should probably initialize mConnection in onStart().

Enable both active and passive stumbling modes for the Client

Client should have separate "turbo" and "background" modes to reduce battery drain.

The background mode could limit its scans to a longer interval, a larger location delta, only when another app has woken up the CPU or turned the screen on, or use Android's passive location provider (that only receives location updates when another app geolocates the device).

https://developer.android.com/reference/android/location/LocationManager.html#PASSIVE_PROVIDER

Legal Notice

We probably want to remind the person running the stumbler that data that they submit is Mozilla's.

Display user token

Depends on user token issue #16.

Client should display user token on screen so users can find their score on the leaderboard.

Optionally use the new getAllCellInfo API

Android 4.2 (API level 17) introduced a new API for getting cell information with TelephonyManager.getAllCellInfo() (http://developer.android.com/reference/android/telephony/TelephonyManager.html#getAllCellInfo%28%29).

The new API offers better and more data and supports LTE / WCDMA networks. We have stumbler users on older Android versions, so we shouldn't require the new API, but fall back on the current code.

nickname not saved unless you hit "Done"

tl;dr: You have to hit "Done" to actually save the nickname edits, which is non-obvious, since you can hide the keyboard and un-focus the field, and the edits look like they persist, until you quit & re-launch the app.

STR:

1. Open app
2. Tap nickname field & modify the content. (do NOT hit "Done")
3. Hit the "Hide Keyboard" downarrow button (at bottom-left of keyboard on my Nexus4)
4. Optional: Start scanning, stop scanning, whatever.
5. quit app by triggering app switcher & swipe app offscreen
6. Re-launch app

EXPECTED RESULTS: My nickname should look the way it was when I quit the app.
ACTUAL RESULTS: My nickname edits weren't saved.

Modifying nickname in UI does not update nickname in reporter service

This problem is related to sharing SharedPreferences from different processes.