Coder Social home page Coder Social logo

sitcomlab / simport-learning-app Goto Github PK

View Code? Open in Web Editor NEW
6.0 5.0 6.0 8.13 MB

Learning tool on location data privacy, that reflects to users, what conclusions can be drawn from their location histories. This is part of the SIMPORT project.

Home Page: https://www.simport.net

License: MIT License

Java 0.31% JavaScript 0.47% TypeScript 86.35% HTML 8.68% Swift 0.88% Ruby 0.42% SCSS 2.75% R 0.14% Shell 0.01%
location-privacy ionic capacitor

simport-learning-app's Introduction

SIMPORT Learning App ·



Learning tool on location data privacy, that reflects to users, what conclusions can be drawn from their location histories. It is used to record location data over a timespan and analyzing that locally to create inferences - e.g. about where the user might work or live. The app works completely autonomously and doesn't need any internet connection whatsoever. This is part of the SIMPORT project.

Status: prototyping 🛠️

Contents

App Usage

It is intended that one records her/his location data of the course of multiple days/weeks/months using this app. By collecting and analyzing the recorded location history (locally), it is tried to infer information about the user - such as her/his home and work location. Users should be notified, if such inferences can be made in order to reflect to them, what conclusions can be drawn using their location histories. Furthermore recorded trajectories can be exported, external trajectories can be imported and examined using the app as well.

Development

This is an hybrid Ionic app, using Capacitator (a drop-in replacement for Cordova) to access native APIs and Angular for UI.

Setup

Quickstart (Tools)


For basic UI development you need node.js (version 16.14) installed - further information can be found here. As an IDE you can basically choose by your own taste. Our recommendation is to use Visual Studio Code with the code formatter Prettier using these settings. This guarantees consistent and uniform code.

Clone the repository
git clone [email protected]:sitcomlab/simport-learning-app
Within the repository, install the ionic CLI, which is used for most management tasks, as well as the frontend dependencies
cd simport-learning-app
npm install -g @ionic/cli # install the ionic CLI
npm install               # install the frontend dependencies

Android

  • Install the latest version of Android Studio
    • on Linux, install to /opt/android-studio/, as this path is configured in capacitor.config.json
  • Download an up-to-date Android SDK (e.g. SDK 29)
    • Within Android Studio: ToolsSDK Manager)
  • Optional: Setup emulators as needed for testing the app on a computer
    • Within Android Studio: ToolsAVD Manager)
  • Make sure to run Sync Project with Gradle Files before building when dependencies have changed
  • Built artifact will be ./android/app/build/outputs/apk/app-debug.apk

iOS

📝 This only works on macOS.

  • Install the latest version of XCode
  • Optional: Setup emulators as needed for testing the app on a computer
    • Within XCode: WindowDevices and SimulatorsSimulators)
  • For running on physical devices, XCode requires that you’ve connected a Team to the project → more information here
  • Make sure to run pod install in your terminal before building when dependencies have changed

📝 With Capacitor, the native build projects are supposed to be checked into version control. This avoids duplicate config and simplifies writing native code without creating plugins.

Build and Run

Hot Reloading Server (→ see here)
ionic serve
Build frontend to ./www/ (→ see here)
ionic build
Prepare app builds & open IDE (→ see here)
Android
ionic cap update
ionic cap build android
iOS
ionic cap update
ionic cap build ios
Sync app builds (→ see here)
Android
ionic cap sync android
iOS
ionic cap sync ios

Version Control

Using Github this project is obviously versioned using git. The branch develop is used as the main branch, while actual developing of new features is done on individual feature branches. These features branches are using the following naming scheme:

feature/<feature name>

When finished (including review) developing a feature, the feature branches are merged develop. For your commits, please use the following commit message if applicable:

#<issue number>: <commit message>

Releases are automatically triggered on push to release-beta using GitHub Actions and distributed to a closed group of beta-testers using Firebase.

Inferences

A core-component of this learning app is the inference-algorithm. By analyzing the recorded location history, it is tried to infer information about the user with this algorightm - information such as her/his home and work location. The corresponding procedure on how to process user-trajectories in order to generate those inferences is explained with a brief flow-diagram. Furthermore a light class-diagramm that focuses on the inference-components gives an overview how the parts are connected and come together.

Test

Apart from basic component testing (default Angular), the testing framework is used to validate the inference algorithms. By running the tests via npm run test a few simple trajectories are generated, that are designed to cover some border cases - e.g. variation in location frequency and accuracy. The creation of this test data is based on a few location files located here. The generated test data is based upon the following location pattern, which result from the assumptions of a typical work day (9 to 5).

Activity Start datetime End datetime
Dwell at home 2021-02-23T18:00:00Z 2021-02-24T08:45:00Z
Ride to work 2021-02-24T08:45:00Z 2021-02-24T09:00:00Z
Dwell at work 2021-02-24T09:00:00Z 2021-02-24T17:00:00Z
Ride home 2021-02-24T17:00:00Z 2021-02-24T17:15:00Z
Dwell at home 2021-02-24T17:15:00Z 2021-02-25T08:45:00Z

Currently the following test cases are created, analyzed and checked against the given expected inferences:

Trajectory Description Expected inferences
Empty Empty trajectory, contains no locations. None
Mobile only Trajectory, that only contains mobile data. Therefore it contains only the ride to and from work, but no stationary location for dwelling at home or work. None
Simple home and work Simple trajectory that simulates a usual day of work. The location data contains the assumed movement data stated above with typical point clouds at the dwelling locations. Without any special constraints. Home/Work
Spatially dense home and work Similar to simple home/work, but stationary data (point clouds for dwelling time) is way more spatially dense, which simulates a lot of movement/higher update frequency during recording. Home/Work
Temporally sparse home and work Similar to simple home/work, but stationary data (point clouds for dwelling time) is way more temporally sparse, which simulates a less movement/lower update frequency during recording. Home/Work

These tests are automatically executed when pushing to the development branch. The last test-result can be seen at the top of this page.

License

SIMPORT Learning App
Copyright (c) 2022 Sitcom Lab

Further information

simport-learning-app's People

Contributors

felixerdy avatar jbraese avatar juanffranco avatar noerw avatar schrooom avatar zven avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

simport-learning-app's Issues

Frequency of notifications to high

Notifications about received location update come in too often. The following screenshot is from a timespan of about 30 to 40 minutes. Maybe a hourly, daily or weekly summary-notification is sufficient (possibly even via user-setting?)

IMG_0904

Export / import trajectories

  • Enable access to filesystem
  • Export trajectories
    • update UI to enable users to export trajectories
    • iOS
    • Android
  • Import trajectories
    • iOS
    • Android
      - Evaluate imports to DB vs .json-files to filesystem (low priority for now)
  • Unit tests

Trajectories should encode time moving/stayed explicitly

As outlined in #8, we need to interpret time between points for inferences.
This is difficult for trajectories consisting of several implicit segments.
So we need to encode start/stop times of recordings and/or movement blocks or use multilinestrings.
This must be compatible with both json encoding and the sqlite schema..

map view should auto update for new data points.

For this, TrajectoryService already exposes a reactive Observable, but it currently does not emit updates.

We could enable that by hooking TrajectoryService.upsertPoint() into the observable

UI: Inference visualization

At the moment, red circular buffer around potential inference location. Diameter is longest distance between two points belonging to the cluster. At the same time, we display a confidence value in a popup.

Potential visualization:

  • Size of buffer (maybe not red?) visualizes uncertainty in meters and not size of cluster?

yarn is not working in this project

I can not use yarn for this project. npm is working.

Steps to reproduce

  • Clone repo
  • yarn && ionic cap sync ios --no-build
  • sync fails
...
✔ Updating iOS plugins in 70.07ms
  Found 1 Capacitor plugin for ios:
    @capacitor-community/sqlite (2.4.2-9)
✖ update ios: ENOENT: no such file or directory, stat '/Users/felix/Documents/git/simport-learning-app/node_modules/@mauron85/cordova-plugin-background-geolocation/ios/common/BackgroundGeolocation/CocoaLumberjack.m'
[error] Error running update: Error: ENOENT: no such file or directory, stat '/Users/felix/Documents/git/simport-learning-app/node_modules/@mauron85/cordova-plugin-background-geolocation/ios/common/BackgroundGeolocation/CocoaLumberjack.m'
    at Object.statSync (fs.js:932:3)
    at Object.statSync (/Users/felix/Documents/git/simport-learning-app/node_modules/graceful-fs/polyfills.js:308:16)
    at Object.copySync (/Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/node_modules/fs-extra/lib/copy-sync/copy-sync.js:31:86)
    at /Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:280:18
    at Array.map (<anonymous>)
    at /Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:268:19
    at Array.map (<anonymous>)
    at copyPluginsNativeFiles (/Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:256:20)
    at Object.updateIOS (/Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:27:9)
    at async /Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/tasks/update.js:66:17 {
  errno: -2,
  syscall: 'stat',
  code: 'ENOENT',
  path: '/Users/felix/Documents/git/simport-learning-app/node_modules/@mauron85/cordova-plugin-background-geolocation/ios/common/BackgroundGeolocation/CocoaLumberjack.m'
}

Fix
use npm install && ionic cap sync ios --no-build

OS and Versions

  • macOS 11.0.1
  • node v12.16.1
  • npm v6.13.4
  • yarn 1.22.10

Wrong duration for user generated trajectory

567312AF-F59F-4489-90FB-A0C4A6EE276A

Yesterday, I freshly installed the app on an iPhone and let it run for a few hours. 51 years is suspiciously close to 1970. I guess there is an issue with computing the duration based on a first / "null" Date object.

Select & source example trajectories

Highlight own trajectories

Simulator Screen Shot - iPhone SE (2nd generation) - 2021-02-03 at 11 15 49

Example trajectories and user trajectory are currently displayed as simple list with the user trajectory in the end of that list. It might make sense to separate example and user trajectory in some way so the user find their trajectory more easily.

Improve UX of viewing own trajectory

Problem

Currently we have weird UX with the user-tracked trajectory:
You enable tracking, go back and open an "example" trajectory, and select "Your Trajectory"..

Proposal

Add a button "analyze my trajectory" below the tracking-toggle on the tracking page, which links to /trajectory/track/user.

Then we have 2 options regarding the trajectory listing:

  • don't list "your trajectory" in the "view example trajectories" listing (as accessed from "view example trajectories" button)
  • rename "view example trajectories" button to something more generic, like "analyze trajectories"

I'm open to other ideas, @felixerdy mentioned something like: removing the separate page for toggling tracking..?

Android: tracking with low & unstable point density

On Android current position updates are very irregular:

  • update frequency averages at around 620s, but not really regular
  • update frequency (?) unrelated to traveled distance, direction
    • during a local trainride, a new update is emitted on every second stop..?!
  • updates stop at some point after the background task was started (irregular, after 2h - 2days)

I see 4 possible causes:

  • the locationProvider we use (default DISTANCE_FILTER_PROVIDER) does not work properly on android
  • the locationProvider we use needs to be configured differently on android
  • the background-process scheduling on these devices is too aggressive, killing the web view running our app
  • the background-process scheduling on these devices is too aggressive, killing the location background service

Might be a device specific issue, we probably need some more device samples.
Affected so far is:

  • LGE Nexus 5X running Android 8.1
  • OnePlus 3T running Android 9 (stock)

Extend location record

Recording should be extended to include speed to each location. On top of that the location-service currently adds a custom timestamp instead of using the given one, which should be used.

  • include speed
  • include (real) timestamps
  • include speed and accuracy in exports
  • include speed and accuracy in imports

UI: Trajectory visualization

When there are a lot of points in one area, those clusters do basically look as if somebody has scribbled across the map with a thick crayon and cover large parts of the map.

Todos:

  • Conceptualize alternative visualizations
  • Create sketches / mockups
  • Discuss in group
  • Implement

Trajectories are replaced rather than expanded

When tracking your own location while already having a recent trajectory, that trajectory gets replaced by a new one rather than expanding the recent trajectory with new recorded locations.

App crashes when notifications are enabled

..and a new point is incoming
edit: The notification is emitted nonetheless. Also the "background tracking stopped"-notification causes a crash.

On Android 8.1 (on a LG nexus 5x). Android API 29 in emulator is affected too.
I'll try to get a stacktrace soon
(Not only the latest version is affected)

Add hidden menu with debug information

Implement a hidden menu that presents debug information (e.g. last gps measurements, number of points, ...)

It should be hidden from the user and only visible after interacting with a hidden / inconspicuous UI element

Integrate staypoint detection into inference computation

This is the follow up card of the first shot of a staypoint detection integration (see #89). After assessing a first feasibility, now an inference engine using the staypoint detection algorithm needs to be implemented (working title: staypoint engine).

  • create staypoint engine skeleton (that satisfies inference engine interface)
  • create tests with data (possible re-use existing tests of simple engine)
  • implement staypiont engine logic for inferences (e.g. adjust scorings etc.)
  • simple mechanisms (for now: in code) to switch between engines
  • implement convex hull
  • make infer of inference engine interface async

File permission prompt when opening "record your own trajectory"

Considering that we're aiming to educate about risks of data collection, this prompt is problematic, as it appears very much out of context and it remains unclear why the permission is needed for that dialog.

Not sure, but I think this is permission prompt is triggered in this dialog, as this is the first access to the sqlite service

permission

Inferences: Rethink confidence value

The confidence value we compute at the moment is somewhat artificial.

Instead we could go for an easier system and work with categories (e.g. low, medium and high confidence) to replace the confidence numbers. This makes the confidence-value way more understandable for users in contrast to bare numbers.

  • Define confidence-categories using thresholds
  • Potentially: remove/replace the term confidence
  • Reiterate the UI for the tab insights
  • Reiterate the UI for inferences in the tab map (popup)
  • Reiterate the UI for filtering the inferences (top-right button)

Inferences: Design Testcases & gather data

  • Load .gpx files for home location, commute to work, work location, commute back home
  • Create staypoint clusters for home and work location
  • Add temporal information
  • Export whole trajectory as .csv
  • Export whole trajectory as .json
  • Export whole trajectory as .gpx
  • Check in .gpx files and load them as default
  • Save output to test-data folder in inferences directory
  • Create separate trajectories with properties that are expected in tests
  • Run test-creation via npm test
  • Load trajectories from .json files in tests

background geolocation plugin seems incompatible with capacitor sqlite plugin

iOS builds fail since adding the @capacitor-community/sqlite plugin in 88146d2 due to incompatibile inclusions of sqlite in that plugin and in @mauron85/cordova-plugin-background-geolocation.

build fail screenshot

Das background geolocation plugin hat die funktionalität daten lokal zu sammeln und in batches an einen server zu schicken. zum cachen der daten nutzt es eine sqlite DB.
Auf android ist standardmäßig schon ein interface zu sqlite systemweit verfügbar, daher gibts jetzt nur auf iOS einen Konflikt mit der anderen sqlite implementierung des sqlite plugins..
Da diese funktionalität für uns schwachsinn ist könnte man drüber nachdenken sie aus dem location plugin rauszuoperieren. Mir fällt grad keine andere option ein, kenne mich aber auch nicht mit dem iOS build process aus, evtl kann man da header umbenennen oder in namespaces stecken um den Namenskonflikt zu vermeiden?

Background-inferences lead to UI freezes

When the inferences are computed, the whole UI freezes for a moment (either while the app is open or when the app is opened, some events are triggered and then the inference computation starts).
This results in a poor user experience and should be investigated and fixed. Some exemplary measures are suggested here:

  • add loading indicator / spinner to UI while inferences are computed

  • don't compute inferences on the UI thread but in the background
    → this seems to be not an option currently: ionic-team/capacitor#2983
    → periodic background tasks coming soon: https://capacitorjs.com/docs/v2/apis/background-task
    ([…] and soon will support periodic background fetch operations.)

  • find performance bottleneck (e.g. with some kind of measurement process)
    → time-wise performance tests with my personal trajectory (~8500 locations):

    • complete inference process takes about 16 seconds
    • clustering (DBSCAN) makes up for 90-95% of that time
  • speed up performance
    → this exceeds the scope of this card
    → staypoint-calculation instead of current clustering possibly delivers a significant performance boost

Research on Trajectory Storage

In the app, we need to

  • store (possibly long) spatial timeseries
  • do efficient spatio-temporal queries on them
  • (run aggregated statistics on the result)
  • on multiple environments / operating systems
  • have privacy preserving storage

We need some tech that fulfills these properties. browser localStorage is probably not sufficient, although it may work for a first prototype.

As a Researcher I want to infer the home location from a trajectory

  • specify semantics of sparse input trajectory data:

    • How do we differentiate between "time between points is time stayed at that point" and "time between points encodes missing spatial data"?
      I'm sure there already plenty of literature & code on this topic.
      A simple and (likely efficient) approach would be to annotate the first point of a new segment / recording session in the databaseare periods between points considered to mean "did not move", or NA, or do we apply some uncertainty metric?

      • Of the example trajectories collected, most consist of several segments (i.e. regular segments combining into a sparse trajectory), so it would make sense to store them as MultiLineStrings.
      • For the user generated trajectectory, location updates are temporally sparse all the time, so it's hard to make a cut between segments in order to interpret temporal gaps in the data.
    • ...?

  • conceptualize an inference engine architecture. ideas:

    • modular: plug in new inference matchers
    • classes of inference matchers, parametrized? for example
      homeInference = new MedianLocationInference({ filters: { timeofDay: [23, 6] } })
      workInference = new MedianLocationInference({ filters: { timeofDay: [10, 14], dayOfWeek: [0, 5] } })
    • streaming: inferences ideally should be able to be computed live on the stream of incoming points, to avoid recomputation on the whole trajectory for each point. This requires some form of serialization/persistence of the inference model state.
    • not relying on DB queries, for it to run in web environment as well? (can we deal with the performance?)

Now inferences shown

When manually triggering inference calculation on my recorded trajectory, they don't show up. Logcat gives this error:

2021-07-10 12:40:04.565 20629-20783/de.ifgi.simport.learning V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 23668412, pluginId: CapacitorSQLite, methodName: query
2021-07-10 12:40:04.566 20629-20783/de.ifgi.simport.learning V/Capacitor: callback: 23668412, pluginId: CapacitorSQLite, methodName: query, methodData: {"database":"trajectories","statement":"SELECT t.type, t.placename, t.durationDays, p.lon, p.lat, p.time, p.accuracy, p.speed FROM trajectories AS t\n        LEFT JOIN points p ON t.id = p.trajectory\n        WHERE t.id = ?\n        ORDER BY time","values":["user"]}
2021-07-10 12:40:12.122 20629-20783/de.ifgi.simport.learning V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 23668413, pluginId: CapacitorSQLite, methodName: run
2021-07-10 12:40:12.123 20629-20783/de.ifgi.simport.learning V/Capacitor: callback: 23668413, pluginId: CapacitorSQLite, methodName: run, methodData: {"database":"trajectories","statement":"DELETE FROM inferences WHERE trajectory='user';","values":[]}
2021-07-10 12:40:12.126 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: runSQL InitChanges 0
2021-07-10 12:40:12.140 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: runSQL Changes 0
2021-07-10 12:40:12.150 20629-20783/de.ifgi.simport.learning V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 23668414, pluginId: CapacitorSQLite, methodName: executeSet
2021-07-10 12:40:12.151 20629-20783/de.ifgi.simport.learning V/Capacitor: callback: 23668414, pluginId: CapacitorSQLite, methodName: executeSet, methodData: {"database":"trajectories","set":[{"statement":"INSERT OR REPLACE INTO inferences VALUES (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?)","values":["user","home","1625913612","6.0985464","50.76815075","wpztHybfd@BoBUs@OBSDkC`Ao@Tr@nEBBtANJCJKz@eBNk@","0.73692792","80.70110939","home","TODO: description","user","work","1625913612","6.0985464","50.76815075","wpztHybfd@BoBUs@OBSDkC`Ao@Tr@nEBBtANJCJKz@eBNk@","0.59989403","80.70110939","work","TODO: description","user","home","1625913612","6.0984369","50.76756668","kpztHyafd@RQQMG?@\\@B@A","0.38337835","9.53531496","home","TODO: description","user","work","1625913612","6.0984369","50.76756668","kpztHyafd@RQQMG?@\\@B@A","0.21246623","9.53531496","work","TODO: description","user","home","1625913612","6.09748248","50.76821082","}sztHk|ed@@CWMCj@XY","0.46929437","11.38825608","home","TODO: description","user","work","1625913612","6.09748248","50.76821082","}sztHk|ed@@CWMCj@XY","0.12530195","11.38825608","work","TODO: description"]}]}
2021-07-10 12:40:12.153 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: ExecuteSet InitChanges 0
2021-07-10 12:40:12.281 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: Error executeSet: table inferences has 9 columns but 10 values were supplied: , while compiling: INSERT OR REPLACE INTO inferences VALUES (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?)
2021-07-10 12:40:12.559 20629-20629/de.ifgi.simport.learning E/Capacitor/Console: File: http://localhost/main-es2015.js - Line 1 - Msg: ERROR Error: Uncaught (in promise): Error: couldnt insert infernence: undefined

I just upgraded to current develop from a rather old version (f873515)
Ex & importing that trajectory, it works. I'll test with a fresh DB, I assume that will work too, so it's some kind of migration issue

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.