sitcomlab / simport-learning-app Goto Github PK

probably a query in SqliteService.upsertTrajectory() fails

TBD what this means:

• data does not leave the device?
• data is encrypted?
  • the sqlite driver we use provides encryption!
• ...?

In general, what is the threat model?

Considering that we're aiming to educate about risks of data collection, this prompt is problematic, as it appears very much out of context and it remains unclear why the permission is needed for that dialog.

Not sure, but I think this is permission prompt is triggered in this dialog, as this is the first access to the sqlite service

In the app, we need to

• store (possibly long) spatial timeseries
• do efficient spatio-temporal queries on them
• (run aggregated statistics on the result)
• on multiple environments / operating systems
• have privacy preserving storage

We need some tech that fulfills these properties. browser localStorage is probably not sufficient, although it may work for a first prototype.

For this, TrajectoryService already exposes a reactive Observable, but it currently does not emit updates.

We could enable that by hooking TrajectoryService.upsertPoint() into the observable

• Import script
  • exists now: https://github.com/sitcomlab/simport-learning-app/blob/d36f9260aacc8b03430fc9ae6a9ae63e1a42455f/dev/import_example_trajectory.ts
• select examples.
  • what criteria?
  • Norwin started a collection of trajectories here. Of these, very few trajectories record a persons general life. Of those that do, none provide high spatial or temporal accuracy. So we probably need to search further, or find a volunteer to record & provide their own trajectory

• specify semantics of sparse input trajectory data:

  • How do we differentiate between "time between points is time stayed at that point" and "time between points encodes missing spatial data"?
    I'm sure there already plenty of literature & code on this topic.
    A simple and (likely efficient) approach would be to annotate the first point of a new segment / recording session in the databaseare periods between points considered to mean "did not move", or NA, or do we apply some uncertainty metric?

    • Of the example trajectories collected, most consist of several segments (i.e. regular segments combining into a sparse trajectory), so it would make sense to store them as MultiLineStrings.
    • For the user generated trajectectory, location updates are temporally sparse all the time, so it's hard to make a cut between segments in order to interpret temporal gaps in the data.

  • ...?

• conceptualize an inference engine architecture. ideas:

  • modular: plug in new inference matchers
  • classes of inference matchers, parametrized? for example

    homeInference = new MedianLocationInference({ filters: { timeofDay: [23, 6] } })
    workInference = new MedianLocationInference({ filters: { timeofDay: [10, 14], dayOfWeek: [0, 5] } })

  • streaming: inferences ideally should be able to be computed live on the stream of incoming points, to avoid recomputation on the whole trajectory for each point. This requires some form of serialization/persistence of the inference model state.
  • not relying on DB queries, for it to run in web environment as well? (can we deal with the performance?)

When tracking your own location while already having a recent trajectory, that trajectory gets replaced by a new one rather than expanding the recent trajectory with new recorded locations.

Implement a hidden menu that presents debug information (e.g. last gps measurements, number of points, ...)

It should be hidden from the user and only visible after interacting with a hidden / inconspicuous UI element

Problem

Currently we have weird UX with the user-tracked trajectory:
You enable tracking, go back and open an "example" trajectory, and select "Your Trajectory"..

Proposal

Add a button "analyze my trajectory" below the tracking-toggle on the tracking page, which links to /trajectory/track/user.

Then we have 2 options regarding the trajectory listing:

• don't list "your trajectory" in the "view example trajectories" listing (as accessed from "view example trajectories" button)
• rename "view example trajectories" button to something more generic, like "analyze trajectories"

I'm open to other ideas, @felixerdy mentioned something like: removing the separate page for toggling tracking..?

...but create a new branch

Recording should be extended to include speed to each location. On top of that the location-service currently adds a custom timestamp instead of using the given one, which should be used.

• include speed
• include (real) timestamps
• include speed and accuracy in exports
• include speed and accuracy in imports

It would be useful to have a enhanced automated testing-setup - here are some notes from Corinna on this:

In general I would use appium to mock geo location. To run an appium test as part of the CI pipeline I would use a dockerized android emulator like this one. It also comes with a Firebase Test Lab integration

To cover more than just virtual android devices, I would use Saucelabs, Browserstack, Lambatest or AWS device farm.

At the moment, the app displays a notification when the user location is updated. Although useful for debugging purposes, it can be rather annoying.

see https://miro.com/app/board/o9J_laryiVc=/

& educational info what enabled each inference

Yesterday, I freshly installed the app on an iPhone and let it run for a few hours. 51 years is suspiciously close to 1970. I guess there is an issue with computing the duration based on a first / "null" Date object.

At the moment, red circular buffer around potential inference location. Diameter is longest distance between two points belonging to the cluster. At the same time, we display a confidence value in a popup.

Potential visualization:

• Size of buffer (maybe not red?) visualizes uncertainty in meters and not size of cluster?

The map should update after a new location is added to the database. Additionally, the last location should be highlighted on the map (e.g. with a position marker)

I can not use yarn for this project. npm is working.

Steps to reproduce

• Clone repo
• yarn && ionic cap sync ios --no-build
• sync fails

...
✔ Updating iOS plugins in 70.07ms
  Found 1 Capacitor plugin for ios:
    @capacitor-community/sqlite (2.4.2-9)
✖ update ios: ENOENT: no such file or directory, stat '/Users/felix/Documents/git/simport-learning-app/node_modules/@mauron85/cordova-plugin-background-geolocation/ios/common/BackgroundGeolocation/CocoaLumberjack.m'
[error] Error running update: Error: ENOENT: no such file or directory, stat '/Users/felix/Documents/git/simport-learning-app/node_modules/@mauron85/cordova-plugin-background-geolocation/ios/common/BackgroundGeolocation/CocoaLumberjack.m'
    at Object.statSync (fs.js:932:3)
    at Object.statSync (/Users/felix/Documents/git/simport-learning-app/node_modules/graceful-fs/polyfills.js:308:16)
    at Object.copySync (/Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/node_modules/fs-extra/lib/copy-sync/copy-sync.js:31:86)
    at /Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:280:18
    at Array.map (<anonymous>)
    at /Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:268:19
    at Array.map (<anonymous>)
    at copyPluginsNativeFiles (/Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:256:20)
    at Object.updateIOS (/Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/ios/update.js:27:9)
    at async /Users/felix/Documents/git/simport-learning-app/node_modules/@capacitor/cli/dist/tasks/update.js:66:17 {
  errno: -2,
  syscall: 'stat',
  code: 'ENOENT',
  path: '/Users/felix/Documents/git/simport-learning-app/node_modules/@mauron85/cordova-plugin-background-geolocation/ios/common/BackgroundGeolocation/CocoaLumberjack.m'
}

Fix
use npm install && ionic cap sync ios --no-build

OS and Versions

• macOS 11.0.1
• node v12.16.1
• npm v6.13.4
• yarn 1.22.10

..and a new point is incoming
edit: The notification is emitted nonetheless. Also the "background tracking stopped"-notification causes a crash.

On Android 8.1 (on a LG nexus 5x). Android API 29 in emulator is affected too.
I'll try to get a stacktrace soon
(Not only the latest version is affected)

• Staypoint detection as service
• Save staypoints to DB
• Execute staypoint detection via manual script

The confidence value we compute at the moment is somewhat artificial.

Instead we could go for an easier system and work with categories (e.g. low, medium and high confidence) to replace the confidence numbers. This makes the confidence-value way more understandable for users in contrast to bare numbers.

• Define confidence-categories using thresholds
• Potentially: remove/replace the term confidence
• Reiterate the UI for the tab insights
• Reiterate the UI for inferences in the tab map (popup)
• Reiterate the UI for filtering the inferences (top-right button)

• add browser fallback for persistence layer (since adding sqlite, we need dropped browser persistence..)
  • think about how that affects our approach to inference calculation, if we can't rely on a DB for filtering etc?
• add exemplary trajectories to persistence layer

• Load .gpx files for home location, commute to work, work location, commute back home
• Create staypoint clusters for home and work location
• Add temporal information
• Export whole trajectory as .csv
• Export whole trajectory as .json
• Export whole trajectory as .gpx
• Check in .gpx files and load them as default
• Save output to test-data folder in inferences directory
• Create separate trajectories with properties that are expected in tests
• Run test-creation via npm test
• Load trajectories from .json files in tests

• Enable access to filesystem
• Export trajectories
  • update UI to enable users to export trajectories
  • iOS
  • Android
• Import trajectories
  • iOS
  • Android
    - Evaluate imports to DB vs .json-files to filesystem (low priority for now)
• Unit tests

based on concept developed in https://miro.com/app/board/o9J_laryiVc=/

Todos:

• Save correct location update timestamps (see #52)
• Try out cluster generation with DBSCAN
• Integrate clustering into app
• Simple temporal analysis
• Integrate into test cases
• Cleanup and make ready for beta-release

• Create test data based on Nepal trajectory
• Set up test harness
• Define API

On Android current position updates are very irregular:

• update frequency averages at around 620s, but not really regular
• update frequency (?) unrelated to traveled distance, direction
  • during a local trainride, a new update is emitted on every second stop..?!
• updates stop at some point after the background task was started (irregular, after 2h - 2days)

I see 4 possible causes:

• the locationProvider we use (default DISTANCE_FILTER_PROVIDER) does not work properly on android
• the locationProvider we use needs to be configured differently on android
• the background-process scheduling on these devices is too aggressive, killing the web view running our app
• the background-process scheduling on these devices is too aggressive, killing the location background service

Might be a device specific issue, we probably need some more device samples.
Affected so far is:

• LGE Nexus 5X running Android 8.1
• OnePlus 3T running Android 9 (stock)

Example trajectories and user trajectory are currently displayed as simple list with the user trajectory in the end of that list. It might make sense to separate example and user trajectory in some way so the user find their trajectory more easily.

• Persistence: actually store locations
• long/midterm background runtime testing on android
  • initial tests on nexus 5 indiciate that location updates stop after a while.

When there are a lot of points in one area, those clusters do basically look as if somebody has scribbled across the map with a thick crayon and cover large parts of the map.

Todos:

• Conceptualize alternative visualizations
• Create sketches / mockups
• Discuss in group
• Implement

When manually triggering inference calculation on my recorded trajectory, they don't show up. Logcat gives this error:

2021-07-10 12:40:04.565 20629-20783/de.ifgi.simport.learning V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 23668412, pluginId: CapacitorSQLite, methodName: query
2021-07-10 12:40:04.566 20629-20783/de.ifgi.simport.learning V/Capacitor: callback: 23668412, pluginId: CapacitorSQLite, methodName: query, methodData: {"database":"trajectories","statement":"SELECT t.type, t.placename, t.durationDays, p.lon, p.lat, p.time, p.accuracy, p.speed FROM trajectories AS t\n        LEFT JOIN points p ON t.id = p.trajectory\n        WHERE t.id = ?\n        ORDER BY time","values":["user"]}
2021-07-10 12:40:12.122 20629-20783/de.ifgi.simport.learning V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 23668413, pluginId: CapacitorSQLite, methodName: run
2021-07-10 12:40:12.123 20629-20783/de.ifgi.simport.learning V/Capacitor: callback: 23668413, pluginId: CapacitorSQLite, methodName: run, methodData: {"database":"trajectories","statement":"DELETE FROM inferences WHERE trajectory='user';","values":[]}
2021-07-10 12:40:12.126 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: runSQL InitChanges 0
2021-07-10 12:40:12.140 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: runSQL Changes 0
2021-07-10 12:40:12.150 20629-20783/de.ifgi.simport.learning V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 23668414, pluginId: CapacitorSQLite, methodName: executeSet
2021-07-10 12:40:12.151 20629-20783/de.ifgi.simport.learning V/Capacitor: callback: 23668414, pluginId: CapacitorSQLite, methodName: executeSet, methodData: {"database":"trajectories","set":[{"statement":"INSERT OR REPLACE INTO inferences VALUES (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?)","values":["user","home","1625913612","6.0985464","50.76815075","wpztHybfd@BoBUs@OBSDkC`Ao@Tr@nEBBtANJCJKz@eBNk@","0.73692792","80.70110939","home","TODO: description","user","work","1625913612","6.0985464","50.76815075","wpztHybfd@BoBUs@OBSDkC`Ao@Tr@nEBBtANJCJKz@eBNk@","0.59989403","80.70110939","work","TODO: description","user","home","1625913612","6.0984369","50.76756668","kpztHyafd@RQQMG?@\\@B@A","0.38337835","9.53531496","home","TODO: description","user","work","1625913612","6.0984369","50.76756668","kpztHyafd@RQQMG?@\\@B@A","0.21246623","9.53531496","work","TODO: description","user","home","1625913612","6.09748248","50.76821082","}sztHk|ed@@CWMCj@XY","0.46929437","11.38825608","home","TODO: description","user","work","1625913612","6.09748248","50.76821082","}sztHk|ed@@CWMCj@XY","0.12530195","11.38825608","work","TODO: description"]}]}
2021-07-10 12:40:12.153 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: ExecuteSet InitChanges 0
2021-07-10 12:40:12.281 20629-20744/de.ifgi.simport.learning V/com.getcapacitor.community.database.sqlite.SQLite.Database: Error executeSet: table inferences has 9 columns but 10 values were supplied: , while compiling: INSERT OR REPLACE INTO inferences VALUES (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?), (?,?,?,?,?,?,?,?,?,?)
2021-07-10 12:40:12.559 20629-20629/de.ifgi.simport.learning E/Capacitor/Console: File: http://localhost/main-es2015.js - Line 1 - Msg: ERROR Error: Uncaught (in promise): Error: couldnt insert infernence: undefined

I just upgraded to current develop from a rather old version (f873515)
Ex & importing that trajectory, it works. I'll test with a fresh DB, I assume that will work too, so it's some kind of migration issue

As outlined in #8, we need to interpret time between points for inferences.
This is difficult for trajectories consisting of several implicit segments.
So we need to encode start/stop times of recordings and/or movement blocks or use multilinestrings.
This must be compatible with both json encoding and the sqlite schema..

When the inferences are computed, the whole UI freezes for a moment (either while the app is open or when the app is opened, some events are triggered and then the inference computation starts).
This results in a poor user experience and should be investigated and fixed. Some exemplary measures are suggested here:

• add loading indicator / spinner to UI while inferences are computed

• don't compute inferences on the UI thread but in the background
  → this seems to be not an option currently: ionic-team/capacitor#2983
  → periodic background tasks coming soon: https://capacitorjs.com/docs/v2/apis/background-task
  ([…] and soon will support periodic background fetch operations.)

• find performance bottleneck (e.g. with some kind of measurement process)
  → time-wise performance tests with my personal trajectory (~8500 locations):

  • complete inference process takes about 16 seconds
  • clustering (DBSCAN) makes up for 90-95% of that time

• speed up performance
  → this exceeds the scope of this card
  → staypoint-calculation instead of current clustering possibly delivers a significant performance boost

Notifications about received location update come in too often. The following screenshot is from a timespan of about 30 to 40 minutes. Maybe a hourly, daily or weekly summary-notification is sufficient (possibly even via user-setting?)

This is the follow up card of the first shot of a staypoint detection integration (see #89). After assessing a first feasibility, now an inference engine using the staypoint detection algorithm needs to be implemented (working title: staypoint engine).

• create staypoint engine skeleton (that satisfies inference engine interface)
• create tests with data (possible re-use existing tests of simple engine)
• implement staypiont engine logic for inferences (e.g. adjust scorings etc.)
• simple mechanisms (for now: in code) to switch between engines
• implement convex hull
• make infer of inference engine interface async

iOS builds fail since adding the @capacitor-community/sqlite plugin in 88146d2 due to incompatibile inclusions of sqlite in that plugin and in @mauron85/cordova-plugin-background-geolocation.

Das background geolocation plugin hat die funktionalität daten lokal zu sammeln und in batches an einen server zu schicken. zum cachen der daten nutzt es eine sqlite DB.
Auf android ist standardmäßig schon ein interface zu sqlite systemweit verfügbar, daher gibts jetzt nur auf iOS einen Konflikt mit der anderen sqlite implementierung des sqlite plugins..
Da diese funktionalität für uns schwachsinn ist könnte man drüber nachdenken sie aus dem location plugin rauszuoperieren. Mir fällt grad keine andere option ein, kenne mich aber auch nicht mit dem iOS build process aus, evtl kann man da header umbenennen oder in namespaces stecken um den Namenskonflikt zu vermeiden?

Additional: State running remains, even if the tracking is not running anymore