sticky-pi / sticky-pi-api

Server and client api for the Sticky Pi project

The API for the Sticky Pi project

main branch:

develop branch:

---

Project organisation:

• src a Python package (sticky_pi_api) that defines the API and its client. Complete documentation of the client and the api is available on our website.

• server a docker-based server, deployed using docker-compose. Complete documentation here. Contains the following services:

  • db the database (MySQL)
  • api a flask server that instantiates our API and routes entry points
  • rshiny a webtool to visualise the data
  • s3 an (optional) s3 server to store and serve resources such as images (can set a remote S3 server instead)
  • nginx an server to route trafic, set subdomains, etc

sticky-pi-api's Issues

TODO: in Image Fetching for Scoping by Selecting Project, Match Specific Device IDs in Addition to Dates

Currently, when a project selected, image fetching (when user goes to Images, Data tabs) is simply done by feeding date ranges from the project series metadata table into the api_get_images() general function for scoping by date range.
However, there may be images and metadata within the date ranges of a project that are not actually part of the project.

Enable user submit by Enter key

Users should be able to submit input fields (for add row, column, edit cell) by hitting Enter, not just clicking outside.

set up branches

Set up dev and other branches and workflow

Implement user delete project and series tables row

Add a "delete" button similar to the current "+" button for adding a new series row.
Can tie button click to a function that deletes the selected row of the underlying table in state.

joblib writes very many files

if we cache the md5sum computations wioth joblib, using filesystem backend, this results in at least thrice the number of files whic could be limiting (e.g. we have file number limis to 500k on ComputeCanada 🤦 )

Special coordinate columns

Special metadata columns lat, lng may exist with these names.

• Users should not be allowed to chose these names
• these names should be mapped lng: Longitude,... for display.
• Column types drop down should only offer these types when they are not already defined

Disallow direct cell editing of "Role" in Projects Table, Implement "invite (users)" modal interface

Implement user add columns to series tables

Site greying out on downloading single images

after clicking opened(in modal) thumbnail and successfully downloading image from server(ComputeCanada) link, browser console: this.$socket is null, modal buttons don't work and cannot re-invoke any new thumbnail modals/popup from timeline anymore

• R Shiny Apps Greys Out and Hangs
  perhaps some safety feature of Shiny site-serving: grey out when unrecognized interruption to connection, e.g. visiting external link
  -> maybe HTML element external links will all break Shiny webapp? for any downloads can only use Shiny's download buttons?
  -> site also grey out when invalid user/login, but can't interact with anything

Improve query performance

It takes too long to simply query image series:

time curl --user XXXXXXX:XXXXXXXXXXX -X POST -H "Content-Type: application/json" -d '[{"start_datetime": "2020-06-01_00-00-00", "end_datetime": "2030-01-01_00-00-00", "device": "%"}]' https://api.sticky-pi.com/get_image_series/image > /dev/null takes 20s on the host that is for ~75k images (URL pre-cached)

https://stackoverflow.com/questions/23185319/why-is-loading-sqlalchemy-objects-via-the-orm-5-8x-slower-than-rows-via-a-raw-my
https://stackoverflow.com/questions/31387980/better-understand-sqlalchemys-yield-per-problems

needs profiling, but it looks like it is due to the use of ORM and how we generate output json (Base.to_dict)

project cell editing content checking

ATM, when editing a cell with an erroneous value e.g. "a" for a numerical column, the server API responds with an error, and shiny crashes with no explanation. We could:

• Check format before sending
• Display an error message when an unknown server error prevented cell edition

User edits to projects metadata table not being saved

After editing a cell of the projects table, the new value is shown (also after clicking away on the side), until the user interacts with any of the DataTable widget. Then, the app seems to refetch the table and the original value at startup is shown again.

Nginx server in docker