aplbrain / colocar

tools for sparse human annotation of three-dimensional [neuroscience] datasets

colocar

Introduction

Welcome to colocar! This repository houses a number of React.js- and p5.js-based tools for sparse annotations of three-dimensional (primarily spatial) datasets and complementary tools for validation of those sparse annotations.

These are primarily used through deployment to Amazon Web Services (AWS), but they can also be run locally with ease. In either case, they require access to a spatial database that houses the data to be annotated and a document-oriented database to receive the annotations. In particular, the current production workflow pulls imagery from the Boss and pushes annotations to colocard.

Setup

Preamble: colocorazon

First, one must build the shared libraries, found in colocorazon. This can be done by cd-ing into colocorazon and running yarn && yarn build. This will transpile the shared libraries into browser-friendly, old-timey JavaScript.

cd colocorazon
yarn run build

Then, for example:

cd ../breadcrumbs
yarn add ../colocorazon

You can also directly link the build like so:

cd colocorazon
yarn run build
yarn link

cd pointfog
yarn link colocorazon

Changes to the colocorazon build will now immediately take effect in the link-downstreams. (So... exercise caution if doing this for a production deploy!)

Amble: installing dependencies

Next, the apps require pulling modules into their node_modules. At present, this can be done by cd-ing into the app directory and running yarn.

Postamble: with your powers combined!

Finally, with all dependencies built and installed, we are ready to run and deploy our apps.

Running Locally

Each of the apps has a configuration file in the src directory called _config.json. Point this configuration file at the colocard host URL before launching.

At this stage, the apps can be run locally by cd-ing into their directories and running yarn start. When the browser launches a window, it will prompt you to enter credentials to the Boss. After successful entry, you will have access to the image data and the web app.

Deploying to AWS

Each of the colocar apps can be easily built and deployed to AWS using the handy script, colocart. This tool requires a python3 installation with the libraries boto3, click, and colored.

Be sure that the apps have already been set up on AWS via the quickstart-website portal (TODO: this tool was deprecated and is unavailable). This connects a CloudFront service to a particular S3 bucket. The name of that S3 bucket must be entered into the build.cfg file such that the contents are as follows.

[UPLOAD]
BucketName = <S3-BUCKET-NAME-OF-APP>

Once the apps have had their dependencies installed, the python3 environment is ready, and the configuration file is updated, one can use ./colocart build <APP-NAME> && ./colocart deploy <APP-NAME> to deploy a fresh version of the current working directory to AWS.

Directories

	pointfog	Pointcloud generation from images
	breadcrumbs	Skeleton 'graph' tracing from images
	matchmaker	Skeleton 'graph' pairwise visualization
	macchiato	forced choice point proofreading
	nazca	forced choice graph proofreading

---

Made with ♥ at

colocar's Issues

Can't copy question ID from UI

MouseDragged might be overwriting the mouse events on UI?

Implement postNodes in Pointfog/db.js

• save button
• [maybe?] override ⌘/ctrl-S

Single localForage key for all questions

The current key for localForage in pointfog does not depend on the questionId, which means that one question's points could be loaded for another question if the localForage was not properly purged ie if the node upload failed. Adding the questionId to the key should fix the problem.

UI to show how many nodes have been placed

Fix centering of pointfog view

Automatically center view and crosshairs at image centers.

Undo placement with Backspace

Hit backspace to undo the last node placement; ideally should work arbitrary number of times

Add border annotation

Add annotation option to indicate the border of a volume in breadcrumbs. Also prohibit upload of trace with leaf nodes without annotation of axon (a), dendrite (d), or border (b).

Fix node placement when given a starting node

When creating a TraceManager, it should be possible to pass a startingGraph ([code deeplink], issue: #30).

Right now, nodes passed to this constructor do not appear in the correct location (based upon this XYZ-remapping code).

Ability to branch from existing nodes

...in order to form acyclic graph.

Mark "bookmark" nodes

! to mark a node to return to it with @

mark starting synapse and prevent type-override

Permit passing a starting-graph to TraceManager

Given nodes with XYZ in data-space and edges between them, spawn a graph when a user loads the page. Should set protected = true on nodes to prevent deletion.

Project node XYZs from p5 space to Image space

TraceManager.js:56

First right-click always hints

If one wants to select a previously placed node in breadcrumbs, right-clicking is the only option. However, right-clicking once will show a hint for node placement even if the cursor is over an existing node. A second click will select the node. This behavior is confusing, and so we should just select the node on the first click.

Add pop-ups for confirmation

When submitting, add an interactive session to confirm that submit is clear and intentional. No analogous pop-up for saving because that happens often in background.

Centralize DB colocard.js functions

Depends on:

• #35
• #34
• #32

Pull colocard.js into its own node module; import from breadcrumbs and pointfog

Breadcrumbs 3D minimap

via either threejs or our own substrate

Move changes to breadcrumbs

• page centering
• remove pan mode
• attach event listeners to canvas instead of p #59 (@j6k4m8)
• submit / save #23 #34
• submit / save buttons

zoom to center of FOV instead of center of image

gracefully handle key/mouse events on No-More-Questions screen

Node deletion seems to randomly fail on small graphs

Steps to reproduce

• Add node to starting node
• Append node to added node
• Delete nodes in LIFO order

Appears to fail with remapping error. Deleting node prematurely from array?
Race condition in render loop?

Move node bookmark boolean to not overwrite node type

...this was dumb of me; marking a node as a bookmark (and then removing the bookmark) removes the node type, erasing axon/dendrite. Should probably have its own separate isBookmarked attribute.

See also #28

Arbitrary delete in pointfog

At present, deleting a node in pointfog selects a new activeNode immediately. If people hit the delete key multiple times, this is almost guaranteed to result in behavior they do not want or expect. We should require explicit select to delete in pointfog.

Highlight bookmarked nodes in a meaningful way

• ...on other slices: https://github.com/aplbrain/colocar/blob/breadcrumbs-2.tucker/breadcrumbs/src/layers/TraceManager.js#L283-L285
• ...on current slice: https://github.com/aplbrain/colocar/blob/breadcrumbs-2.tucker/breadcrumbs/src/layers/TraceManager.js#L330-L333

Add save functionality

Include some kind of UI to access the saveNodes function in pointfog to store nodes in the browser before submitting

• save button
• save keys (ctrl+s)
• autosave

Restrict node selection to those within n frames of current z

Currently, you can select ANY node (only checks X and Y distance). Need to restrict to "near" Z slices.

TraceManager.js:47

Frame starts at hard-coded center

Even for volumes with 250 z-slices, the initial frame is set to near 50. This seems likely to be hard-coded.

Fade out on both sides of path (not just "top")

Currently, on slice n-1, a path on slice n is invisible (though on n+1 it is faded)

Entity indicators in app scrollbars

Show entities in scrollbar; e.g. bookmarks, synapses, and nodes

Load from boss (mock)

• ability to load from boss as capability proof (non-question-loading)

user-facing instructions

#79

Fail silently if setting a node type and no node is selected

Position values do not change with current zoom

It seems like the position values stay constant as one zooms the UI, despite the fact that the zoom moves the position to UI center. This seems like it would have to result in wrong values for things like node position placement and might explain the "spider web" patterns we've seen.

"Saving..." indicator

Perhaps a toast of some sort to indicate that a roundtrip is taking place so the user doesn't mash the submit button.

Separately; should debounce the submit button so the user can't upload twice

Centralize ImageManager to be imported into apps

Make single ImageManager implementation, importable into apps

Add pointfog select old node

Right now it seems to be difficult to select a node that was dropped before. It may not be possible to delete a node if this is the case.

Question ID indicator in-app

• breadcrumbs
• pointfog

Implement Colocard#postGraph in db.js

See also #32.

Should be able to post a graph to colocard; may need to modify the graph in order to hit the colocard schema.

Will definitely need to add author (maybe add argument to fn?), timestamp (new Date() * 1), etc.

postGraph(graph: graphlib.DiGraph): Promise<Object>

Use cutout service instead of image slice

This saves on data-use pretty dramatically

This code currently uses the image endpoint. There is a cutout endpoint (docs) which accepts an Accept-Type header of jpeg, which returns a 'film-strip'-style image.

Then we'll need to "window" that long image to get individual square XY images. You can use the p5#image fn to do this (see last example on this page)

• breadcrumbs
• pointfog

Gotchas:

• We use the ImageManager.imageURLs.length in a few places around the codebase (don't remember the exact name of that var, but something like that). Need to replace with a getZDepth() or something like that.
• It's possible that the full volume is too big to load in one cutout request, so I'd suggest we cut the download into five-ish segments:

[ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ]

...where the 3 segment contains the starting synapse, and is loaded first.

Support annotation of node-types

a to notate Axon

d to notate Dendrite

Should override wasd nav (can use arrows, or just disable for now)

Implement colocar#getNextQuestion in db.js

Implement the colocard API for only required endpoints in breadcrumbs/src/db.js.

There is an existing interface, Database, which has one required function, getNextQuestion. This currently returns a question AND a volume; probably we should just return a single object which contains all of the required information, in line with the colocard schema. (This was developed prior to finishing the question work in colocard, so when there are disagreements between this implementation and colocard, use the colocard schemas/types!)

Scoping this issue a bit tighter, JUST implement the getNextQuestion function; postGraph can happen as a separate issue.

getNextQuestion: (user: string, type: string): Promise<colocard.Question>

Prevent points from reappearing in wrong layers in pointfog

If a user drops a point and then scrolls, the point gets smaller until it disappears. However, if the user then drops another point, the first point also reappears, possibly at its full original size.

Allow deletion of selected nodes

...even if this would break graph continuity.

confirm that node types are forwarded to colocard

Interactions with ui insert nodes

If you click on the zoom button or submit button, nodes will appear on the EM underneath where you clicked. We should probably disable this and have some sense of what is in the foreground.

Bookmarks render quirk

Bookmarks appear opaque even when out of view or many slices away. Whenever a new node is dropped, the bookmark opacity returns to full.

Safely handle out-of-questions behavior instead of failing

Currently, fails with "Cannot read property question of undefined", because of the dictionary-unpacking syntax.

Should fix by assigning a single fn arg, and then checking for falsiness

colocar/pointfog/src/PointfogApp.js

Line 100 in 70851a9

).then(({question, volume}) => {

• breadcrumbs
• pointfog

Enable graph save

Allow for localForage save of graph in breadcrumbs.

Can delete non-leaf nodes

Current breadcrumbs allows deletion of active node, even when this would break trace into multiple connected components. Should disallow and only delete from leaves or else maintain tree structure somehow.

Remove pan/trace modality