kitware / dive

Media annotation and analysis tools for web and desktop. Get started at https://viame.kitware.com

DIVE is a web interface for performing data management, video annotation, and running a portion of the algorithms stored within the VIAME repository. When compiled, docker instances for DIVE can be run either as local servers or online in web services. A sample instance of DIVE is running on a public server at viame.kitware.com.

Features

• video annotation
• still image (and image sequence) annotation
• deep integration with VIAME computer vision analysis tools
• single-frame boxes, polygons, and lines
• multi-frame bounding box tracks with interpolation
• Automatic transcoding to support most video formats
• Customizable labeling with text, numeric, multiple-choice attributes

Documentation

• Client User Guide
• Client Development Docs
• Server Development Docs
• Deployment Overview
• Running with Docker Compose

Technologies Used

DIVE uses Girder for data management and has a typical girder + girder worker + docker architecture. See docker scripts for additional details.

• The client application is a standard @vue/cli application.
• The job runner is built on celery and Girder Worker. Command-line executables for VIAME and FFmpeg are built inside the worker docker image.

Example Data

Input

DIVE takes two different kinds of input data, either a video file (e.g. .mpg) or an image sequence. Both types can be optionally accompanied with a CSV file containing video annotations. Example input sequences are available at https://viame.kitware.com/girder#collections.

Output

When running an algorithmic pipelines or performing manual video annotation (and saving the annotations with the save button) output CSV files are produced containing output detections. Simultaneously a detection plot of results is shown underneath each video sequence.

dive's Issues

Add ability to change track colors manually

Set up testing and continuous deployment

Preliminary objection detection training - Allow users to train for object detection

Currently models are trained via calls to a ‘viame_train_detector’ command line executable tool, e.g.:
https://github.com/VIAME/VIAME/tree/master/examples/object_detector_training
Takes in imagery in a specific format, not sure if we can use this through girder work, directory of directory of images with annotations

Build VIAME with all of the pipelines for docker image

Zooming is laggy

Zooming the video window is too slow, should be more responsive. Seems like the window is doing inertial scroll, but I don't have scrolling inertia turned on.

Also, window seems to have fixed zoom levels, so zooming doesn't happen until you go over some threshold, which makes lag feel worse.

Allow changing of filenames

Pulled out from #18.

If uploading only 1 file, we should allow the renaming of files. This would necessitate an update to the fileUploader mixin in GWC.

Migrate to latest Vue CLI service (v4)

Review Docker VIAME image

@AlmightyYakob we probably need to flush out this task some more

When adding new track go to it in the track list for easier immediate type adjustment

Allow annotator to handle images of different resolutions in the same sequence

Warn user of unsaved edits

When a user tries to leave the annotation page, either by closing the tab or going back to the data browser, they should get a modal warning if there are unsaved annotations in the session.

Disable track editing mode after saving changes

Follow up from #64 (review).

When hitting the save button, the editing track continues to be in edit mode. Changing this back to non-edit mode would be a nice UI/UX enhancement.

Show progress indicator during delete

For large delete operations, it can take several seconds. Need a progress indicator.

Fix running the pipelines

Confirm standard users can't over-write anything in core 'Training Data' folder

Advanced Image caching features

Once complete with the initial cache and sitting there idle, it should start continuing on and loading additional frames. This secondary buffer should be cleared at any time a user interaction starts. This prevents idle network time from not being used while also not polluting too much of the main image buffer.

The caching system should be smart enough to append on if the current frames are loaded.

1. Go to a frame location
2. Look forward and back to see if it is already cached
3. If it is already cached expand the looking distance to prevent idle times

If the previous frames are already loaded while using the front/back ratio it should append to the list the items that should be loaded instead.

If the movement is continuous backwards (indicate that they are seeking backwards) the front back ratio should be changed and priority given to frames before the current frame.

Single frame advancement outside of playback (utilizing the keys) should force the caching to be directional as well. If the user is holding the ‘forward’ key it should be trying to reach forward in frames, the opposite is true if the user is holding the backwards key.

A single frame jump of decent size should be like before where it grabs frames alternating between front and back just in case we don’t know where the user is going.

Video Annotations not working properly

User issue:
"I found your email on your github page for the VIAME. I'm trying to use the web app to upload .mp4 files and then do annotation. I am able to upload the MP4 files successfully, but I never get a 'annotate' blue button next to the files. Is there something I am doing wrong?

The MP4 files are around 2.10GB each. Is there a max file size issue I am running into for using VIAME web app?"

Matthew has noted that no videos have Annotations available for them.

Add split track option (on current frame)

Investigate pipeline-generated annotaiton alignment issue with videos

• Pipelines generate at 5hz.
• Playback works
• Make sure the annotation format is consistent (should have frame identifiers consistent with the video)

Add a processing queue for handling multiple training and/or detection jobs

This might already be the case if jobs or queued succesfully or run in batches of max 2 jobs, but needs verification if this is the case

When clicking the track id number in the track list go to the track start location

Unless there's a better 1 click way here

Migrate to latest vue-cli-plugin-vuetify and vuetify-loader

Wrap breadcrumb instead of action buttons

Change how the data browser's widget area wraps.

Upload progress reporting is insufficient.

The upload progress controls need some attention. All you get is an indeterminate progress bar and a label that says 0 images the whole time an upload is happening.

There's nothing to indicate to the user if things are still happening or the UI has gotten stuck, particularly if the upload is large.

I'd consider this of moderate priority.

Left clicking off a track on background should de-select track

When a region not inside the track box or near the track classification text right of the box is selected, it would be ideal to de-select the currently selected track

Add distinction between exporting filtered and unfiltered tracks

Document basic interactions and link the document from the application

species types as attribute confusion

"Another thing that's come up is people keep on adding species types as attributes and we'd want to make sure it's highlighted in the document which is which"

@mattdawkins please clarify what different parts of the app are being confused and what needs to be communicated. As far as I'm aware, there's only one way to provide freeform text.

Upload improvements (support folder of folder of images or folder of videos .mpg instead of single video or single folder of images)

UPDATE Clarification: Highest priority we want to be able to create a folder for each high level item in a folder. Mostly Videos want to be able to uploaded as individual folders instead of a single folder. Add in a toggle to make it so there are single files.

Bug: placing head or tail point while a track is in edit mode enters invalid state

• right click a track to enter edit mode
• press f
• click anywhere until a large dot appears.

You end up with something that looks like a new track but is different than all the others and as far as I can tell is not supposed to be there. You can drag the dot once to turn it into a box:

Improve Run Pipeline drop down ordering

Ideally the ordering would be:

default pipelines first
default_*

generic pipelines next
generic_*

then all others in alphabetical order

Ideally it also doesn't show the .pipe extension

Add ability to download individual files or just the annotation csv in a folder

Instead of the entire folder

Track selection behavior doesn't make sense.

• Left click to select track A
• right click to edit track B

Track A is still highlighted in the track list and is still green, which doesn't make sense to me

• press f

You're now in an invalid state (#10) but now track A has been deleted.

Timeline scaling on resize

Currently there is no timeline scaling when the user resizes the window. Instead it just remains at the default view size.

The Timeline and anything below the timeline should scale accordingly with the window size.

Fix linting to use AirBnB instead of Prettier

The difference between prettier and airbnb is causing me pain, but chainging it will touch every file and be a huge headache.

Unreadable colors for UI elements

Change to the base color makes it more difficult to read some of the UI elements in the page

Update to either use the accent color used in other UI elements instead of the base color for the text fields.
Test in any/every text field or other UI element location: login, register, new folder, attribute editing.
Test in both chrome and firefox to confirm that the representation is the same.

Code restructuring to support? Which code to restructure for what feature? Output in a document would be desired.

Make it more clear which track is highlighted in the list when selecting a track

Controls are undocumented (Create modals within the APP)

Most of the controls are undocumented. I found out, for example, that you have to press "escape" in order to save the state of a track, otherwise if you adjust a track then move to the next frame, the adjustment is lost.

Cannot check type filter check boxes only text, same thing for attribute bullets when making new attribute, can only click text next to bullet

When up or down arrows are pressed cycle to the next track in the track list

[bug] checkbox click not working

upgrading to vuetify ^2.2.4 fixes it

Document how to do annotation using VIAME-Web as it stands today

Seeking backward doesn't work, prefetching doesn't wait for frames to load.

Pre-fetching only works when playing the video forward in time. However, when doing an annotation, a user will want to find the beginning of the track, which can involve seeking backward.

• Find a fish you want to track
• grab the track head and drag backward to look for the fish's first appearance.

result: lots of waiting for an image to appear.

Make configurable settings and detection/track attributes be user-specific, not global with no access restrictions

People keep on changing them (ideally it would show whatever attributes are a part of the currently loaded displayed video [if any] + user attributes

Applies to detection and track attributes

When delete key is pressed it deletes current track if there's one selected

click handler misses

About 25% of my clicks don't do the thing they're supposed to do. Highlighting tracks, laying head and tail points, and everything else on the video window does nothing about a quarter of the time.

I don't know if this is a problem with this app or GeoJS's event hander. I use an apple Magic Trackpad, so I wonder if the slight movement of my cursor during a click is causing the click events to get interpreted as drags. Maybe the threshold for lateral movement during a click can be tuned?

Add switch that controls creating a folder in uploader

Feature that was removed from #18, due to conflict with #17, and missing functionality in the GWC fileUploader mixin.

Review color choices for the application with UI/UX designe

Image filter list not supporting all types of images (SGI)

The image filter in the uploading of files filters out specific images.
One of the ones to add is SGI
The current list supports:

• .jpg
• .jpeg
• .png
• .bmp

Get from @mattdawkins a list of images we would like to support