biigle / videos Goto Github PK

View Code? Open in Web Editor NEW

0.0 4.0 0.0 13.67 MB

:m: BIIGLE module to create, edit and explore video annotations

biigle biigle-module video-annotation

videos's Introduction

BIIGLE Videos Module

This is the BIIGLE module to create, edit and explore video annotations.

Deprecated: This module is deprecated as the features have been merged into biigle/core.

videos's People

Contributors

Watchers

videos's Issues

Implement linking of annotation clips

If an object moves out of screen and then back in again, there are two separate annotations that mark the same object. Users should be able to link these annotations. This can be implemented by introducing "gaps" for an annotation clip. A gap between two keyframes is defined by a negative frame time and an empty points array. Example:

frames: [1, 2, null, 4, 5]
points: [[10, 10], [15, 15], [], [30, 30], [40, 40]]

This will be displayed as a rectangle with two keyframes from second 1 to 2, a rectangle with two keyframes from second 4 to 5 and a thin horizontal line in the color of the label from second 2 to 4.

Two (or more) annotations can be merged by selecting all of them and then clicking on a special "merge" button. This will delete all but one of the annotations and merge the frames and points of the deleted annotations in the remaining one. The annotation labels are merged, too. If a deleted annotation had a label (by a user) that the remaining annotation does not have, it is changed to point to the remaining annotation.

Object tracking for remote videos

Extend the image cache to a more generic "file cache" (?) that can cache any kind of file. Use this for videos when an object tracking is requested so the tracking works for any video and not only locally stored ones.

Improve the timeline

Implement zoom and pan in the timeline. This should include different styles with different "densities" for the annotation clip. If an annotation clip becomes too short, the 9px wide keyframe elements can shrink to 3px width. If this is still too big, they can vanish completely.

Also improve the progress display at the top of the timeline. It should show ticks at regular intervals and the time on mouseover.

DOI link is not shown

The DOI link button in the navbar is not shown if the video has a DOI.

Improve keyframe sampling of object tracking

The fixed distance in seconds between two object tracking keyframes does not work well for some videos. If there are fast movements, the annotation moves too quickly or too slowly between keyframes. Implement a dynamic keyframe sampling that is based of offset deltas. Based on a fixed offset delta x, a new keyframe is sampled if the old keyframe position is more than x pixels away from the new keyframe or the tracking has been finished.

Edit video tab

Add an "edit video" tab to the sidebar of the video annotation tool if the user is admin of the project. Usersh should be able to update the name, url etc. there.

Object tracking

I implemented a very basic proof of concept object tracking with the mean shift algorithm. It can be found in the object-tracking branch. For production we probably want to use a better algorithm.

Mean shift runs with OpenCV which can be installed in the alpine worker container like this:

ARG OPENCV_VERSION=3.4.5
RUN apk add --no-cache python clang ffmpeg ffmpeg-libs \
    && apk add --no-cache --virtual .build-deps \
        gcc g++ build-base cmake clang-dev python-dev ffmpeg-dev linux-headers \
    && cd /tmp \
    && curl -L https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip -o ${OPENCV_VERSION}.zip \
    && unzip -q ${OPENCV_VERSION}.zip \
    && curl -L https://github.com/opencv/opencv_contrib/archive/${OPENCV_VERSION}.zip -o ${OPENCV_VERSION}.zip \
    && unzip -q ${OPENCV_VERSION}.zip \
    && cd /tmp/opencv-${OPENCV_VERSION} \
    && mkdir build \
    && cd build \
    && cmake \
        -D CMAKE_BUILD_TYPE=RELEASE \
        -D BUILD_TESTS=OFF \
        -D BUILD_PERF_TESTS=OFF \
        -D BUILD_EXAMPLES=OFF \
        -D INSTALL_PYTHON_EXAMPLES=OFF \
        -D INSTALL_C_EXAMPLES=OFF \
        -D BUILD_DOCS=OFF \
        -D WITH_WIN32UI=OFF \
        -D WITH_QT=OFF \
        -D OPENCV_EXTRA_MODULES_PATH=/tmp/opencv_contrib-${OPENCV_VERSION}/modules \
        .. \
    && make -j $(nproc) \
    && make install \
    && ln -s /usr/local/lib/python2.7/site-packages/cv2 \
        /usr/lib/python2.7/site-packages/cv2 \
    && apk del --purge .build-deps \
    && rm -r /tmp/*

Improve the error message if the video cannot be decoded

If the video codec cannot be read by the browser, it should say so in the error message. ~~This error message should be displayed in the middle of the video screen instead of the smaller user feedback bessage.~~

Try to implement the polygon brush

The polygon brush interaction has been implemented in our OpenLayers fork. Try to add it to the video annotation tool.

Test how the polygon interpolation handles the complex polygons created by the brush
Implement the PolygonBrush interaction
Implement the ModifyPolygonBrush interaction
Update the manual

Error while loading video file

If the page is unloaded, the error message "Error while loading video file" is sometimes shown. See if this can be hidden.

Video validation is not working correctly

An MPG video could be created but did not pass some validation during processing:

[2020-03-04 01:16:10] production.WARNING: Could not process new video 1448: Error while caching file 'https://xxxx.mpg': MIME type 'image/x-tga' not allowed.

Also video 1449 is a MOV which cannot be read by the object tracking script:

[2020-03-04 01:47:28] production.ERROR: Error while executing Python script with '/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py /tmp/object_tracking_input_74321.json /tmp/object_tracking_output_74321.json':
Traceback (most recent call last):
  File "/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py", line 82, in <module>
    for keyframe in ObjectTracker(params):
  File "/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py", line 20, in __init__
    raise IOError('The video file could not be read: {}'.format(params['video_path']))
IOError: The video file could not be read: /var/www/storage/framework/cache/files/8af81c15d12ca942d2070d1cae4e725fd4bc7198b00b780b876a030f6a932acb {"exception":"[object] (Exception(code: 0): Error while executing Python script with '/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py /tmp/object_tracking_input_74321.json /tmp/object_tracking_output_74321.json':
Traceback (most recent call last):
  File \"/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py\", line 82, in <module>
    for keyframe in ObjectTracker(params):
  File \"/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py\", line 20, in __init__
    raise IOError('The video file could not be read: {}'.format(params['video_path']))
IOError: The video file could not be read: /var/www/storage/framework/cache/files/8af81c15d12ca942d2070d1cae4e725fd4bc7198b00b780b876a030f6a932acb at /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php:188)
[stacktrace]
#0 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(97): Biigle\\Modules\\Videos\\Jobs\\TrackObject->python('/var/www/vendor...')
#1 [internal function]: Biigle\\Modules\\Videos\\Jobs\\TrackObject->Biigle\\Modules\\Videos\\Jobs\\{closure}(Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#2 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(79): call_user_func(Object(Closure), Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#3 [internal function]: Biigle\\FileCache\\FileCache->Biigle\\FileCache\\{closure}(Array, Array)
#4 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(139): call_user_func(Object(Closure), Array, Array)
#5 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(80): Biigle\\FileCache\\FileCache->batch(Array, Object(Closure))
#6 /var/www/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(245): Biigle\\FileCache\\FileCache->get(Object(Biigle\\Modules\\Videos\\Video), Object(Closure))
#7 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(110): Illuminate\\Support\\Facades\\Facade::__callStatic('get', Array)
#8 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(47): Biigle\\Modules\\Videos\\Jobs\\TrackObject->getTrackingKeyframes(Object(Biigle\\Modules\\Videos\\VideoAnnotation))
#9 [internal function]: Biigle\\Modules\\Videos\\Jobs\\TrackObject->handle()
#10 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#11 /var/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(34): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#12 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#13 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#14 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#15 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\\Container\\Container->call(Array)
#16 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#17 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#18 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#19 /var/www/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(83): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject), false)
#20 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#21 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#22 /var/www/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(85): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#23 /var/www/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(59): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#24 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(88): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\RedisJob), Array)
#25 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(348): Illuminate\\Queue\\Jobs\\Job->fire()
#26 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(294): Illuminate\\Queue\\Worker->process('redis', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions))
#27 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(129): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), 'redis', Object(Illuminate\\Queue\\WorkerOptions))
#28 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(112): Illuminate\\Queue\\Worker->daemon('redis', 'high,default', Object(Illuminate\\Queue\\WorkerOptions))
#29 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(96): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis', 'high,default')
#30 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#31 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#32 /var/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(34): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#33 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#34 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#35 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#36 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(202): Illuminate\\Container\\Container->call(Array)
#37 /var/www/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#38 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(189): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#39 /var/www/vendor/symfony/console/Application.php(1011): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#40 /var/www/vendor/symfony/console/Application.php(272): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#41 /var/www/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#42 /var/www/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#43 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(131): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#44 /var/www/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#45 {main}
"}

Fix the validation of the create video endpoint so only valid video formats are allowed!

Invisible tracking animation

If a label has a dark color, the striped tracking animation of a keyframe element is invisible on the dark background. Maybe use a white "layer" behind the annotation keyframe element so the animation is still visible.

Restrict height of videos panel

The videos panel in the project overview has no max-height like the volumes panel. Reuse the volumes-grid class here.

How to display an annotation with multiple labels in the timeline

If an annotation has multiple labels attached in the still image annotation tool it looks no different to other annotations in the image (although we thought about changing this). Each attached label and user is shown in the annotation list in the sidebar, though.

This is the same in the video annotation tool. But how should the annotation be displayed in the video timeline at the bottom? Let's say the annotation has labels A and B attached by user 1 and label A attached by user 2. Should the annotation be shown twice in the track for label A and once in the track for label B? Or just once in each track? I think the former would be more consistent with our definition of such an annotation as "multiple markings on the same location with the same shape" but the latter would be more compact. @dlangenk any thoughts?

Video search

Add a videos tab to the global search page.

Implement all annotation shapes with draw and edit interactions

Implement draw interactions for the following shapes:

Also implement:

Maybe exclude the polygon if the draw/edit interaction can't be implemented in an easy to use way. We have to decide if we want to allow a variable number of polygon points for the same annotation clip or not. Interpolating between polygon shapes with a variable number of points might be hard.

The regular edit interaction might work like this: The interpolated feature is drawn and selected while the video plays. If the feature is moved or modified, this creates an new keyframe for the annotation clip.

Implement minimap

Implement a minimap that works like the one of the still image annotation tool.

Video panel filter

Similar to biigle/projects#27, create a video panel filter typeahead.

Implement variable playback speeds

Offer 0.5x, 1x, 1.5x and 2x as playback speeds. Or just add a number input with these four speeds as quick select buttons to the settings (#11).

Annotation list in sidebar

Implement an annotation list (like in the still image annotation (SIA) tool) in the sidebar. This list is required to detach individual labels from annotations that have multiple labels. Also it might be a nice overview for the whole video and we can implement an annotation filter, too.

Implement list of annotations / annotation labels grouped by label like in the SIA tool
Implement detaching of annotation labels from the annotations list
Implement an annotation filtering like in the SIA tool
~~Implement a toggle "only visible" that shows only currently visible annotations in the list~~
Extend the manual article on the sidebar

Show single frame annotations at the end of the timeline

If a single frame annotation is created at the end of the video (timeline), it is invisible in the timeline because it overflows the timeline element at the right. Find a solution for this. Maybe determine the x-position of the keyframe element of a single frame annotation as:

min(desired_x, timeline_width - keyframe_element_width)

Error while extracting video codec

There is no error handling during extraction of the video codec (here). This can happen if the video is malformed. Implement proper error handling and reject invalid videos.

[2020-04-21 08:47:57] production.ERROR: Unable to probe xxx {"userId":1,"exception":"[object] (FFMpeg\\Exception\\RuntimeException(code: 0): Unable to probe xxx at /var/www/vendor/php-ffmpeg/php-ffmpeg/src/FFMpeg/FFProbe.php:263, Alchemy\\BinaryDriver\\Exception\\ExecutionFailureException(code: 0): ffprobe failed to execute command '/usr/bin/ffprobe' 'xxx' '-show_streams' '-print_format' 'json':

Error Output:

 ffprobe version 4.2.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 9.2.0 (Alpine 9.2.0)
  configuration: --prefix=/usr --enable-avresample --enable-avfilter --enable-gnutls --enable-gpl --enable-libass --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-libxcb --disable-stripping --disable-static --disable-librtmp --enable-vaapi --enable-vdpau --enable-libopus --disable-debug
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f4d43e90740] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f4d43e90740] moov atom not found
xxx: Invalid data found when processing input
 at /var/www/vendor/alchemy/binary-driver/src/Alchemy/BinaryDriver/ProcessRunner.php:95)
[stacktrace]
#0 /var/www/vendor/php-ffmpeg/php-ffmpeg/src/FFMpeg/FFProbe.php(206): FFMpeg\\FFProbe->probe('https://opensta...', '-show_streams', 'streams')
#1 /var/www/vendor/biigle/videos/src/Support/VideoCodecExtractor.php(29): FFMpeg\\FFProbe->streams('https://opensta...')
#2 /var/www/vendor/biigle/videos/src/Rules/VideoUrl.php(187): Biigle\\Modules\\Videos\\Support\\VideoCodecExtractor->extract('https://opensta...')
#3 /var/www/vendor/biigle/videos/src/Rules/VideoUrl.php(157): Biigle\\Modules\\Videos\\Rules\\VideoUrl->passesCodec('https://opensta...')
#4 /var/www/vendor/biigle/videos/src/Rules/VideoUrl.php(69): Biigle\\Modules\\Videos\\Rules\\VideoUrl->passesRemoteUrl('https://opensta...')
#5 /var/www/vendor/laravel/framework/src/Illuminate/Validation/Validator.php(635): Biigle\\Modules\\Videos\\Rules\\VideoUrl->passes('url', 'https://opensta...')
#6 /var/www/vendor/laravel/framework/src/Illuminate/Validation/Validator.php(451): Illuminate\\Validation\\Validator->validateUsingCustomRule('url', 'https://opensta...', Object(Biigle\\Modules\\Videos\\Rules\\VideoUrl))
#7 /var/www/vendor/laravel/framework/src/Illuminate/Validation/Validator.php(300): Illuminate\\Validation\\Validator->validateAttribute('url', Object(Biigle\\Modules\\Videos\\Rules\\VideoUrl))
#8 /var/www/vendor/laravel/framework/src/Illuminate/Validation/Validator.php(331): Illuminate\\Validation\\Validator->passes()
#9 /var/www/vendor/laravel/framework/src/Illuminate/Validation/ValidatesWhenResolvedTrait.php(25): Illuminate\\Validation\\Validator->fails()
#10 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Providers/FormRequestServiceProvider.php(30): Illuminate\\Foundation\\Http\\FormRequest->validateResolved()
#11 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(1102): Illuminate\\Foundation\\Providers\\FormRequestServiceProvider->Illuminate\\Foundation\\Providers\\{closure}(Object(Biigle\\Modules\\Videos\\Http\\Requests\\StoreVideo), Object(Illuminate\\Foundation\\Application))
#12 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(1066): Illuminate\\Container\\Container->fireCallbackArray(Object(Biigle\\Modules\\Videos\\Http\\Requests\\StoreVideo), Array)
#13 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(1051): Illuminate\\Container\\Container->fireAfterResolvingCallbacks('Biigle\\\\Modules\\\\...', Object(Biigle\\Modules\\Videos\\Http\\Requests\\StoreVideo))
#14 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(701): Illuminate\\Container\\Container->fireResolvingCallbacks('Biigle\\\\Modules\\\\...', Object(Biigle\\Modules\\Videos\\Http\\Requests\\StoreVideo))
#15 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(785): Illuminate\\Container\\Container->resolve('Biigle\\\\Modules\\\\...', Array, true)
#16 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(629): Illuminate\\Foundation\\Application->resolve('Biigle\\\\Modules\\\\...', Array)
#17 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(770): Illuminate\\Container\\Container->make('Biigle\\\\Modules\\\\...', Array)
#18 /var/www/vendor/laravel/framework/src/Illuminate/Routing/RouteDependencyResolverTrait.php(79): Illuminate\\Foundation\\Application->make('Biigle\\\\Modules\\\\...')
#19 /var/www/vendor/laravel/framework/src/Illuminate/Routing/RouteDependencyResolverTrait.php(46): Illuminate\\Routing\\ControllerDispatcher->transformDependency(Object(ReflectionParameter), Array)
#20 /var/www/vendor/laravel/framework/src/Illuminate/Routing/RouteDependencyResolverTrait.php(27): Illuminate\\Routing\\ControllerDispatcher->resolveMethodDependencies(Array, Object(ReflectionMethod))
#21 /var/www/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(41): Illuminate\\Routing\\ControllerDispatcher->resolveClassMethodDependencies(Array, Object(Biigle\\Modules\\Videos\\Http\\Controllers\\Api\\ProjectVideoController), 'store')
#22 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Route.php(219): Illuminate\\Routing\\ControllerDispatcher->dispatch(Object(Illuminate\\Routing\\Route), Object(Biigle\\Modules\\Videos\\Http\\Controllers\\Api\\ProjectVideoController), 'store')
#23 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Route.php(176): Illuminate\\Routing\\Route->runController()
#24 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Router.php(681): Illuminate\\Routing\\Route->run()
#25 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Routing\\Router->Illuminate\\Routing\\{closure}(Object(Illuminate\\Http\\Request))
#26 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(41): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#27 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#28 /var/www/vendor/laravel/framework/src/Illuminate/Auth/Middleware/Authenticate.php(43): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#29 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Auth\\Middleware\\Authenticate->handle(Object(Illuminate\\Http\\Request), Object(Closure), 'web', 'api')
#30 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(76): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#31 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\VerifyCsrfToken->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#32 /var/www/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#33 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#34 /var/www/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(56): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#35 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Session\\Middleware\\StartSession->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#36 /var/www/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#37 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#38 /var/www/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(66): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#39 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#40 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#41 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Router.php(683): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#42 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Router.php(658): Illuminate\\Routing\\Router->runRouteWithinStack(Object(Illuminate\\Routing\\Route), Object(Illuminate\\Http\\Request))
#43 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Router.php(624): Illuminate\\Routing\\Router->runRoute(Object(Illuminate\\Http\\Request), Object(Illuminate\\Routing\\Route))
#44 /var/www/vendor/laravel/framework/src/Illuminate/Routing/Router.php(613): Illuminate\\Routing\\Router->dispatchToRoute(Object(Illuminate\\Http\\Request))
#45 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(170): Illuminate\\Routing\\Router->dispatch(Object(Illuminate\\Http\\Request))
#46 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}(Object(Illuminate\\Http\\Request))
#47 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#48 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#49 /var/www/app/Http/Middleware/UpdateUserActivity.php(28): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#50 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Biigle\\Http\\Middleware\\UpdateUserActivity->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#51 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#52 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#53 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#54 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#55 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/CheckForMaintenanceMode.php(63): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#56 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\CheckForMaintenanceMode->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#57 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#58 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(145): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#59 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(110): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter(Object(Illuminate\\Http\\Request))
#60 /var/www/public/index.php(55): Illuminate\\Foundation\\Http\\Kernel->handle(Object(Illuminate\\Http\\Request))
#61 {main}
"}

Attach label interaction

Implement the attach label interaction like in the still image annotation tool. Extend the manual on the edit interactions.

Admin area statistics

Add a panel to the admin area dashboard that shows the number of video annotations. The total hours of video material would be interesting, too, but we don't store that yet. We could add this once ffmpeg is added to create video thumbnails (#26) and maybe handle video streams (#7) in the future.

Frame accurate seeking

Apparently there are some issues with HTML media elements that make it impossible to perform frame accurate seeking. One of the root causes seems to be that an accurate calculation of the frame rate is not straight forward.

This could affect users of the video annotation tool who want to determine the number of the frame that belongs to a particular annotation (frame). I'm not sure how serious this issue would be, though.

Integrate in BIIGLE

Integrate the prototype in BIIGLE if possible. We could use the media type of a volume to decide if BIIGLE should switch to "image annotation mode" or to "video annotation mode". The critical question is if the BIIGLE database architecture can accomodate the requirements of video annotation.

Remote videos

As with remote volumes, allow remote videos:

Allow http and https URLs, check for existence when a video is created
Redirect to the remote URL when the video file is requested
Write a manual article on remote videos

Link to supported video formats

Add a link to the browser support table of video formats somewhere. Maybe in the manual. The content type matching of BIIGLE should take care of most incompatibility issues but there are some unsupported codecs that are also called MP4 (e.g. H.263).

Delete videos

Implement the UI to delete videos (from projects).

Video reports

Users should be able to download video annotations in reports. I'm not sure how to handle this, yet. On the one hand, video annotations are quite different from still image annotations, which might require a whole new logic for video annotation reports. This could be implemented directly in this module. On the other hand, videos are associated with projects and therefore should be included in project reports, too. Maybe we can add a new tab "Video annotation report" to the existing "Annotation report" and "Image label report"? This would require the new functions to be implemented in the reports module.

Implement configurable queue for object tracking jobs

Use the high queue by default but allow configuration.

Incorrect item count in search page

The video pagination of the search page shows not enough items. It shows ten but should show twelve.

Open label trees tab on "select a label" message

As with the image annotation tool, automatically open the label trees tab if the user selects an annotation tool but has not selected a label.

Implement a prototype

Implement a prototype of the video annotation tool based on the requirements determined in #1. Find and use an appropriate object tracking method that automatically determines the future positions of an initial annotation in subsequent video frames (if we agree that an annotation may span multiple frames). Iterate the prototype based on feedback of the MI2 participants.

Project thumbnails from videos

Currently the project thumbnail in the global search is only taken from the volumes of the project. If there are no volumes, it should be taken from a video instead. The same applies to the dashboard preview of the projects. The preview should contain video thumbnails if not enough volumes exist to fill the preview.

Sync video frame on pause

It seems like the displayed video frame can be out of sync. For some videos with a fast moving camera (e.g. 1258) some single-frame annotations do not match the positons of the OOI. Make sure to sync the displayed video frame if the video is paused (render sync after canceling the animation frame).

No video is shown when URL time parameter is used in Chrome

e.g. https://biigle.de/videos/102?x=658&y=628&r=146&t=4 no video is shown.

Video timeline bottom shadow does not disappear

The video timeline bottom shadow indicates that there is still something to scroll down. If the timeline is scrolled down completely, the shadow should disappear. In some cases, it doesn't (see video 113). Maybe something causes the scroll position to be off.

Linking of a single frame annotation with multiple labels

When I attempt to link two single frame annotations that occur in the same frame and who were created by different users (with the same attached label) there are two problems:

I shouldn't be able to link two single frame annotations that occur at the same time. This should be the same than overlapping annotation clips.
The resulting linked annotation has four annotation labels instead of two. At least four items are shown in the annotations tab.

Error in object tracking script

Implement error handling for this:

[2019-02-25 22:14:17] production.ERROR: Error while executing Python script with '/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py /tmp/object_tracking_input_16.json /tmp/object_tracking_output_16.json':
Traceback (most recent call last):
  File "/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py", line 86, in <module>
    if keyframes[-1][0] != last_keyframe[0]:
IndexError: list index out of range {"exception":"[object] (Exception(code: 0): Error while executing Python script with '/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py /tmp/object_tracking_input_16.json /tmp/object_tracking_output_16.json':
Traceback (most recent call last):
  File \"/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py\", line 86, in <module>
    if keyframes[-1][0] != last_keyframe[0]:
IndexError: list index out of range at /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php:183)
[stacktrace]
#0 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(92): Biigle\\Modules\\Videos\\Jobs\\TrackObject->python('/var/www/vendor...')
#1 [internal function]: Biigle\\Modules\\Videos\\Jobs\\TrackObject->Biigle\\Modules\\Videos\\Jobs\\{closure}(Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#2 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(62): call_user_func(Object(Closure), Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#3 [internal function]: Biigle\\FileCache\\FileCache->Biigle\\FileCache\\{closure}(Array, Array)
#4 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(122): call_user_func(Object(Closure), Array, Array)
#5 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(63): Biigle\\FileCache\\FileCache->batch(Array, Object(Closure))
#6 /var/www/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Biigle\\FileCache\\FileCache->get(Object(Biigle\\Modules\\Videos\\Video), Object(Closure))
#7 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(105): Illuminate\\Support\\Facades\\Facade::__callStatic('get', Array)
#8 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(48): Biigle\\Modules\\Videos\\Jobs\\TrackObject->getTrackingKeyframes(Object(Biigle\\Modules\\Videos\\VideoAnnotation))
#9 [internal function]: Biigle\\Modules\\Videos\\Jobs\\TrackObject->handle()
#10 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#11 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#12 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#13 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#14 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\\Container\\Container->call(Array)
#15 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#16 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#17 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#18 /var/www/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(49): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject), false)
#19 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(76): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\RedisJob), Array)
#20 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(320): Illuminate\\Queue\\Jobs\\Job->fire()
#21 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(270): Illuminate\\Queue\\Worker->process('redis', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions))
#22 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(114): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), 'redis', Object(Illuminate\\Queue\\WorkerOptions))
#23 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Worker->daemon('redis', 'high,default', Object(Illuminate\\Queue\\WorkerOptions))
#24 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis', 'high,default')
#25 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#26 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#27 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#28 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#29 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#30 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#31 /var/www/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#32 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#33 /var/www/vendor/symfony/console/Application.php(953): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 /var/www/vendor/symfony/console/Application.php(248): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#35 /var/www/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#36 /var/www/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#37 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#38 /var/www/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#39 {main}
"}

Error when attaching duplicate label

If a user attempts to attach a label to an annotation that already exists (by them), it prints uncaught exception: Object in the console.

Adjustable timeline height

Implement an adjustable height for the video timeline. Users should be able to grab the upper border of the timeline and push it down until it only shows the current time of the video. Store this setting in local storage and apply it automatically whenever the video annotation tool is reopened.

Create a specification for video annotation

Examine existing tools for video annotation and compile a collection of their features. Then organize a workshop with MI2 participants to agree on a specification an feature requirement for a video annotation tool. Important questions are:

Is an annotation only valid for a single video frame? Or is it valid for a single "object" across multiple frames (i.e. can it move)?
...

Error in object tracking script

[2019-09-04 12:02:24] production.ERROR: Error while executing Python script with '/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py /tmp/object_tracking_input_28074.json /tmp/object_tracking_output_28074.json':
Traceback (most recent call last):
  File "/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py", line 82, in <module>
    for keyframe in ObjectTracker(params):
  File "/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py", line 22, in __init__
    self.tracker.init(frame, track_window)
cv2.error: OpenCV(3.4.5) /tmp/opencv-3.4.5/modules/core/src/dxt.cpp:3335: error: (-215:Assertion failed) type == CV_32FC1 || type == CV_32FC2 || type == CV_64FC1 || type == CV_64FC2 in function 'dft'
 {"exception":"[object] (Exception(code: 0): Error while executing Python script with '/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py /tmp/object_tracking_input_28074.json /tmp/object_tracking_output_28074.json':
Traceback (most recent call last):
  File \"/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py\", line 82, in <module>
    for keyframe in ObjectTracker(params):
  File \"/var/www/vendor/biigle/videos/src/config/../resources/scripts/ObjectTracker.py\", line 22, in __init__
    self.tracker.init(frame, track_window)
cv2.error: OpenCV(3.4.5) /tmp/opencv-3.4.5/modules/core/src/dxt.cpp:3335: error: (-215:Assertion failed) type == CV_32FC1 || type == CV_32FC2 || type == CV_64FC1 || type == CV_64FC2 in function 'dft'
 at /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php:188)
[stacktrace]
#0 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(97): Biigle\\Modules\\Videos\\Jobs\\TrackObject->python('/var/www/vendor...')
#1 [internal function]: Biigle\\Modules\\Videos\\Jobs\\TrackObject->Biigle\\Modules\\Videos\\Jobs\\{closure}(Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#2 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(62): call_user_func(Object(Closure), Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#3 [internal function]: Biigle\\FileCache\\FileCache->Biigle\\FileCache\\{closure}(Array, Array)
#4 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(122): call_user_func(Object(Closure), Array, Array)
#5 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(63): Biigle\\FileCache\\FileCache->batch(Array, Object(Closure))
#6 /var/www/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Biigle\\FileCache\\FileCache->get(Object(Biigle\\Modules\\Videos\\Video), Object(Closure))
#7 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(110): Illuminate\\Support\\Facades\\Facade::__callStatic('get', Array)
#8 /var/www/vendor/biigle/videos/src/Jobs/TrackObject.php(47): Biigle\\Modules\\Videos\\Jobs\\TrackObject->getTrackingKeyframes(Object(Biigle\\Modules\\Videos\\VideoAnnotation))
#9 [internal function]: Biigle\\Modules\\Videos\\Jobs\\TrackObject->handle()
#10 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#11 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#12 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#13 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#14 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\\Container\\Container->call(Array)
#15 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#16 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject))
#17 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#18 /var/www/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(49): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(Biigle\\Modules\\Videos\\Jobs\\TrackObject), false)
#19 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(76): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\RedisJob), Array)
#20 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(320): Illuminate\\Queue\\Jobs\\Job->fire()
#21 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(270): Illuminate\\Queue\\Worker->process('redis', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions))
#22 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(114): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), 'redis', Object(Illuminate\\Queue\\WorkerOptions))
#23 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Worker->daemon('redis', 'high,default', Object(Illuminate\\Queue\\WorkerOptions))
#24 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis', 'high,default')
#25 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#26 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#27 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#28 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#29 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#30 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#31 /var/www/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#32 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#33 /var/www/vendor/symfony/console/Application.php(969): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 /var/www/vendor/symfony/console/Application.php(255): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#35 /var/www/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#36 /var/www/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#37 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#38 /var/www/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#39 {main}
"}

Implement settings

Implement a new tab with these settings:

Slider for the annotation opacity.
Switch for autoplay while drawing a new annotation. Switched off by default. Should have a configurable play/pause interval.
Switch for a label tooltip.
Switch for a mouse position indicator.
Switch for the minimap (#10).

Settings are stored in local storage like in the still image annotation tool. Maybe a generic "settings service" can be implemented, which handles local storage and url parameters and can be used for all tools that have settings like these in BIIGLE.

Create manual articles

Create extensive manual articles on video annotation:

Getting started (explain UI and how to create the first annotation)
Creating video annotations (including single-/multi-frame annotations, automatic object tracking)
Navigating the annotation timeline (explain annotation clips, keyframes etc)
Editing video annotations (including editing/deleting of keyframes, linking and splitting of annotations)
Sidebar
Shortcuts
URL Parameters

Video thumbnails

Implement video thumbnails similar to the volume thumbnails. The thumbnail element should show several frames on mousemove, which are extracted at even intervals from the video. This needs ffmpeg (or similar) support in the backend so the video files can be processed.

Error when generating video thumbnails

Genetraing of video thumbnails for the existing four videos on biigle.de produced this error (for three of them):

[2019-03-11 14:33:31] production.ERROR: VipsForeignLoad: "/tmp/video-thumb-dOoIik" is not a known file format
 {"exception":"[object] (Jcupitt\\Vips\\Exception(code: 0): VipsForeignLoad: \"/tmp/video-thumb-dOoIik\" is not a known file format
 at /var/www/vendor/jcupitt/vips/src/Image.php:777)
[stacktrace]
#0 /var/www/vendor/jcupitt/vips/src/Image.php(800): Jcupitt\\Vips\\Image::errorVips()
#1 /var/www/vendor/jcupitt/vips/src/Image.php(1412): Jcupitt\\Vips\\Image::errorIsArray(-1)
#2 /var/www/vendor/jcupitt/vips/src/Image.php(1496): Jcupitt\\Vips\\Image::callBase('thumbnail', Object(Jcupitt\\Vips\\Image), Array)
#3 /var/www/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Jcupitt\\Vips\\Image->__call('thumbnail', Array)
#4 /var/www/vendor/biigle/videos/src/Jobs/ProcessNewVideo.php(106): Illuminate\\Support\\Facades\\Facade::__callStatic('thumbnail', Array)
#5 /var/www/vendor/biigle/videos/src/Jobs/ProcessNewVideo.php(65): Biigle\\Modules\\Videos\\Jobs\\ProcessNewVideo->generateVideoThumbnail('/var/www/storag...', 30, 180, 135, 'jpg')
#6 [internal function]: Biigle\\Modules\\Videos\\Jobs\\ProcessNewVideo->Biigle\\Modules\\Videos\\Jobs\\{closure}(Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#7 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(72): call_user_func(Object(Closure), Object(Biigle\\Modules\\Videos\\Video), '/var/www/storag...')
#8 [internal function]: Biigle\\FileCache\\FileCache->Biigle\\FileCache\\{closure}(Array, Array)
#9 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(146): call_user_func(Object(Closure), Array, Array)
#10 /var/www/vendor/biigle/laravel-file-cache/src/FileCache.php(73): Biigle\\FileCache\\FileCache->batchOnce(Array, Object(Closure))
#11 /var/www/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Biigle\\FileCache\\FileCache->getOnce(Object(Biigle\\Modules\\Videos\\Video), Object(Closure))
#12 /var/www/vendor/biigle/videos/src/Jobs/ProcessNewVideo.php(68): Illuminate\\Support\\Facades\\Facade::__callStatic('getOnce', Array)
#13 [internal function]: Biigle\\Modules\\Videos\\Jobs\\ProcessNewVideo->handle()
#14 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#15 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#16 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#17 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#18 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\\Container\\Container->call(Array)
#19 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\ProcessNewVideo))
#20 /var/www/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Biigle\\Modules\\Videos\\Jobs\\ProcessNewVideo))
#21 /var/www/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#22 /var/www/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(49): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(Biigle\\Modules\\Videos\\Jobs\\ProcessNewVideo), false)
#23 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(76): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\RedisJob), Array)
#24 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(320): Illuminate\\Queue\\Jobs\\Job->fire()
#25 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(270): Illuminate\\Queue\\Worker->process('redis', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions))
#26 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(114): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), 'redis', Object(Illuminate\\Queue\\WorkerOptions))
#27 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Worker->daemon('redis', 'high,default', Object(Illuminate\\Queue\\WorkerOptions))
#28 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis', 'high,default')
#29 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#30 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#31 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#32 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#33 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#34 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#35 /var/www/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#36 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#37 /var/www/vendor/symfony/console/Application.php(953): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#38 /var/www/vendor/symfony/console/Application.php(248): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#39 /var/www/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#40 /var/www/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#41 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#42 /var/www/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#43 {main}
"}

For all three videos, the last thumbnail wasn't generated. I think the error is related to that. I already tried to fix an issue with the last frame by rounding the seek time but apparently the fix does not work for all videos.

Statistics on user admin page

Add video and video annotation statistics to the user admin page. Show which and how many videos and how many video annotations were created by the user.

Implement URL parameters

Implement these URL parameters: x-position (x), y-position (y), resolution/zoom (r) and current time (t). Like in the still image annotation tool, these URL parameters update and are restored if they are initially present.