hcmlab / nova

NOVA is a tool for annotating and analyzing behaviours in social interactions. It supports Annotators using Machine Learning already during the coding process. Further it features both, discrete labels and continuous scores and a visuzalization of streams recorded with the SSI Framework.

NOVA is a tool for annotating and analyzing behaviours in social interactions. It allows to visualize data recorded with the SSI Framework, as well as from external sources. A main feature of NOVA is that it allows to employ a collaborative annotation database where annotation work can be split between multiple sides, but also between a human annotator and a machine by supporting human annotators with machine learning techniques already during the annotation process - A process we call Collaborative Machine Learning.

NOVA allows framewise labeling for a precise coding experience, and value-continuous annotations for labeling e.g emotions or social attitudes. The interface is customizable and allows loading and labeling data of multiple persons.

The Annotation format can easily be imported in other tools, like ELAN or Excel. NOVA further supports the Import of Discrete Annotation files from ELAN and ANVIL for a seamless workflow.

The Cooperative Machine Learning capabilities allow to train and evaluate machine learning models, such as Support Vector machines or Artificial neural networks directly from the interface with both, a session completion step, where a model is trained on the first minutes of an annotated sessions to predict the remaining session, and a session transfer step where a model is trained on multiple sessions to predict completly unknown data. With the help of human input the models can be refined. The collaborative ML workflow is illustrated below.

The latest binaries can always be found here.

FAQ:

Help, I opened a video but it doesn't show up

Please make sure you installed the according Video Codec on your System. E.g. Shark007 Codecs might be a good solution for most missing codecs, which enable you to control audio/video codecs with dedicated GUI. Also make sure you installed a Visual Studio 2015 redistribution package if it doesn't come with your system.

I don't know what to do, where can I get help?

The offical documentation can be found here. If you run into trouble, please create an issue on git.

Will NOVA run on my Mac/Linux Machine?

As NOVA is an WPF Application it will run on Windows. Nevertheless you can of course use a Virtual Machine to use it.

I found a bug, can you fix it?

NOVA is Software under development and is provided “as is”. If you run into any problems or find bugs (or want to contribute to the project) feel free to open an issue here on github.

Is there an example pipeline to automatically create annotations?

check out https://github.com/hcmlab/kinect2-record for a kinect 2 example recording pipeline.

I would like to contribute to the project

Please feel free to fork or create an issue

Publications:

If you are using NOVA in your research please consider giving us a citation:

"eXplainable Cooperative Machine Learning with NOVA", Tobias Baur, Alexander Heimerl, Florian Lingenfelser, Johannes Wagner, Michel F. Valstar, Björn Schuller, Elisabeth André, in Ki - Künstliche Intelligenz, 2020

@Article{Baur2020,
author="Baur, Tobias
and Heimerl, Alexander
and Lingenfelser, Florian
and Wagner, Johannes
and Valstar, Michel F.
and Schuller, Bj{\"o}rn
and Andr{\'e}, Elisabeth",
title="eXplainable Cooperative Machine Learning with NOVA",
journal="KI - K{\"u}nstliche Intelligenz",
year="2020",
month="Jan",
day="19",
issn="1610-1987",
doi="10.1007/s13218-020-00632-3",
url="https://doi.org/10.1007/s13218-020-00632-3"
}

“NOVA - A tool for eXplainable Cooperative Machine Learning”, Alexander Heimerl, Tobias Baur, Florian Lingenfelser, Johannes Wagner, Elisabeth André, in Proceedings of the 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), Cambridge, September 2019

@INPROCEEDINGS{Heimerl2019,
author="Heimerl, Alexander
and Baur, Tobias
and Lingenfelser, Florian
and Wagner, Johannes
and Andr{\'e}, Elisabeth",
booktitle={2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)},
title={NOVA - A tool for eXplainable Cooperative Machine Learning},
year={2019},
pages={109-115},
doi={10.1109/ACII.2019.8925519},
ISSN={2156-8103},
month={Sep.}
}
  

nova's Issues

Database ToDos

As long as Database Todos are not fullfilled, current updates might change the database design and make existing databases unusable. For now this feature is only meant for testing and helping the development process.

• Create and manage a collection for Annotators (Right now Annotators are identified by their Database Login credentials)
• Add possibility to edit AnnotationSchemes (instead of delete and recreate)
• AutoCreate AnnottionSchemes based on added tracks?
• Add used Media to Annotations
• Add DB Meta Collection
• Add ftp /httpPost (untested)
• Add Fields to Subjects
• Add Clean Database Function (delete all unreferred objects)
• Add Database Functions for calculating gold standard, rater agreement

Undo functionality

A general ctrl-z undo function would be great to have.

FormatException Error while loading CSV files

Hi,

I am getting this error after I select the format and frequency. My csv file is formatted as:
1536932595;-0.979; -0.177; 0.060; -1.678; 3.181; 7.851; 0.982; -0.158; 0.010; 1.495; -0.801; 5.310
The first value is unix timestamp.
Even if I remove the first values, it still gives me the same error.

I don't know if this causes the issue, but my sampling frequency varies from 50 to 60.

I installed the Windows version on Windows 10.

skeleton/face doesnt play without video

• if skeleton or face is dragged in without video source, it currently doesnt play alone.

UriFormatException when attempting to save a project created using the latest commit of the master branch

UriFormatException when attempting to save a project created using the latest commit of the master branch in

nova-master\Tools\FileTools.cs

line 66:

Uri pathUri = new Uri(filespec);

An unhandled exception of type 'System.UriFormatException' occurred in System.dll. Additional information: Invalid URI: The format of the URI could not be determined.

Geometric and Continuous table not updating

The tables for both geometric and continuous data do not update when clicking to move the video to another time in the tier window.

Redundant post-build event declared at the "nova" project

The following post-build event is found declared in the sources of the latest stable release of NOVA (0.9.9.4.6) :

xcopy /Y ..\bin\nova.exe ..\..\..\bin\x64\vc140

This command seems redundant to me. It also contains relative path to what seems to be a VS installation, which can vary from one computer to another. This is not critical, however should be addressed, or alternatively added to the documentation / readme.

Allow setting of Tier names

Tier names cannot currently be set or changed when clicking the gear icon of a selected tier.

i would like to make a new tier, and call it 'Manual Fear', of an annotation Scheme 'Binary Basic Emotion', with myself as annotator. Currently you can change the scheme and the annotator but not the tier name.

Feature for Automatic Peak Signal Navigation

For long videos with sparse signals, it can be difficult to quickly navigate to areas with peak activity. A navigation button that will allow you to skip to the next peak activity of a signal (say, the probability of the presence of a smile) would be really useful.

Question About Streams

Hello, I am a new user of nova,thank you for developing nova.

Actually, I'd like to ask question about the Streams of NOVA. After connected to local server, I have tried to add 3 Streams in "Manage Database", but when I try to "Load Session", I found those 3 Streams turns red, which means they are Missing Streams.

Therefore, I'd like to ask you to explain the specific function of Streams, and how to solve the problem of that Missing Streams.

I'm glad if you can answer my question. #

Close button, doesn't give option to cancel close

The close button for the main program does not give the option to cancel the close procedure and carry on with working. I know it currently asks to save open thing but you still have to load the program if it is accidentally closed.

projetfile format consistency

• filepath option is available for annotations, but the path for signals is stored in the tag. This should be consistent (and backward compatible)

Zoom doesnt follow selection in listview

when anno is not in zoom range, the range doesnt change

Deleting data should be centralized

Since the database only provides links to files on the local filesystem I need to delete data in two different places if I want to get rid of it - db and fs. Otherwise the data still persists as either the entry in the database (if it has just been deleted from the fs) or on the filesystem without any according entry in the db. (if deleted in the db).
It would be nice to be able to delete data directly from the NOVA-gui in both places.

use cml on continuous annotations

So far CML is only available for discrete labels

add possibility to sort on databasemainwindow

sorting does currently not work

Cross corpora training

It would be really nice to be able to train and / or evaluate over sessions from multiple corpora at once.

Database: File Download Protocols

• http Post [x]
• http Get [partly]
• FTP [ ]
• SFTP [x]

When adding Continues Teir always has BlueRed Background

Currently working on a fix, seems to be just that the max value is not set for the anno so defaulting.

Wave signal window can not be displayed on windows 7

On windows 7, when loading a wav file, the application crashes. Video appears to run fine, and I have not tested the other data types.

The relevant part of the actual exception which happens is the following:
{"Could not load file or assembly 'PresentationFramework.Aero2, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.":"PresentationFramework.Aero2, PublicKeyToken=31bf3856ad364e35"}

It appears you explicitly reference the Aero2 framework, which according to Microsoft (cf. https://connect.microsoft.com/VisualStudio/feedback/details/2634836/presentationframework-aero2-not-on-any-win-7) you should never do. You should reference the Aero framework, which will use whatever implementation is available on the guest OS (Aero on Windows 7, Aero2 on Windows 8+). Referencing explicitly forces the application to look like a windows 8 application, which looks ugly on windows 10 and crashes on windows 7.

It would be great to correct that so that Nova can run on windows 7!

Annotations timing is saved in a separate *.annotation~ file and not correctly read on project reload

Thanks for helping with the previous issues! Everything seems to be working now, however when I try to save the annotations I've made it saves the project file *.nova containing reference to the corresponding video file and the corresponding *.annotation file containing a discrete tier I've created. The actual timing of the annotation is however missing from both of these files and instead can be found at *.annotation~ file - I assume it's a bug because when I attempt to load the saved project back it gets the annotation timing wrong which suggests that they're saved or read in a wrong way.

Nova crashes when attempting to save project on Windows 10

Hi, when I try to save previously loaded project composed of a video and some annotations Nova crashes. Do you know what can possibly be causing this? Thanks.

Add default value for Nan in Continuous tracks

Need to implement a default value for continuous tier for parts where data is missing

Creating custom subsets of sessions

At the moment the training and evaluation process is inconvenient if you want to use different subsets of a corpus for each of the steps since the respective sessions need to be selected manually every time.
It would be really nice to be able to create custom sets for training, evaluation, development, etc....

CML problems when a class did not appear in the training

possible fix: ignore the class then for prediction, or dont undersample.

Facial Landmark Annotation

A nice addition would be to visualize and edit facial landmarks in a video. How exactly is up for discussion

Copying a label to a new tier does not take the annotation scheme into account

Copying a label by using the shortcut alt + arrow key up / down to another tier does not check the annotation type of the tier the label is copied to. Therefore it is possible to create discrete labels in a continous tier.

.csv External Signal Data Format

Hi. New user of NOVA here.

I have some accelerometer signals (in .csv format) I want to load in parallel with a video, to help the labeling process. How should my .csv files be formated? I cannot find info about it in the documentation.

Thanks!

Moving green cursor with 's' hotkey does not update start position of annotation

The 'w' hotkey inserts an annotation between the green and red cursor, but only if the green cursor has been places by the mouse. If the green cursor has been moved to a new location by pressing the 's' hotkey, the annotation is still generated between the red cursor and the last position where the green cursor was placed by the mouse. Not where the green cursor is currently displayed.

When the 's' hotkey is pressed, the program should also internally update update the position of the green cursor.

It's a small bug, but it makes it impossible to annotate without using the mouse.

Scaling of imported CSV Signals

Hi, many thanks for the awesome tool. I'm loading accelerometer data into NOVA. This works flawless, however I wanted to know if it is possible to set the Y-Scaling of the graph to constant range?

Many thanks

Frame by Frame

We would like an option to move frame by frame for video when using continuous data.
Will also be needed for geometric annotations.

Additionally to project files, create annotation-scheme files.

Project files now contain all the media/signals/annotations to be loaded for a session.
The difference would be that annotation-scheme files don't contain specific paths to media/signals, but only define which annotations are placed on which tiers, and what colours they have. This would be especially useful when labelling multiple sessions.

Train and Evaluate accross multiple corpora

It would be awesome to add the possibility to train a model over multiple sessions from different corpora.

Right click for lasso

Made a start on this but didn't get it working.

Interpolate between points in lasso to get bounding line
Area in bounding line should be classed set to current selected segment colour

Make order of loading and showing tiers consistent

I noticed that the order in which annotation tiers are displayed is (semi) random. We played with a number of videos that all had the six basic emotions predicted, but each time I loaded a different video by selecting all annotation files and the source video, the emotion prediction tiers were shown in a different order.

I did not experiment with loading the same video + annotations multiple times, to see if it was truly random.

It would be great if the order was consistent.

Add confidence for annotations

Add confidence values per label to model certainty/uncertainty

The latest stable release of NOVA (0.9.9.4.6) crashes when attempting to create a new label for a discrete tier

The latest stable release of NOVA (0.9.9.4.6) crashes when attempting to create a new label for a discrete tier. The crash occurs when loading a video file from local drive, then clicking on the "New Tier" button, selecting "New Discrete Tier" and clicking on the "+" button at the bottom of the window. Debugger points to line 119 in "nova-0.9.9.4.6\dll\LabelInputBox.xaml.cs":

SolidColorBrush b = AnnoTrack.GetSelectedTrack().BackgroundColor as SolidColorBrush;

The exception is NullReferenceException and it is described as follows:

An exception of type 'System.NullReferenceException' occurred in ssiview.dll but was not handled in user code. Additional information: Object reference not set to an instance of an object.

Feature extraction loops when it cant find a file

When the feature extraction can't find a file it will retry a few seconds later. This is problematic when calculating computational expensive features (e.g. openface) since Nova will be stuck in that loop till the problem has been resolved manually. This can result in an large loss of effectively used computation time unless the user is willing to check regularly if the pipeline is still running.
Any soft error like this should be nonblocking with enough information in the error log afterwards to see what happend.

deleted

deleted

Display annotation tier names on top of tier

When you have a number of similar annotation tiers opened (e.g. predictions of the six basic emotions) It is pretty hard to remember which is which. Currently you have to click on a tier to see its name. It would be great if the name (or a shortened name) could be displayed, say on the top-left hand side of each tier.

Binarised mask for continuous value tiers

It would be nice to be able to create a binarised version of a continuous valued annotation, by providing/setting a threshold. Either as a separate (linked) tier, or as a flip-switch of an existing tier, where you can flip between the continuous and binary view. This would be useful in particular for automatic predictions that output a likelihood, which would require some threshold set for decision making.

Multiple Geometric tiers displayed on single video

Michel has requested multiple geometric annotation to be displayed on a single video, to compare between ground truths and predicted.

When multiple discrete csv files are loaded, the tool tries to store them alltogether in the first one

The idea behind this was to have one file for all annotations. If multiple input files are loaded, this behaviour might not be wished.

Possible solutions:

• ask the user with a dialogue
• keep each track in a seperate file (might cause problems with ELAN import)

TimeoutException

When saving a new anno I get a TimeoutException:

An exception of type 'System.TimeoutException' occurred in MongoDB.Driver.Core.dll but was not handled in user code

Additional information: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = [] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "Unknown", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "127.0.0.1:27017" }", EndPoint: "127.0.0.1:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:27017

at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)

at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MongoDB.Driver.Core.Connections.TcpStreamFactory.d__7.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MongoDB.Driver.Core.Connections.TcpStreamFactory.d__4.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MongoDB.Driver.Core.Connections.BinaryConnection.d__48.MoveNext()

--- End of inner exception stack trace ---

at MongoDB.Driver.Core.Connections.BinaryConnection.d__48.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at MongoDB.Driver.Core.Servers.ServerMonitor.d__27.MoveNext()" }] }.

Ability to change colour of Cursor

The ability to change the colour of the cursor (red line), as if you have red tier cannot see.

My suggestions is to have multiple change colour options

1. change single colour
2. have a line with a border, therefore two colours, such as black border and white inner colour, meaning visible on all backgrounds.

General Compatibility to Databases

Additional to loading/storing files and annotations locally, a useful addition would be to access files from a database (e.g. mongodb) and store labels to the database.

• Create General Possibility to save/load annotations to/from MongoDB
• Make the UI customizable to select the database location
• Get a list of available sessions from the database

update latest db changes

renaming in the listview doesn't allow spaces

Currently it's not possible to add spaces in the annotation list view when renaming a label.
This is caused by the public void OnPreviewKeyDown function in the ViewHandler, which ensures that space is only used for the playback control

Overlay for saving

Having the same overlay used when loading a file for saving files, as with some of the geometric files take a while to save and this would give a visual representation of when it has finished saving.

Space in Path crashes feature extraction

If the path to the log files contain a SPACE, feature extraction crashes.

NOVA already calls cmltrain with "to path", but when cmltrain tries to create the log files it still crashes.