ucsd-e4e / pyrenote

Pyrenote, the Audio Labeling System for Acoustics Specifies Identification Project, is built to generate human moment-to-moment labels on audio files. At present, being used to label bird audio data. Derived from the open source repo Audino

Pyrenote, The E4E Manual Audio Labeling System

This project, Pyrenote, creates moment to moment or strong labels for audio data. Pyrenote and much of this README are based on heavily on Audino as well as Wavesurfer.js. The name is a combination of Py, Lyrebird, and note (such as making a note on a label).

If you want to use Pyrenote, use the following to get started!

NOTE: Before making any changes to the code, make sure to create a branch to safely make changes. Never commit directly to main or production branch. Read github_procedures.md for more detailed information before contributing to the repo.

Usage

Note: Before getting the project set up, message project leads for env file. This file should be put in /audino. Make sure the file is never pushed to the github

Please install the following dependencies to run Pyrenote on your system:

1. git [tested on v2.23.0]
2. docker [tested on v19.03.8, build afacb8b]
3. docker-compose [tested on v1.25.5, build 8a1c60f6]

Clone the repository

$ git clone https://github.com/UCSD-E4E/Pyrenote.git
$ cd audino

Note for Windows users: Please configure git to handle line endings correctly as services might throw an error and not come up. You can do this by cloning the project this way:

$ git clone https://github.com/UCSD-E4E/Pyrenote.git --config core.autocrlf=input

For Development (Note this is the one we will test on and use)

Similar to production setup, you need to use development configuration for working on the project, fixing bugs and making contributions. Note: Before proceeding further, you might need to give docker sudo access or run the commands listed below as sudo.

To build the services (do this when you first start it), run:
Note: Remember to cd into audino before starting

$ docker-compose -f docker-compose.dev.yml build

To bring up the services, run:

$ docker-compose -f docker-compose.dev.yml up

Then, in browser, go to http://localhost:3000/ to view the application.

To bring down the services, run:

$ docker-compose -f docker-compose.dev.yml down

Troubleshooting for starting docker

1. Docker containers do not even get a chance to start

• Make sure docker is set up properly
• Make sure docker itself has started. On Windows, check the system tray and hover over the icon to see the current status. Restart it if necessary

2. Backend crashes

• For this error, check the top of the log. It should be complaining about /r characters in the run-dev.sh files
• The backend will crash if the endline characters are set to CRLF rather than LF
• On VSCode, you can swap this locally via going into the file and changing the CRLF icon in the bottom right to LF
• Do this for frontend/scripts/run-dev.sh and backend/scripts/run-dev.sh

3. Database migration issues

• If the backend complains about compiler issues while the database migration is occurring go into backend/scripts/run-dev.sh
• On line 25, check and make sure that the stamp command is pointing to the right migration for the database
  • Ask for help on this one

Getting Started

At this point, the docker should have gotten everything set up. After going to http://localhost:3000/ you should be able to log into the docker

To access the site, sign in with the username of admin and password of password. On logging in, navigate to the admin-portal to create your first project. Make sure to make a label group and some labels for the project!

After creating a project, get the API key by returning to the admin portal. You can use the API key to add data to a project. Create a new terminal (while docker is running the severs) and cd into audino/backend/scripts. Here use the following command:

python upload_mass.py --username admin.test --is_marked_for_review True --audio_file C:\REPLACE\THIS\WITH\FOLDER\PATH\TO\AUDIO\DATA --host localhost --port 5000 --api_key REPLACE_THIS_WITH_API_KEY

Make sure to have a folder with the audio data ready to be added. For testing purposes, get a folder with about 20 clips.

Once that runs, you are ready to start testing!

For Production (Don't use on windows)

You can either run the project on default configuration or modify them to your need.
Note: Before proceeding further, you might need to give docker sudo access or run the commands listed below as sudo.
Note: Remember to cd into audino before starting

To build the services, run:

$ docker-compose -f docker-compose.prod.yml build

To bring up the services, run:

$ docker-compose -f docker-compose.prod.yml up

Then, in browser, go to http://0.0.0.0/ to view the application.

To bring down the services, run:

$ docker-compose -f docker-compose.prod.yml down

For Dev Team:

Features should be turned on and off by admins for individual projects. When adding a new feature to either a project's data page or annotation page, make sure to do the following:

1. Go to .\audino\frontend\src\containers\forms\featureForm.js
2. Add a new item in the featuresEnabled directory. This will be the name of the feature_toggle variable.
3. Return to the page you are working on.

• For example, if you are working on the annotation page, navigate to the componentDidMount() method
• about 20 lines down in the setState callback, add to the list SOME_VAR: response.data.features_list['VARIABLE_NAMED_IN_STEP_2'].

pyrenote's Issues

Reminder to fix Admin Portal Project Rename/Delete

Putting this task on hold while I work on creating a data upload System. Right now we need to create a delete api call and create a frontend button for it as well as add a rename api call for projects (right now it just creates a new project with the inputted name)).

If anyone wants to pick up on this, let me know and I'll tell you what I have done so far. I'll probably work on this after spring break once I can get data to be uploaded from the frontend.

Investigate MySQL Bugs In Docker

I got an error earlier today when attempting to start up docker between production and dev docker compose files. Appears to have something to do with improperly bringing down each thing. Error message along the lines of:

[ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.

and then later down

[ERROR] Fatal error: Can’t open and lock privilege tables: Table ‘mysql.user’ doesn’t exist [ERROR] Fatal error: Failed to initialize ACL/grant/time zones structures or failed to remove temporary table files. [ERROR] Aborting

It is possible this is just me not bringing it down correctly but the test here is trying to open and close docker-compose.dev.yml and docker-compose.prod.yml on main branch and see how it breaks the other.

I did find a workaround of just stopping the containers built from docker-compose.dev.yml rather than bringing them down as that way docker-compose.prod.yml treats the previously exsiting container from docker-compose.dev.yml as its own.

Fix centering issues when window becomes smaller

The styling becomes stranger when the window becomes smaller.
The pause and play button becomes shifted to the left
Make it so it is centered

Rebuild Admin Portal to Manage Database ~Thread~

Create a fully function database manager that adds the features of:

• Deleting users, projects, data, labels
• Allows for editing label names/project names/etc
• makes it easier to assign new users to a project

Create ability to change username in admin portal

• Can use this issue as a modal for renaming all database items

FRONTEND
Need to create frontend variable to store old username for inital backend look up
Add text field to rewrite the username
Add new usernames to json, initally set it to the old username for error handling
BACKEND
if old_username != username
change the username to the new username

Improve the alert system by turn alerts into a dismissable popup

Currently the alert pushes down all elements on the page, lets have the alert system be overlayed with the screen

Add second button for json only output for annotations

Add clip duration and sampling rate to data uploaded from admin portal

Infinite scrolling on Project data page

Page that displays all the different files should scroll infinitely (renders as you scroll) as opposed to being broken up into multiple pages.

Speed up Spectrogram rendering

Previous button should be able to pull that exists but has not been accessed yet?

If the user goes to the last file in the project, should they be able to go back

Save All Button disappears when hitting delete

Visual glitch. When deleting an annotation from the annotation page, save all button disappears. The save all button should only be disabled to avoid accidental saves

Track User's Time On Clip

Currently, we track when the annotation was made and when the clip was last modified
What To Add:

• To Segmentations table, add a clip first opened to track when each clip was opened (prior to creating a segmentation. If a user looks at a clip but doesn't make a segmentation, don't put this)by a segmentation and who made the segmentation
• To Data Clip, Add a section to add a json of the timestamps and the user who opened the clip.

Then add parts to the requests need to create and modify these to track time.

Change styling of a region to indicate that it is selected

List of ideas for implementation:

• Create a symbol in spectrogram to denote region
• temporarily change color of spectrogram (bad idea removes save colorization)
• Create a button to go to the region's timestamp

refactor next/previous buttons

change the next/previous buttons to use state so they are more efficient instead of simply caching

Password recovery idea

Create systems for admins and or users to recover lost passwords

Block user from renaming themselves in admin portal (Or remind them to log back in)

If they change thier username in admin, the redis gets confused and blocks the user from doing anything. Best thing to do is give a warning to inform the user to sign back in should they rename themselves.

Segmentation Delete Logging

Create a table and every time a segment is deleted, logs it in the table. Possibly add button to backend that makes an api call to pull a json of the log.

Table should look like this were it records every about the segmentation that was deleted as well as the user who deleted and time of deletion:

class SegmentationDeletedLog(db.Model):
    __tablename__ = "segmentation_deleted_log"
    
    id = db.Column("id", db.Integer(), primary_key=True)
    
    username = db.Column(
        "username", db.String(128), index=True, unique=True, nullable=False
    )

    segmentation_id = db.Column("id", db.Integer())

    data_id = db.Column(
        "data_id", db.Integer(), db.ForeignKey("data.id"), nullable=False
    )

    start_time = db.Column("start_time", db.Float(), nullable=False)

    end_time = db.Column("end_time", db.Float(), nullable=False)

    transcription = db.Column("transcription", db.Text(), nullable=True)

    created_at = db.Column(
        "created_at", db.DateTime(), nullable=False, default=db.func.now()
    )

    created_by = db.Column(
        "created_by", db.String(128), nullable=False,
    )

    clip_frist_opened_at = db.Column(
        "clip_frist_opened_at", db.DateTime(), nullable=False,
    )

    last_modified_by = db.Column(
        "last_modified_by", db.JSON(), nullable=False,
    )
    last_modified = db.Column(
        "last_modified",
        db.DateTime(),
        nullable=False,
        default=db.func.now(),
        onupdate=db.func.utc_timestamp(),
    )

    values = db.relationship(
        "LabelValue", secondary=annotation_table, back_populates="segmentations",
    )

    last_modified = db.Column(
        "last_modified",
        db.DateTime(),
        nullable=False,
        default=db.func.now(),
        onupdate=db.func.utc_timestamp(),
    )

Data upload Production Build Bug

Sever in production builds has difficulties reading files/json request sent from windows 10, google chrome but only for the data upload. Dev Builds can upload data from dev fine and it seems like command line on sever is still working.

I do not believe this will impact the summer release of the sever. I can bring the site down temporarily to upload from a dev sever if need be and since we will first make users and projects before users come onto the site, the bug will not interrupt service for our users.

Test: Go to admin portal while on production (and double check dev still works too) and use the upload data button to see if sever runs into a error

Create super-admin role that controls deletions

Need to - Add and create a 3rd role
Then add that 3rd role and give it the same permissions as the admin
then create a special permission for that main-admin's control over user deletion

Update csv to include sample rates and length of the clip itself

Fix layout of labeling controls to take up less vertical space

Ideas

• Create 2 columns, one contains segment transition and the other contains the actual labeling

Mutli Annotation Support

Currently we have mostly worked our code to work well with a single label category, however projects should have the ability to have at least 2 label categories (Say a bird and bat label category).

TODO:

• Create 2 label categories in a test project
• Check to see if they both appear on the labeling page, fix if it doesn't
• Make sure all annotation data is sent to backend
• On backend save the data and make sure it appears in the json output
• Figure out a way to have the annotations of multiple categories appear on csv output (consult with jacob on this)

Linting and comment out debug code

Frontend and backend contain old debug code and has not been linted for serval months now. A priority for early summer would be to clean up the code to make it easier to prepare for the larger projects to come, such as the creation of a tutorial sequence for new users.

Rebuild Admin Portal

Create a fully function database manager that adds the features of:

• Deleting users, projects, data, labels
• Allows for editing label names/project names/etc
• makes it easier to assign new users to a project

Data Upload From Admin portal

• Create button to upload data
• Figure out how to create systems to upload serval files to a backend
• Have it go through the same pipeline that Victoria is working on.

Handle overlap ideas

Toggle layers?
Different colors for diffrent labels

Update June 18, 2021:
Updating because old description is terrible.

As of now wavesurfer renders regions one on top of the other. This might make it hard for labeling things.

Add a file comments list for people collaborator

Have a comment box for people to commicate not real time

• We will need another table to describe these commicate
• Add react component for that in the file
• AHHHHHHHHHHHHHH file storage issues to keep in mind

To Next Unlabeled Region Button

When there are unsaved clips and the user hits the "next" button, automatically set the position of the waveform to the first unsaved clip

Filename on spectrogram display

Add the filename to the spectrogram page

Make expandable menu for small screen sizes

Currently, the buttons in the navbar disappear for small screens and are inaccessible. Create a button that will appear for small screen sizes which displays an expandable menu to show navbar items in a vertical list.

Turn regions into bounding boxes

This is a WIP on the branch y-axis-bounding-region

Basically:

• create a 2d region that measures both range of time and frequency of bird call (DONE)
• Save data to backend (DONE)
• push data onto data download file in admin portal (DONE)

New Backend API Calls

Here is a combined list of API calls we need to create for the backend (including the ones we have so far)

Note: I will edit this to mark which items on this list will be addressed by who under what GitHub issue. Please make a new issue for any item that you will address.

ENTRYPOINTS WE NEED

• Patch/Post request to delete a user given a username (json)
• Patch/Post request to delete serval users given a json of usernames
• Patch/Post request to delete serval projects given a json of projectsIDs
• Patch/Post request to delete a projects given a json of a projectsID
• Patch/Post request to rename a projects
• Patch/Post request to delete a projects labels (see how json for how labels are made)
• Patch/Post request to rename a projects labels (see how json for how labels are made)
• Patch/Post request to delete a projects labels groups (see how json for how labels are made)
• Patch/Post request to rename a projects labels groups (see how json for how labels are made)
• Pathc/Post request to only swap a user's type from normal to admin (optional)

ENTRYPOINTS WE SHOULD ALREADY HAVE BUT NEED TO FIX

• Patch/Post request to rename a projects
• Patch/Post request to rename a users
• Patch/Post request to rename a projects
• Patch/Post request to add data to a project (we need to handle pickling from website)
  This requires thinking as to how data will be added, so do this one later

ENTRYPOINTS WE SHOULD ALREADY HAVE
use these as references and be sure to review in case we need to change these calls

• Ability to create a new project, user, or label
• Ability to edit some values of users and labels
• Ability to get a list of users/projects

Tweak user sign up page to validate password

Have the user enter their selected password twice.
Helps differentiate between the sign in page and the sign up page.
Lower priority.

Continuous integration with Github Actions

Start integrations with Github Actions to deploy onto SDCC server

Needing to add an extra annotation before being able to delete 1 annotation and save all.

Weird bug where the user must add an annotation in order to hit delete and save all on the annotation page. Frist reported by Jacob.

Test: Be able to delete and then hit save all on the annotation page without issues. Double check via admin portal to make sure it actually saved.

To Next Unlabeled Region Button

Create a button that will scroll the spectrogram to the next unlabeled region. An unlabeled region has not had every annotation sections filled out (so in the case of having 2 label categories, 0 or 1 filled categories count as the whole region being unlabeled).

For simplicity, this button could just take the user to the first clip in an array containing every clip that has not been fully labeled. So there would be no more clips when that array is empty.

This makes use of one of two wave surfer functions:

• seekAndCenter(progress) – Seeks to a progress and centers view [0..1] (0 = beginning, 1 = end). (Probably this one frist)
• seekTo(progress) – Seeks to a progress [0..1] (0 = beginning, 1 = end).

Progress parameter could take in region.start / wavesurfer.duration to get the location to scroll to.

Add auto assign annotation

Create button which toggles auto annotate allowing new regions to be automatically assigned a annotation

What is needed

Add button to toggle the mode
create popup/dropdown to select the label the user wants to automatically put

Add scaling for playback speed

Count number of unsaved clips on hitting next button

When there are unsaved clips and the user hits the next button, count the number of unsaved clips and display it to the user in the pop up alert message

Pass title into wavesurfer.js????? Check to see if wavesurfer can do this

Admin Portal 3.0 Release + Data Dashboard 2.0: File Management

Implement a basic file management system that can allow admins to move files into folders for better organization
(Note depending on the implementation this may impact how I create the completed, needs review sections.

Changes:
Database updates
Each file (data table) needs

• a col to describe the file path of each audio file
• a table to organize the folder management????

frontend

• Dashboard now organizes each file into a diffrent folder. Folder links appear on the top of the list, and audio clips appear below (display preexisting information for them?)
• Users can click into each folder
• For each project, give users a quick-start button that gives them a random assortment of data to look over (this requires a rework of the system that gives data to look at) (Maybe let user select which folder directory they want to randomly choose files from)
• maybe have them review previosuly reviewed data in this system via user cross correlation of audio clips and or have them simply review a completed clip from this quick start button (have time dicuss this idea further. See BAT's techical paper for user cross correlation: http://eecs.qmul.ac.uk/~keno/17.pdf)
• Admin Portal now has a button to display magement system and audio files
• Admins have the ability to move serval files, delete files, rename files, etc
• Admins can move files into diffrent folders, they also can create folders

Backend:

• Add ability to upload a folder and have it upload everything in each folder recursivelike
• Other changes needed to mange the file mangement stuff

Very experimental stuff here. This kind of thing should get discussed for collaborator meeting before jumping into

Next Button Rework 3 Idea: Cutting the refresh

http://eecs.qmul.ac.uk/~keno/17.pdf

I was reviewing this audio labeling system. One feature they include is loading audio clips on the same page rather than refreshing the page. We should spend experiment and see if we can implement a similar feature and if that improves the amount of time needed to label data.

Bug with users signing up where they are not sent to dashboard

After a user fills out data and hits save, they should have been redirect to dashboard. On the docker-compose-fixes-nginx branch this doesn't happen

Workaround: user goes back to login page and logins in with the account they just made.

Branch bug was found in: docker-compose-fixes-nginx

• Could be related to http to https rerouting?

Possible way to hide annotations on spectrograms

Having annotations on top of the spectrograms makes the spectrogram harder to see. Need a button (or toggle) to hide and show annotations.

Delete User

Database issues when trying to open project on local machine

Ran into a bunch of errors with not creating admin accounts when opening the machine

Delete Label Category Bug

There is a conflict error when adding a new label after deleting a label. Thus a label is not getting deleted properly (ei, the label ID is shared by a previously deleted label).

• Test: Create Labels then delete them, if you can add additional labels, the issue is fixed.

Previous file Button

Go back to the previous file (if there are any)

Restore Backup System

We can always manually restore the mysql database. However it would be nice to create a docker container that can run an rclone command that restores the mysql container.