openimaging / miqa-phase1

A web application for medical imaging quality assurance

MIQA

MIQA is a medical scan QA/QC application. It takes .nifti files and a CSV file as input, and output the same CSV file with additional QA/QC results.

The server is built based on Girder, a python based data management solution, and the client application is built with Vue.js, Vuetify, and Vue CLI. The medical scan visualization is powered by VTK.js

A screen record

Active learning

MIQA has an implementation of active learning. It uses MRIQC to extract features out of scans then uses labeled data and Random Forest Regression to evaluate if a dataset meets the standard.

Development

MIQA can be developed on Linux. See development for detail.

Deployment

MIQA can be deployed on any Linux system. This repo provides a solution with Docker.

Try out MIQA

The simplest way to try out MIQA is to follow the steps here. You could also build MIQA from scratch with the scripts here.

miqa-phase1's Issues

Fix "We found a potential security vulnerability in one of your dependencies." from GitHub

Image for Wiki page

Remove link in email for now

Do #89 first and see if the link is now useful before removing.

Add sample data to the repository

Design and develop or merge with existing ticketing system

Keep track of which scans are reviewed, which remain, and who has been informed of QA activity is currently done largely through email. This process might be streamlined if it could be managed better through MIQA directly. Explore what functionality we want and how to add or incorporate an existing system.

Upgrade docker deployment python version

The base image for our Dockerfile is node:10.15.3-stretch where the python3 in that image is python3.5, which reached end of life September 2020. Even the latest node image still has python 3.5, so we need to install newer python in the image. We are starting to get errors and warnings when building the docker stack about this old python, such as the following:

/usr/local/lib/python3.5/dist-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.
  from cryptography import utils, x509

DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.

Reorganize information in lower three panels

The biggest complaint here is that the % loaded lives adjacent to the timestep slider, seeming to indicate that the % loaded has something to do with the which timestep is being viewed in the current scan.

The lower panels need a little reorganization in general, and this should include moving the percent loaded indicator closer to the metadata, and best would be to associate it with the experiment id in some way to help indicate that it's the amount of the experiment that is loaded.

Support adding scans to an already imported set of sessions

Currently we have to import a whole new csv or json, which moves the currently imported session to a backup folder in girder, in some way losing the annotations made up to that point.

Instead we should support adding one or more scans or experiments to the existing session.

Whether this is done via some new UI element, or via a script run by an admin is still uncertain.

Set up CI

This could be a very involved or quite simple issue, depending on what we intend to do in CI. At a minimum, we should decide on client/server code styles we want to enforce, and at least make sure contributors are adhering to those.

More involved would be deciding how we want the project to be tested, choosing and implementing a testing framework, and writing tests to cover the application code.

Making sliding window approach working on pixel based NN approach

• Better classification accuracy than rescaling > 50%
• F1 scores be > 0.5 for each class

Press + hold single arrows to scan through time series

If we press+hold the single arrow, it should scan as fast as possible through the timesteps in a scan. Eventually the rate at which it scans through the series could be scaled by some user configurable "frame rate".

Add Current Team with developer access to this repository.

The phase2 team needs to be added with developer privileges so that we may more efficiently manage the issues.

Iowa: First cut of Iowa Imaging data due in September

The first QC related data needs to be shared for development purposes by September 15

Investigate performance degradation after scanning through many timesteps

I have noticed that after scanning through hundreds of timesteps, that the time to move to the next time step slowly grows the more timesteps we view. At first I see a framerate above 10 images/second, but eventually it can take more than a second to move to the next timestep. Reloading the application resets the performance back to expected level.

Use the performance tool in Chrome, it seems the time to switch is partly attributed to vtk.js setting new inputData on the trivial producer which is the source for the visualization, and partly attributed to vue.js re-rendering the DOM. As the timestep switching takes longer and longer, that time becomes dominated by vue.js.

We need to figure out what is causing vue.js to take longer and longer.

This performance degradation is distinct from that we have seen when loading many experiments, and the load time grows over time. However, the two may be related.

Change "undefined" to session name in screenshot name / email subject

Implement server-side session timeout

User should need to log back in after a fixed time since login expires.

Access error message when scans fail to load

Currently it just shows "N failed of M" but does not give a reason for the failure. Provide more information, perhaps the error stack trace directly.

Another option is to provide an error ID which can be queried in the server log by admins, but that is probably overkill for now.

Clicking single arrows should stop at end of time series

The single arrows should not, when the end (or beginning) of time series is reached, automatically skip to the next scan. Instead these arrows should become greyed out (the left one when the beginning is reach, the right one when the end is reached) to indicate to the user there are no more timesteps in that direction.

The only ways to skip to the next (or previous) scan should be via 1) clicking the double arrow, or by 2) annotating the scan.

Collaborators should be able to add notes, but not change anything else in MIQA

As a workaround, we have left collaborators with read only access and we just don't show them and "edit" type widgets in the UI, because none of those would work for them. Fixing this in the long run seems like it will involve at the least reorganizing how metadata is stored (currently notes and ratings are both part of generic metadata attached to the scan folder), but might be even more challenging given how Girder groups work in version 3.

Resend email verification not working

This should be fixed upstream, see girder/girder_web_components#306

Docker image available and updated

Update the README with a Screenshot of the latest UI

Create MIQA static website

Resolve network performance issues at Stanford

Add a client status of time till session expiry

4/14 discussion that UI will show a "time remaining" when the timeout time is approaching. System will not auto-renew timeout for initial version, but this is enough to warn the user for the first version.

Process PredictHD data and use it for pixel based NN training

Cache more images in memory

Currently the application fetches and caches only the timesteps for the current scan. We should increase to caching the entire current session. It would also be nice to show some indication in the UI of which scans are currently in memory.

We can evaluate how this works in practice (does it put too much memory pressure on the application), and perhaps eventually allow the user more control of which images are loaded.

Window level resets when moving to next timestep

The window level sliders should not reset to default values on their own, but stick wherever you put them. Currently they seem to reset when you move to the next timestep in a scan.

As scans usually have different intensity ranges, resetting the window level when navigating between scans is probably fine.

Add Licensing file to the repository

Update README with description on Active Learning

When loading multiple sessions, queue the clicked one first

When multiple sessions are loading in an experiment (a feature added in #35), it always starts with the first session, even if you clicked on the last one. Instead, it should queue up the one you clicked first, then the others can load in the normal order.

Design new version of the application

Design the next version of the MIQA application. The output is a design document that discusses the motivations for inclusion and exclusion of features in MIQA. This effort may also include restricted scope prototypes as needed.

Where to store image QC metrics in JSON and MIQA data model

We now have the ability to augment the JSON dataset spec with Quality Control metrics, currently attributes are added to a 'volumes' tag at the scan level. Do we want to rename / relocate this in the JSON and where should we load this data into MIQA. For example, in an annotations table with a foreign key to an image table?

Fix performance issue with the vue-utilities v-mousetrap

MIQA uses v-mousetrap from vue-utilities, and actually moved that library code into the application since it's no longer being maintained at its previous home.

Debugging performance degradation over many experiment loads, I discovered over 100K listeners bound to the DOM, most of which were created by `v-mousetrap. In this branch commit, I experimented with just removing the use of that library, and the number of listeners stopped growing without bound, and experiment loading performance was greatly enhanced.

However, key bindings are an important feature in the application, so we need to either fix v-mousetrap, or else replace it with something else.

Screenshots are not present in the email received

Single arrow must visit every timestep

Kilian Pohl reported that sometimes clicking the single arrow (either back or forward) skips one or more images in a time series. We need to make sure when we click the single arrows we never skip over a timestep. It's fine if the slider does that because the user is probably trying to skip directly to some timestep, but the single arrow needs to visit every step.

So far, I have not been able to reproduce this issue, either on the SRI instance or on the application running on my machine, so more work is required to identify the problem.

Rough CSV of imaging QC measures sent to team

After the email is sent, clear the sent screenshots

Client build broken

Following the developer setup instructions, the step building the client application (npm run serve) fails with a bunch of errors like the following:

 error  in ./src/App.vue?vue&type=style&index=0&lang=scss&

Module build failed (from ./node_modules/sass-loader/lib/loader.js):
Error: Cannot find module 'node-sass'
Require stack:
- /Users/scott/projects/miqa/miqa/client/node_modules/sass-loader/lib/loader.js
- /Users/scott/projects/miqa/miqa/client/node_modules/loader-runner/lib/loadLoader.js
- /Users/scott/projects/miqa/miqa/client/node_modules/loader-runner/lib/LoaderRunner.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/NormalModule.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/NormalModuleFactory.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/Compiler.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/webpack.js
- /Users/scott/projects/miqa/miqa/client/vue.config.js
- /Users/scott/projects/miqa/miqa/client/node_modules/@vue/cli-service/lib/Service.js
- /Users/scott/projects/miqa/miqa/client/node_modules/@vue/cli-service/bin/vue-cli-service.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:982:15)
    at Function.Module._load (internal/modules/cjs/loader.js:864:27)
    at Module.require (internal/modules/cjs/loader.js:1044:19)
    at require (internal/modules/cjs/helpers.js:77:18)
    at Object.sassLoader (/Users/scott/projects/miqa/miqa/client/node_modules/sass-loader/lib/loader.js:46:72)

 @ ./node_modules/vue-style-loader??ref--8-oneOf-1-0!./node_modules/css-loader??ref--8-oneOf-1-1!./node_modules/vue-loader/lib/loaders/stylePostLoader.js!./node_modules/postcss-loader/src??ref--8-oneOf-1-2!./node_modules/sass-loader/lib/loader.js??ref--8-oneOf-1-3!./node_modules/cache-loader/dist/cjs.js??ref--0-0!./node_modules/vue-loader/lib??vue-loader-options!./src/App.vue?vue&type=style&index=0&lang=scss& 4:14-415 14:3-18:5 15:22-423
 @ ./src/App.vue?vue&type=style&index=0&lang=scss&
 @ ./src/App.vue
 @ ./src/main.js
 @ multi (webpack)-dev-server/client?http://localhost:8081/sockjs-node (webpack)/hot/dev-server.js ./src/main.js

Complete error log attached.

client-build-error.txt

Improve experiment loading performance using itk.js WorkerPool.

Using the WorkerPool, we can save the repeated loads of the heavy javascript (emscripten wrapped C++ library methods) associated with the imageio worker. Instead of reloading those 3 heavy files for each image we need to read, we can just load them as many times as we have workers in the pool, which should provide much faster experiment loading.

This is under development in #39

MIQA email is not configured properly

When users create accounts, they are asked to check their email to confirm their email address, but the email never arrives, and then the administrator needs to explicitly activate their account for them.

If you send email, BCC yourself so you see the email that was sent

Add name of person sending the email to the email

Create better data statistics for PredictHD

Hans said there is a minor update to data, so I should probably wait for that.

MIQA Year 1 Report

Improve pixel-based NN classifier

Is Admin one person or multiple people?
overall_QA: do multi-class instead of regression?
Correlate NN output with IQM metrics
Correlate human ratings with IQM metrics
Weighted random sampler for PyTorch link
Lower learning rate? Other hyper-parameter tuning
focal loss?
Learning rate decay? Or other LR schedules? Cosine

Basic documentation on how to use the application

No indication something is wrong if import fails

After following the development setup instructions, including the final steps for importing the sample data, the application continues to show nothing. Only opening the console indicates anything might be wrong.

Support CSV import as well as JSON

MIQA should continue to support importing the CSV format used previously.

Since #27 adds a conversion script to the repo already, it shouldn't be too hard simply to invoke that functionality internally if the user tries to load a csv file instead of the new json format.

Review MIQA design choices