mida-project / prototype-heatmap

[IJHCS] Assistive heatmaps prototype for promoting eXplainable AI (XAI) in the medical workflow for the breast cancer diagnosis. The work was published in the International Journal of Human-Computer Studies.

Assistive Heatmaps Prototype

⚠️ This repository information is not yet curated. For similar use, please follow the prototype-multi-modality-assistant repository on the MIDA organization.

MIDA is a Research Project that deals with the use of a recently proposed technique in literature: Deep Convolutional Neural Networks (CNN). These deep networks will incorporate information from several different modes by a User Interface (UI) implemented based on our Prototype Breast Screening repository. The hereby repository is a mirror of our Prototype Breast Screening repository which is an Open Source Project with the goal to deliver an example of web based medical imaging platform for the breast cancer diagnosis. We also have several demos to see in our YouTube Channel, please follow us.

Citing

We kindly ask scientific works and studies that make use of the repository to cite it in their associated publications. Similarly, we ask open-source and closed-source works that make use of the repository to warn us about this use.

You can cite our work using the following BibTeX entry:

@article{CALISTO2021102607,
title = {Introduction of Human-Centric AI Assistant to Aid Radiologists for Multimodal Breast Image Classification},
journal = {International Journal of Human-Computer Studies},
pages = {102607},
year = {2021},
issn = {1071-5819},
doi = {https://doi.org/10.1016/j.ijhcs.2021.102607},
url = {https://www.sciencedirect.com/science/article/pii/S1071581921000252},
author = {Francisco Maria Calisto and Carlos Santiago and Nuno Nunes and Jacinto C. Nascimento},
keywords = {Human-Computer Interaction, Artificial Intelligence, Healthcare, Medical Imaging, Breast Cancer},
abstract = {In this research, we take an HCI perspective on the opportunities provided by AI techniques in medical imaging, focusing on workflow efficiency and quality, preventing errors and variability of diagnosis in Breast Cancer. Starting from a holistic understanding of the clinical context, we developed BreastScreening to support Multimodality and integrate AI techniques (using a deep neural network to support automatic and reliable classification) in the medical diagnosis workflow. This was assessed by using a significant number of clinical settings and radiologists. Here we present: i) user study findings of 45 physicians comprising nine clinical institutions; ii) list of design recommendations for visualization to support breast screening radiomics; iii) evaluation results of a proof-of-concept BreastScreening prototype for two conditions Current (without AI assistant) and AI-Assisted; and iv) evidence from the impact of a Multimodality and AI-Assisted strategy in diagnosing and severity classification of lesions. The above strategies will allow us to conclude about the behaviour of clinicians when an AI module is present in a diagnostic system. This behaviour will have a direct impact in the clinicians workflow that is thoroughly addressed herein. Our results show a high level of acceptance of AI techniques from radiologists and point to a significant reduction of cognitive workload and improvement in diagnosis execution.}
}

Instructions

First of all, you will need NodeJS installed locally on your machine. This project needs both npm and http-server dependencies to install and run the core project. If you do not have those installed please follow the INSTALL instructions.

DICOM Server

The following assumes you will be using a git version control for this repository, storing thanks to GitHub. First, Download and Install git. Our system needs to be integrated with WADO-URI servers, DICOMWeb servers or any HTTP based server that returns a DICOM P10 instances. We suggest you to use an Orthanc server, since it is a simple and powerful standalone DICOM server by providing a RESTful API.

You can download a latest version or you can use our own sample of an Orthanc version with our examples of patient images. The instructions to use our solution are as follows.

Clone

1.1.1. Clone the DICOM Server repository:

git clone [email protected]:MIMBCD-UI/dicom-server.git

1.1.2. Go inside the project folder:

cd dicom-server/

Install

1.2.1. Install the local dependencies:

npm install

1.2.2. You can now Run the project, just follow the next section.

Run

1.3.1. Inside the project folder:

cd dicom-server

1.3.2. Start the DICOM Server (Orthanc) for MacOS:

npm run start:multi

OR

cd ..
cd dicom-server/orthancAndPluginsOSX.stable/
./startOrthanc.command

NOTE: If you are not using MacOS, for instance, if you are using Windows or Debian you have a documentation for that. Just follow the Windows or Debian documentations. You also have several other options.

1.3.3. Open the link:

localhost:8248

NOTE: If you need some help see the Demo.

Main Server

Our main server uses NodeJS and has several dependencies. For the following steps you must have already installed both NodeJS and npm in your machine.

Clone

2.1.1. Clone the project repository:

git clone [email protected]:MIMBCD-UI/prototype-heatmap.git

2.1.2. Go inside the project folder:

cd prototype-heatmap/

Install

2.2.1. Install the local dependencies:

npm install

2.2.2. You can now Run the project, just follow the next section.

Run

2.3.1. Inside the project folder:

cd prototype-heatmap/

2.3.2. If you have already run the DICOM Server on a previous section, please jump to the 2.3.3. point, otherwise do:

npm run dicom-server

2.3.3. Run the code:

npm run build:multi

2.3.4. Start the project:

npm run start:multi

2.3.5. Open the link:

localhost:8286/src/public/index.html

Allow-Control-Allow-Origin

Access-Control-Allow-Origin is a CORS (Cross-Origin Resource Sharing) header. If you want to know How does Access-Control-Allow-Origin header work? follow the link.

Google Chrome

• To deal with the CORS issue it is necessary to open Google Chrome with --disable-web-security flag on:

open /Applications/Google\ Chrome.app --args --disable-web-security --user-data-dir

• Or install the CORS plugin for Google Chrome.

About

For more information about the MIDA Project just follow the link. Pieces of information about details of this repository are also in a wiki. This prototype was developed using several libraries and dependencies. Despite that all libraries had their importance and supported the development, one of it was of chief importance. The CornerstoneJS library and secondary libraries, respectively, are supporting this prototype. We Acknowledge all people involved in the path.

License

Copyright © 2017 Instituto Superior Técnico (IST)

The prototype-heatmap repository is distributed under the terms of both Academic License and Commercial License, for academic and commercial purpose, respectively. For more information regarding the License of the hereby repository, just follow both ACADEMIC and COMMERCIAL files.

Acknowledgments

A special thanks to Chris Hafey, the propelling person of CornerstoneJS, who also developed the cornerstoneDemo. Not forgetting the three supporters of the CornerstoneJS library, Aloïs Dreyfus, Danny Brown and Erik Ziegler. We also would like to give a special thanks to Erik Ziegler who support several issues during this path.

• Aloïs Dreyfus (adreyfus)

• Chris Hafey (chafey)

• Danny Brown (dannyrb)

• Erik Ziegler (swederik)

• Jason Hostetter

• Marcelo Ribeiro (sysline)

• Sébastien Jodogne (jodogne)

• Steve Pieper

prototype-heatmap's Issues

Create AI-Assistive Button

The idea of this issue is to create an AI-Assistive button that will trigger the #3 issue functionality. From here, the user can turn-on and turn-off our Assistant option to view the heatmaps across the medical image.

Implementing First Features

The presented issues are part of an early developement from an external prototype. The prototype is presented on prototype-multi-modality-assistant repository. Also, this issue will serve the basis of the next developments (issue #5) for our Assistant system. More precisely, this part will provide eXplainability (XAI) across the Assistant answers.

List of new features from meeting:

• AI-Assistive Button #2

• Medical Imaging Overlap #3

Importing Randomly N Patients Routine from M Patients Set Functionality

For this issue, we aim to import and bring early functionality regarding the random control of the way that the studyList.json file is generating the Patient ID respectively. We need to force the studyList.json file to generate only N patients, from a set of M patients. The functionality is deeper explained on an early issue #19 of the prototype-multi-modality-assistant repository.

Create Basic Image Processing Source

This issue aims at developing a new repository, as well as the source code. We will need to develop several routines to read .dcm image, and from this, apply the heatmapping strategy. The implementation and development details will be presented on the list of issues of the new prototype-basic-image-processing repository.

Heatmap Pre-User Testing Number Seven Improvements

On an early phase (issue #1), we developed an Assistant feature, called heatmaps, to help us eXplaining (XAI) the results of our autonomous AI-Assisted methods across the breast cancer diagnosis. Such automation is crucial since it reduces the inspection performed by the radiologists, that is still rudimentary in current clinical setups. Therefore, the methods eXplainability (XAI) will provide radiologists answers for the recommendations and decisions made.

Based on the literature, we plan to improve and prepare our Heatmap prototype for future user tests, within a context of scaling our solution. We aim to understand how clinical institutions can use our system with impactful healthcare systems.

In this set of issues, our requirements are as follows. A twofold of conditions must be addressed across the final solution. We aim to achieve some prototype improvements to support the future of our Heatmap prototype user tests, more detailed below.

List of enhancing features from pre-user testing phases:

• Importing Randomly N Patients Routine from M Patients Set Functionality #6

• Create Basic Image Processing Source #7

• Basic Overlay #8

The first issue, titled as Importing Randomly N Patients Routine from M Patients Set Functionality (issue #6), is based on early developments (issues #18 and #19 of the prototype-multi-modality-assistant repository). The idea is to import and bring functionality regarding the random control of the way that the studyList.json file is generating the Patient ID respectively. We need to force the studyList.json file to generate only N patients, from a set of M patients.

The second issue, titled as Create Basic Image Processing Source (issue #7), aims at developing a new repository, as well as the source code. We will need to develop several routines to read .dcm image, and from this, apply the heatmapping strategy.

The third issue, titles as Basic Overlay (issue #8), aims at developing the overlay functionality of the medical images. We will need to develop several routines to read the images directly from the WADO (Orthanc) server and the ".png" images, and from this, overlay the several ".png" images (i.e., heatmaps) within the image on the WADO (Orthanc) server.

Medical Imaging Overlap

From the #2 issue functionality, the system namely called as Assistant can now overlap two images. The first image is the vanilla DICOM format image of the patient. The second image is the resulting image from our AI-Assisted techniques with the heatmaps. From this second image, it will show a JPG, PNG or another standard, with the image heatmaps. The idea is to overlap both images.

Addressing First Machine Learning eXplainability Models

I just found out two exciting and potential ideas to our solution. All information and links are listed as follows.

The first one is SHapley Additive exPlanations (SHAP). SHAP is a unified approach to explaining the output of any ML model. They also have interesting issues regarding breast cancer.

Second, a regroup of publications and software on developing new methods to understand nonlinear predictions of State-Of-The-Art (SOTA) for ML models. The site is called heatmapping.org and has lots of content.

Basic Overlay

For this issue, we aim at developing the overlay functionality of the medical images. We will need to develop several routines to read the images directly from the WADO (Orthanc) server and the ".png" images, and from this, overlay the several ".png" images (i.e., heatmaps) within the image on the WADO (Orthanc) server.

By providing an API that, given a Patient ID (e.g., 12345-12345-12345/file which are .dcm format image) and a folder with two PNG files. The two PNG files are, for instance, 12345-12345-12345_cloud.png and 12345-12345-12345_full.png file names. In the end, it is just needed to overlay the two PNG images with the WADO (Orthanc) server image.