farrell236 / ratchet

RAdiological Text Captioning for Human Examined Thoraxes

RATCHET: RAdiological Text Captioning for Human Examined Thoraxes

RATCHET is a Medical Transformer for Chest X-ray Diagnosis and Reporting. Based on the architecture featured in Attention Is All You Need. This network is trained and validated on the MIMIC-CXR v2.0.0 dataset.

Architecture

Run the code

Download pretrained weights and put in ./checkpoints folder.

• ratchet_model_weights_202303111506.zip
  Size: 1.5G
  MD5: 26ab19cf18908841320205e192dabe9f
  

• ratchet_model_weights_202309022247.zip
  Size: 695M
  MD5: e028c1551419c059f62b1598e8ef92f3
  

Start streamlit to run the webapp:

streamlit run web_demo.py

Environment:

Python 3.9.10

Packages:

imageio                  2.26.0
matplotlib               3.7.1
numpy                    1.23.5
pandas                   1.5.3
scikit-image             0.20.0
streamlit                1.20.0
tensorflow               2.11.0
tokenizers               0.13.2
tqdm                     4.64.1

Docker Container

Build the docker container:

docker build -t ratchet ./Dockerfile

Run the docker image on CXR images:

docker run --user $(id -u):$(id -g) \
-v /path/to/image_input_folder:/code/RATCHET/inp_folder \
-v /path/to/report_output_folder:/code/RATCHET/out_folder:rw \
-i -t ratchet python run_model.py

Each image in inp_folder would have a corresponding .txt report saved in out_folder.

Results

                   

Generated Text:

In comparison with the study of ___, there is little overall change. Again there is substantial enlargement of the cardiac silhouette with a dual-channel pacer device in place. No evidence of vascular congestion or acute focal pneumonia. Blunting of the costophrenic angles is again seen.

More Examples

ratchet's Issues

Training the model using 10% of the dataset

@farrell236
Hey farrell,
Since the dataset is too large can I train the model using only 10% of the dataset? (Currently I don't have enough computational power and other resources to train the model with whole MIMIC-CXR dataset of size ~500GB). If that is possible what are the steps, I need to follow to successfully train the model with only 10% of the dataset. As far as I understand, I may need to,

• Create the MIMIC_AP_PA_train.csv with only 10% of the original train dataset
• Create both mimic-merges.txt & mimic-vocab.json from the (10% train + test + validate) data

Other than these do I have to follow any other steps?

Also, instead of DenseNet-121 can't we use something like MobileNetV2 for the encoder?

Running web_demo

I get a

"No Pretrained Densenet-121 weights specified"

Used report section

Hi! Thanks for your work and publishing your code.
Which section of the report did you predict? The findings or impression section or both? Or the output of the create_section_files.py, which can be either of them depending on what's included in the report.

Thank you in advance!

DenseNet121 Pre-trained Weights Error

I am using the pre-trained weights in the classifer/assets/ folder and I get the following error when running the code:

ValueError: The name "zero_padding2d" is used 2 times in the model. All layer names should be unique.

Any help would be appreciated.

How to train or evaluate the model?

Considering the training and evaluating the model, how can it be done?
I tried to evaluate the model using python3 evaluate.py but it looks like there is a missing file

FileNotFoundError: [Errno 2] No such file or directory: 'preprocessing/mimic/MIMIC_AP_PA_test.csv'