jennyziyi-xu / cxr-report-metric

Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation

Repository referenced in the paper "Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation". This repository provides code for computing metric scores for radiology report evaluation. The metrics are:

• BLEU
• BERTscore
• CheXbert labeler vector similarity
• RadGraph entity and relation F1
• Composite metric RadCliQ

Table of Contents

• Prerequisites
• Requirements
• Usage
• License
• Citing

Prerequisites

To install the dependencies, run the following command with Python 3.7:

pip install -r requirements.txt

Requirements

Ground Truth and Predicted reports must be arranged in the same order in a column named "report" in two CSV files. The CSVs should also contain a corresponding "study_id" column that contains unique identifies for the reports.

In config.py, set GT_REPORTS and PREDICTED_REPORTS to paths to the CSVs. Set OUT_FILE to the desired path for the output metric scores.

CheXbert

To compute the CheXbert vector similarity (s_emb) metric score, download the CheXbert model checkpoint here. Set CHEXBERT_PATH in config.py to the path to the downloaded checkpoint.

The code for computing the CheXbert metric score is adapted from stanfordmlgroup/CheXbert.

Paper (Accepted to EMNLP 2020): https://arxiv.org/abs/2004.09167.

RadGraph

To compute the RadGraph metric score, download the RadGraph model checkpoint from PhysioNet here. The checkpoint file can be found under the "Files" section at path models/model_checkpoint/. Set RADGRAPH_PATH in config.py to the path to the downloaded checkpoint.

The code for computing the RadGraph metric score is adapted from dwadden/dygiepp. Note: You need to apply for credentialed access to RadGraph on PhysioNet.

Paper (Accepted to NeurIPS 2021): https://arxiv.org/abs/2106.14463.

Usage

from CXRMetric.run_eval import calc_metric
calc_metric(gt_reports, predicted_reports, out_file, use_idf)

License

This repository is made publicly available under the MIT License.

Citing

If you are using this repo, please cite this paper:

@article {Yu2022.08.30.22279318,
	author = {Yu, Feiyang and Endo, Mark and Krishnan, Rayan and Pan, Ian and Tsai, Andy and Reis, Eduardo Pontes and Fonseca, Eduardo Kaiser Ururahy Nunes and Ho Lee, Henrique Min and Abad, Zahra Shakeri Hossein and Ng, Andrew Y. and Langlotz, Curtis P. and Venugopal, Vasantha Kumar and Rajpurkar, Pranav},
	title = {Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation},
	elocation-id = {2022.08.30.22279318},
	year = {2022},
	doi = {10.1101/2022.08.30.22279318},
	publisher = {Cold Spring Harbor Laboratory Press},
	URL = {https://www.medrxiv.org/content/early/2022/08/31/2022.08.30.22279318},
	eprint = {https://www.medrxiv.org/content/early/2022/08/31/2022.08.30.22279318.full.pdf},
	journal = {medRxiv}
}