nyukat / birads_classifier

High-resolution breast cancer screening with multi-view deep convolutional neural networks

High-resolution breast cancer screening with multi-view deep convolutional neural networks

Introduction

This is an implementation of the model used for BI-RADS classification as described in our paper "High-resolution breast cancer screening with multi-view deep convolutional neural networks". The implementation allows users to get the BI-RADS prediction by applying our pretrained CNN model on standard screening mammogram exam with four views. As a part of this repository, we provide a sample exam (in images directory). The model is implemented in both TensorFlow and PyTorch.

Prerequisites

• Python (3.6)
• TensorFlow (1.5.0) or PyTorch (0.4.0)
• NumPy (1.14.3)
• SciPy (1.0.0)
• Pillow (5.1.0)

Data

To use the pretrained model, the input is required to consist of four images, one for each view (L-CC, L-MLO, R-CC, R-MLO). Each image has to have the size of 2600x2000 pixels. The images in the provided sample exam were already cropped to the correct size.

How to run the code

Available options can be found at the bottom of the files birads_prediction_tf.py or birads_prediction_torch.py.

Run the following command to use the model.

# Using TensorFlow
python birads_prediction_tf.py

# Using PyTorch
python birads_prediction_torch.py

This loads an included sample of four scan views, feeds them into a pretrained copy of our model, and outputs the predicted probabilities of each BI-RADS classification.

You should get the following output:

BI-RADS prediction:
        BI-RADS 0:      0.21831559
        BI-RADS 1:      0.38092783
        BI-RADS 2:      0.4007566

Additional options

Additional flags can be provided to the above script:

• --model-path: path to a TensorFlow checkpoint or PyTorch pickle of a saved model. By default, this points to the saved model in this repository.
• --device-type: whether to use a CPU or GPU. By default, the CPU is used.
• --gpu-number: which GPU is used. By default, GPU 0 is used. (Not used if running with CPU)
• --image-path: path to saved images. By default, this points to the saved images in this repository.

For example, to run this script using TensorFlow on GPU 2, run:

python birads_prediction_tf.py --device-type gpu --gpu-number 2

Converting TensorFlow Models

This repository contains pretrained models in both TensorFlow and PyTorch. The model was originally trained in TensorFlow and translated to PyTorch using the following script:

python convert_model.py saved_models/model.ckpt saved_models/model.p

Tests

Tests can be configured to your environment.

# Using TensorFlow, without GPU support
python test_inference.py --using tf

# Using PyTorch, without GPU support
python test_inference.py --using torch

# Using TensorFlow, with GPU support
python test_inference.py --using tf --with-gpu

# Using PyTorch, with GPU support
python test_inference.py --using torch --with-gpu

Reference

If you found this code useful, please cite our paper:

"High-resolution breast cancer screening with multi-view deep convolutional neural networks"
Krzysztof J. Geras, Stacey Wolfson, Yiqiu Shen, Nan Wu, S. Gene Kim, Eric Kim, Laura Heacock, Ujas Parikh, Linda Moy, Kyunghyun Cho
2017

@article{geras2017high, 
    title = {High-resolution breast cancer screening with multi-view deep convolutional neural networks},
    author = {Krzysztof J. Geras and Stacey Wolfson and Yiqiu Shen and Nan Wu and S. Gene Kim and Eric Kim and Laura Heacock and Ujas Parikh and Linda Moy and Kyunghyun Cho}, 
    journal = {arXiv:1703.07047},
    year = {2017}
}

birads_classifier's Issues

How can I use your model to train my data set

But I don't have enough data to retrain the model. I can only use my own data to adjust your weight to get the results I want

dataset

Is this dataset public available?

Can the code of your training network be published

Is the model in this code a trained model, and where is its weight saved

Could I use the models you provide to retrain my data set directly?

Excuse me,I want to use the method of transfer learning to retrain the model you provided.Should I divide my dataset to 4 parts because there are 4 views?I don't know how to do before retrain the models.Could you give me some advice?

Thank you!
Looking forward to your reply!

Source code

Dear Krzysztof,

we are working on a similar projet and struggle using high resolution pictures in our neuronal network.
as we understand, you have managed to bypass this problem and explain the way you did it in your paper. What an amazing job!

Is there a chance that you share your source code with us in order for us to understand how you have implemented the concept in your code.

We would be more than greatful :-).

We are looking forward to read from you,

When I used the parameters you provided and the pre-trained model, I used the DDSM data set to fine-tune the network, but the AUC value was 0.5. I wonder why the results in your paper could not be reached

I'm using the loss function:cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=y_prediction_birads)),The optimizer is :train_step = tf.train.AdamOptimizer(lr).minimize(cross_entropy)

How can I use your model to train my data set and look forward to your reply

PyTorch model has two-views instead of four?

Hello there,

In the paper you say that the model has four views and that the pipelines for four views are separate until global average pool and then concatenated. Something like following ASCII art:

R-MLO  |-a1-|-a2-|-a3-|--|--|--|--|--|--| \
L-MLO  |-b1-|-b2-|-b3-|--|--|--|--|--|--|\ \
                                           > >--|--|--|--|
R-CC   |-c1-|-c2-|-c3-|--|--|--|--|--|--|/ /
L-CC   |-d1-|-d2-|-d3-|--|--|--|--|--|--| /

But when I look at the layers for the python model, for MLO and CC, only one layer is defined for each one, instead of two. In the forward phase both MLO and CC images are put forward through same layer. Something like:

L-CC  \           \           \
       \|-a1-/|-a1-\|-a2-/|-a2-\|-a3-/|-a3-|... 
R-CC        /           /           /

Which creates a bizarre architecture like:

R-MLO  |-a1-|-a2-|-a3-|--|--|--|--|--|--| \
L-MLO  |-a1-|-a2-|-a3-|--|--|--|--|--|--|\ \
                                           > >--|--|--|--|
R-CC   |-b1-|-b2-|-b3-|--|--|--|--|--|--|/ /
L-CC   |-b1-|-b2-|-b3-|--|--|--|--|--|--| /

Here is the code for conv. layers taken from layers_torch.py

    def __init__(self, in_channels, number_of_filters=32, filter_size=(3, 3), stride=(1, 1)):
        super(AllViewsConvLayer, self).__init__()
        self.cc = nn.Conv2d(
            in_channels=in_channels,
            out_channels=number_of_filters,
            kernel_size=filter_size,
            stride=stride,
        )
        self.mlo = nn.Conv2d(
            in_channels=in_channels,
            out_channels=number_of_filters,
            kernel_size=filter_size,
            stride=stride,
        )

    def forward(self, x):
        return {
            "L-CC": F.relu(self.cc(x["L-CC"])),    # [Addition] (1)
            "L-MLO": F.relu(self.mlo(x["L-MLO"])),
            "R-CC": F.relu(self.cc(x["R-CC"])),    # [Addition] (2)
            "R-MLO": F.relu(self.mlo(x["R-MLO"])),
        }

Notice that in lines (1) and (2) L-CC and R-CC is forwarded from the same layer. Also the same issue for R-MLO and L-MLO.

Here is what I get when I try to reach for the first layer in the model, there is only two conv layers instead of four. Therefore, the model effectively has two views, not four.

model._conv_layer_ls[0]
Out[20]: 
AllViewsConvLayer(
  (cc): Conv2d(1, 32, kernel_size=(3, 3), stride=(2, 2))
  (mlo): Conv2d(1, 32, kernel_size=(3, 3), stride=(2, 2))
)

For TF however, the layers seems to be in line with the paper:

def all_views_conv_layer(input_layer, layer_name, number_of_filters=32, filter_size=(3, 3), stride=(1, 1),
                         padding='VALID', biases_initializer=tf.zeros_initializer()):
    """Convolutional layers across all 4 views"""
  
    input_l_cc, input_r_cc, input_l_mlo, input_r_mlo = input_layer

    with tf.variable_scope(layer_name + "_CC") as cc_cope:
        h_l_cc = tf.contrib.layers.convolution2d(inputs=input_l_cc, num_outputs=number_of_filters,
                                                 kernel_size=filter_size, stride=stride, padding=padding,
                                                 scope=cc_cope, biases_initializer=biases_initializer)
        h_r_cc = tf.contrib.layers.convolution2d(inputs=input_r_cc, num_outputs=number_of_filters,
                                                 kernel_size=filter_size, stride=stride, padding=padding, reuse=True,
                                                 scope=cc_cope, biases_initializer=biases_initializer)

    with tf.variable_scope(layer_name + "_MLO") as mlo_cope:
        h_l_mlo = tf.contrib.layers.convolution2d(inputs=input_l_mlo, num_outputs=number_of_filters,
                                                  kernel_size=filter_size, stride=stride, padding=padding,
                                                  scope=mlo_cope, biases_initializer=biases_initializer)
        h_r_mlo = tf.contrib.layers.convolution2d(inputs=input_r_mlo, num_outputs=number_of_filters,
                                                  kernel_size=filter_size, stride=stride, padding=padding, reuse=True,
                                                  scope=mlo_cope, biases_initializer=biases_initializer)

    h = (h_l_cc, h_r_cc, h_l_mlo, h_r_mlo)

    return h

Is this perhaps an oversight or am I missing something? Does this pytorch model achieve similar accuracy in the dataset?

Thanks.

How is your training data distributed when you train the network

Birads 3 Categories

Hello , i'm using this repository for my school project 'Breast cancer classification using mammoram pictures' the project is working perfectly
my question is why it shows only 3 birads categories while birads got 7 categories ?
Thank you for reading my question 👍

Image normalization

Hello,

Thank you for making this code available!

My question is regarding data preprocessing. I noticed that the example images you provide in the images directory have values in the range 0 - 255. Is this how you scaled the training images? Other than the whitening mentioned in the paper (subtracting the mean and dividing by the standard deviation), what other preprocessing did you apply during training and inference (e.g. adaptive histogram equalization)?

Thanks,
-Richard

When training the network, what's your batch number? How do you input n times 4 images into the network for training