Hi, hope you see this. Which version of TensorFlow and python did you use to train the model, 'cause my version fails to execute the following code:
precision, recall, thresholds = precision_recall_curve(y_true, y_preds) in training isic dataset.
Guess it might be some version issues. Thanks!

Hi there,
Thanks for sharing your code. I try to reproduce your article and use your code intact, but the experimental results are quite different from the experimental results in your paper. I would like to ask you what you need to pay attention to in the process of reproducing the code.
model:attn_reg,loss:focal_tversky
my results:
DSC 0.748
Precision 0.860
Recall 0.752
results in article:
DSC 0.804
Precision 0.829
Recall 0.817
Hope to get your help! thanks!

From newmodels.AttnGatingBlock:

• theta_x is half the (spatial) size of x (due to the strided conv)
• g (as well as phi_g) is half the (spatial) size of x (due to max pooling in the unet)

So upsample_g is also half the (spatial) size of x and there is no need to calculate the strides in Conv2DTranspose since it will always be 1, or am I missing something?

The 2x UP step on x_l in Fig. 3 of your paper should be 2x DOWN then, right?

Hope you can see this.
I am really exciting to have chance to learn attention unet with your work and codes, i wanna to learn how to figure out the feature maps with CAM (class activate map), is the 'attention coefficiency' tensor named 'q_att_bn_1'?

Traceback (most recent call last):
File "isic_train.py", line 93, in
model = newmodels.attn_reg(sgd, input_size, losses.focal_tversky)
File "/media/nomachine/project/focal-tversky-unet-master/newmodels.py", line 241, in attn_reg
img_input = Input(shape=input_size, name='input_scale1')
File "/home/hfcui/.local/lib/python3.6/site-packages/keras/engine/input_layer.py", line 178, in Input
input_tensor=tensor)
File "/home/hfcui/.local/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/hfcui/.local/lib/python3.6/site-packages/keras/engine/input_layer.py", line 87, in init
name=self.name)
File "/home/hfcui/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 541, in placeholder
x = tf.placeholder(dtype, shape=shape, name=name)
AttributeError: module 'tensorflow' has no attribute 'placeholder'

Hi sorry to bother you with this but my model is not learning at all the loss stays the same after every epoch -0.3370

This is the model

def unet(input_shape=(128, 128, 128), optimizer=Adam, initial_learning_rate=5e-4,
loss_function=weighted_dice_coefficient_loss):

inputs = Input(shape=input_shape)
conv1 = UnetConv3D(inputs, 32, is_batchnorm=False, name='conv1')
pool1 = MaxPooling3D(pool_size=(2, 2,2 ))(conv1)

conv2 = UnetConv3D(pool1, 64, is_batchnorm=False, name='conv2')
pool2 = MaxPooling3D(pool_size=(2, 2,2 ))(conv2)

conv3 = UnetConv3D(pool2, 128, is_batchnorm=False, name='conv3')
pool3 = MaxPooling3D(pool_size=(2, 2,2 ))(conv3)

conv4 = UnetConv3D(pool3, 256, is_batchnorm=False, name='conv4')
pool4 = MaxPooling3D(pool_size=(2, 2,2 ))(conv4)

conv5 = Conv3D(512, (3, 3, 3), activation='relu', kernel_initializer=kinit, padding='same', data_format = 'channels_first')(pool4)
conv5 = Conv3D(512, (3, 3, 3), activation='relu', kernel_initializer=kinit, padding='same', data_format = 'channels_first')(conv5)

up6 = concatenate([Conv3DTranspose(256, (2, 2,2 ), strides=(2, 2,2 ), kernel_initializer=kinit, padding='same', data_format = 'channels_first')(conv5), conv4], axis=1)
conv6 = Conv3D(256, (3, 3, 3), activation='relu', padding='same', data_format = 'channels_first')(up6)
conv6 = Conv3D(256, (3, 3, 3), activation='relu', padding='same', data_format = 'channels_first')(conv6)

up7 = concatenate([Conv3DTranspose(128, (2, 2,2 ), strides=(2, 2,2 ), padding='same', data_format = 'channels_first')(conv6), conv3], axis=1)
conv7 = Conv3D(128, (3, 3, 3), activation='relu', kernel_initializer=kinit, padding='same', data_format = 'channels_first')(up7)
conv7 = Conv3D(128, (3, 3, 3), activation='relu', kernel_initializer=kinit, padding='same', data_format = 'channels_first')(conv7)

up8 = concatenate([Conv3DTranspose(64, (2, 2,2 ), strides=(2,2,2 ), kernel_initializer=kinit, padding='same', data_format = 'channels_first')(conv7), conv2], axis=1)
conv8 = Conv3D(64, (3, 3, 3), activation='relu', kernel_initializer=kinit, padding='same', data_format = 'channels_first')(up8)

up9    = concatenate([Conv3DTranspose(32, (2, 2,2 ), strides=(2, 2,2 ), kernel_initializer=kinit, padding='same', data_format = 'channels_first')(conv8), conv1], axis=1)
conv9  = Conv3D(32, (3, 3, 3), activation='relu',  kernel_initializer=kinit, padding='same', data_format = 'channels_first')(up9)
conv9  = Conv3D(32, (3, 3, 3), activation='relu', kernel_initializer=kinit, padding='same', data_format = 'channels_first')(conv9)
conv10 = Conv3D(3, (1, 1, 1), activation='relu', kernel_initializer=kinit,padding = 'same', name='final', data_format = 'channels_first')(conv9)

activation_name = 'sigmoid'
activation_block = Activation(activation_name)(conv10)
model = Model(inputs=[inputs], outputs=[activation_block])
model.compile(optimizer=optimizer(), loss=loss_function)
return model

And this is the helper function

def UnetConv3D(input, outdim, is_batchnorm, name):
x = Conv3D(outdim, (3, 3, 3), strides=(1, 1, 1), kernel_initializer=kinit, padding="same", name=name+'_1', data_format = 'channels_first')(input)
if is_batchnorm:
    x =BatchNormalization(name=name + '_1_bn')(x)
x = Activation('relu',name=name + '_1_act')(x)

x = Conv3D(outdim, (3, 3, 3), strides=(1, 1, 1), kernel_initializer=kinit, padding="same", name=name+'_2', data_format = 'channels_first')(x)
if is_batchnorm:
    x = BatchNormalization(name=name + '_2_bn')(x)
x = Activation('relu', name=name + '_2_act')(x)
return x

And this is the loss function

def weighted_dice_coefficient(y_true, y_pred, axis=(-3, -2, -1), smooth=0.00001):
"""
Weighted dice coefficient. Default axis assumes a "channels first" data structure
:param smooth:
:param y_true:
:param y_pred:
:param axis:
:return:
"""
return K.mean(2. * (K.sum(y_true * y_pred,
                          axis=axis) + smooth/2)/(K.sum(y_true,
                                                        axis=axis) + K.sum(y_pred,
                                                                           axis=axis) + smooth)

My input is (128,128,128), am i doing an obvious mistake? Please let me know if more info needed.

Hi,

When you've written the loss functions, did you calculate with batch data?
E.g. reading your tversky loss implementation, I think you've accidentally summed up everything aggregated over the batch instead of the data points.

    true_pos = K.sum(y_true_pos * y_pred_pos)
    false_neg = K.sum(y_true_pos * (1-y_pred_pos))
    false_pos = K.sum((1-y_true_pos)*y_pred_pos)

When you do this, you sum up every prediction, but you actually want to calculate the loss for each image input.
Let me suggest the following:

def tversky():
    y_true_pos = tf.reshape(y_true, (batch_size, -1))
    # [...]
    true_pos = K.sum(y_true_pos * y_pred_pos, axis=1)
    # [...]

def tversky_loss():
    return K.sum(1 - tversky())

For this reason your focal loss should make no difference to the simple tversky loss, because you raised the aggregated sum to the power of gamma. Instead, I believe you should raise the outputs to the power of gamma before the sum.

Of course if somehow your loss is calculated per a sample, than everything looks fine and keras made an automatic aggregation later, then everything looks fine.

Hi there,
Thanks for sharing your code. I tried to apply your code to my datasets. However, I got very weird loss values (inf or nan). I used attn_reg model and apply focal loss for first three outputs and tversky_loss for final output. I am wondering to know that how can I resolve this issue. I checked my data and try to add gradient clipping to prevent the exploding gradient problem. But, non of them did not work out.

Hello, I was just reading your paper and came across the statement "Both models were optimized using stochastic gradient descent with momentum, using an initial learning rate at 0.01 which decays by 10−6 on every epoch." So I was wondering that if 1.e-6 is actually learning rate decay or is it weight decay and you made a typo. Nevertheless, I did not see any mentions of both of the scenarios so I would be glad if you clarify that, because 1.e-6 decay every epoch is nearly nothing, it goes from 0.01 to 0.00999 in 100 epochs, thanks a lot.

thanks for your work.

I have encountered an issue that when I training the model with my own data, the dsc and val_dsc during training is bigger than 1? if I am right, I think the DSC is for dice coefficient, it should be less than 1. thanks.

The original focal loss paper initialize the bias on the final sigmoid conv layer like so:

init = tf.constant_initializer([-np.log(0.99/0.01)])

so that negative examples will start training with no loss, and positive examples will have very high loss. Did you experiment with this bias setting? I could not find good results with it with FTL.

Hello

Firtst of all thanks for creating this repository. I have a question regarding the use of the of the multi-scale models for segmentation of test set images. I have succesfully trained the model and want to use on my test data. If i understand the model correctly, the output is a list containing 4 arrays of different resolutions (original input size and 3 down-scaled versions, yes?). Do i only use the sigmoid/softmax probability map of the original input sized output, or do i use all 4 and perform some sort of resizing with interpolation and average across in order to get to full benefit of the models?

BR.

hi，your code and paper is help me a lot. and thanks for you open source your code. i have question about your Conv2DTranspose . in your paper Fig2

you concatenate two input,then use 3x3 conv + bn + relu ,and use upconv(you use Conv2DTranspose in code). but in your code , you did'nt use (3x3 conv + bn + relu) after concatenate,and you use it in output. is it the same?

i,m trying to run the code using ISIC 2018 dataset, i,m getting the following error:

File "C:\Users\LaeeqAhmed.spyder-py3\isic_train.py", line 146, in
precision, recall, thresholds = precision_recall_curve(y_true, y_preds)

File "C:\Users\LaeeqAhmed\anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics_ranking.py", line 673, in precision_recall_curve
sample_weight=sample_weight)

File "C:\Users\LaeeqAhmed\anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics_ranking.py", line 536, in _binary_clf_curve
raise ValueError("{0} format is not supported".format(y_type))

ValueError: continuous format is not supported

plz guide me what am i doing wrong here
Regards,

Thanks a lot for this wonderful software! I am wondering under which license is the code released? Would appreciate if you could provide a commercial-friendly license, i.e. MIT or BSD license. Thanks!

Dear Nabila Abraham:
I have a question about your paper, when i run your codes, the ISIC dataset result can be able to reach the scope of the paper dsc: 0.849 ~ 0.863, but the bus dataset B result is uncertain and can't achieve the results in the paper dsc: 0.78 ~ 0.839. Do you konw why?
the paper mention the experiment is 5-fold cross-validation, it says your hyperparameters got through this?

Hey,

I was working through the paper and the code together.
In the Paper in Figure 2, at each level the output of the convolution is passed to the next up convolution and is used for the gating signal.
However, in the code for the first upconvolution this is consistent (using center, which is output of the convolutional block). But for the next levels in the expanding path, there is no 3x3 convolutions applied for the input for the next level and the gating signal, but just previous concatenated "attn*" and "up*".

But for the output at each level, covolutional blocks are applied (not just a single 3x3 convolution):
conv6 = UnetConv2D(up1, 256, is_batchnorm=True, name='conv6')
conv7 = UnetConv2D(up2, 128, is_batchnorm=True, name='conv7')
conv8 = UnetConv2D(up3, 64, is_batchnorm=True, name='conv8')
conv9 = UnetConv2D(up4, 32, is_batchnorm=True, name='conv9')

So, the implementation seems not to be consistent with the Figure for me. I'm totally new to attention networks, so I'm very glad about any help to understand this architecture.

Thanks :)

Thank you very much for your code work.
Where can I download the dataset? Can you give me a download link

Hi! First of all, than you very much for sharing your code. It's a great work!

I'm having troubles when I create gt_train.

The error is: ValueError: could not broadcast input array from shape (7009,32,32,1) into shape (7009)

I use 256x256x1 images and augmented data with Imagedatagenerator.

My code is as follows: (Y_train is ground truth)
gt1 = Y_train[:,::8,::8,:]
gt2 = Y_train[:,::4,::4,:]
gt3 = Y_train[:,::2,::2,:]
gt4 = Y_train
gt_train = [gt1,gt2,gt3,gt4]

train_datagen = ImageDataGenerator(shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True)
mask_datagen = ImageDataGenerator(shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True)
train_datagen.fit(X_train)
mask_datagen.fit(gt_train)

train_set = train_datagen.flow(X_train,seed=seed, batch_size=config.BATCH_SIZE, shuffle=True)
mask_set = mask_datagen.flow(gt_train,seed=seed, batch_size=config.BATCH_SIZE, shuffle=True)

train_generator = zip(train_set, mask_set)

results=model.fit_generator(train_generator, steps_per_epoch...)

Thank you very much in advance!
Best wishes.

Can we use the tversky loss for multiple classes ?
Or do we have to make changes accordingly.

Please tell me whether you have used a batch size of 16 or 8 with ISIC 2018 datatset
Because in paper you have mentioned that you are using batch size of 8 for isic dataset but in code you are using batch size of 16

Hi @nabsabraham,

I'm implementing the model you developped for your paper A NOVEL FOCAL TVERSKY LOSS FUNCTION WITH IMPROVED ATTENTION U-NET FOR LESION SEGMENTATION".

I might be wrong but it seems that the scale_img_2, scale_img_3 and scale_img_4 are created but not used in the subsequent part of the function, which is a bit weird.

You can find below a suggestion for this function. I added the scale_img_2 to the concatenation of input2 and pool1, the scale_img_3 to the concatenation of input3 and pool2, etc. I indicated the modifications with the comments "# HERE".

I hope it was what you had in mind when creating the model. If not, I'd be really grateful if you could explain this part of model to me.

def attn_reg(input_size):
    
    img_input = Input(shape=input_size, name='input_scale1')
    scale_img_2 = AveragePooling2D(pool_size=(2, 2), name='input_scale2')(img_input)
    scale_img_3 = AveragePooling2D(pool_size=(2, 2), name='input_scale3')(scale_img_2)
    scale_img_4 = AveragePooling2D(pool_size=(2, 2), name='input_scale4')(scale_img_3)

    conv1 = UnetConv2D(img_input, 32, is_batchnorm=True, name='conv1')
    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
    
    input2 = Conv2D(64, (3, 3), padding='same', activation='relu', name='conv_scale2')(scale_img_2)
    input2 = concatenate([scale_img_2, input2, pool1], axis=3) # HERE 
    conv2 = UnetConv2D(input2, 64, is_batchnorm=True, name='conv2')
    pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
    
    input3 = Conv2D(128, (3, 3), padding='same', activation='relu', name='conv_scale3')(scale_img_3)
    input3 = concatenate([scale_img_3, input3, pool2], axis=3) # HERE
    conv3 = UnetConv2D(input3, 128, is_batchnorm=True, name='conv3')
    pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
    
    input4 = Conv2D(256, (3, 3), padding='same', activation='relu', name='conv_scale4')(scale_img_4)
    input4 = concatenate([scale_img_4, input4, pool3], axis=3) # HERE
    conv4 = UnetConv2D(input4, 64, is_batchnorm=True, name='conv4')
    pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
        
    center = UnetConv2D(pool4, 512, is_batchnorm=True, name='center')
    
    g1 = UnetGatingSignal(center, is_batchnorm=True, name='g1')
    attn1 = AttnGatingBlock(conv4, g1, 128, '_1')
    up1 = concatenate([Conv2DTranspose(32, (3,3), strides=(2,2), padding='same', activation='relu', kernel_initializer=kinit)(center), attn1], name='up1')

    g2 = UnetGatingSignal(up1, is_batchnorm=True, name='g2')
    attn2 = AttnGatingBlock(conv3, g2, 64, '_2')
    up2 = concatenate([Conv2DTranspose(64, (3,3), strides=(2,2), padding='same', activation='relu', kernel_initializer=kinit)(up1), attn2], name='up2')

    g3 = UnetGatingSignal(up1, is_batchnorm=True, name='g3')
    attn3 = AttnGatingBlock(conv2, g3, 32, '_3')
    up3 = concatenate([Conv2DTranspose(32, (3,3), strides=(2,2), padding='same', activation='relu', kernel_initializer=kinit)(up2), attn3], name='up3')

    up4 = concatenate([Conv2DTranspose(32, (3,3), strides=(2,2), padding='same', activation='relu', kernel_initializer=kinit)(up3), conv1], name='up4')
    
    conv6 = UnetConv2D(up1, 256, is_batchnorm=True, name='conv6')
    conv7 = UnetConv2D(up2, 128, is_batchnorm=True, name='conv7')
    conv8 = UnetConv2D(up3, 64, is_batchnorm=True, name='conv8')
    conv9 = UnetConv2D(up4, 32, is_batchnorm=True, name='conv9')

    out6 = Conv2D(1, (1, 1), activation='sigmoid', name='pred1')(conv6)
    out7 = Conv2D(1, (1, 1), activation='sigmoid', name='pred2')(conv7)
    out8 = Conv2D(1, (1, 1), activation='sigmoid', name='pred3')(conv8)
    out9 = Conv2D(1, (1, 1), activation='sigmoid', name='final')(conv9)

    model = Model(inputs=[img_input], outputs=[out6, out7, out8, out9])

    return model

Hi, I'm curious about the focal_tversky loss can be implemented into a multi class problem?
I saw the model is the sigmoid layer for binary class, how should I modify the loss?

Hi! Very nice work! Thanks for sharing your code!
What's the difference of these two folders: orig_gt, resized-gt?
Thanks for your reply.

Hi, thanks for your work.

I have succeeded to run the code. But when I ran it in a new GPU, the data, the code, all the same. However, during the training, even in the first epoch. The loss is shown with Nan. Do you know why? or any possible of causing this??

Thanks a lot.

Hi

I am trying to use the attn_reg_ds and attn_reg and this is how it looks at the end.

model = Model(inputs=[img_input], outputs=[out6, out7, out8, out9])
 
    loss = {'pred1':loss_function,
            'pred2':loss_function,
            'pred3':loss_function,
            'final': loss_function}
    
    loss_weights = {'pred1':1,
                    'pred2':1,
                    'pred3':1,
                    'final':1}
    # model.compile(optimizer=opt, loss=loss, loss_weights=loss_weights)
    model.compile(optimizer=optimizer(lr = initial_learning_rate), loss=loss)
    return model

and i have modified the function to look like this

def class_tversky(y_true, y_pred):
smooth = 1

    y_true = K.permute_dimensions(y_true, (1,2,3,4,0))
    y_pred = K.permute_dimensions(y_pred, (1,2,3,4,0))

    y_true_pos = K.batch_flatten(y_true)
    y_pred_pos = K.batch_flatten(y_pred)
    true_pos = K.sum(y_true_pos * y_pred_pos, 1)
    false_neg = K.sum(y_true_pos * (1-y_pred_pos), 1)
    false_pos = K.sum((1-y_true_pos)*y_pred_pos, 1)
    alpha = 0.7
    return (true_pos + smooth)/(true_pos + alpha*false_neg + (1-alpha)*false_pos + smooth)

def focal_tversky_loss(y_true,y_pred):
    pt_1 = class_tversky(y_true, y_pred)
    gamma = 0.75
    return K.sum(K.pow((1-pt_1), gamma))

but when i run this i get

DUnetCNN/unet3d/training.py", line 88, in train_model
early_stopping_patience=early_stopping_patience))
File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 2230, in fit_generator
class_weight=class_weight)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1877, in train_on_batch
class_weight=class_weight)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1480, in _standardize_user_data
exception_prefix='target')
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 86, in _standardize_input_data
str(len(data)) + ' arrays: ' + str(data)[:200] + '...')
ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 4 array(s), but instead got the following list of 1 arrays: [array([[[[[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],
...,
[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],

Any idea on what the issue might be?

Thanks

Hi and thanks for open sourcing the code. I have been testing out proposed model attn_reg on images with sizes 128x1024x1. Everything runs well until after a few epochs there's an exception caused by pred1 outputting nan:

Invalid argument:  assertion failed: [predictions must be >= 0] [Condition x >= y did not hold element-wise:] [x (functional_1/pred1/Sigmoid:0) = ] [[[[nan][nan][nan]]]...] [y (Cast_8/x:0) = ] [0]

I believe this happens in a tf.keras.metrics callback. Learning seems to converge but training always ends with this error after 10-50 epochs of 5k images.

Have you seen this or have any idea what could be causing it? The losses are steadily going down so I'm a bit confused as to what's happening..

Hello,
I need some guidance from you, I,m working on aneurysm detection thesis
where I want to segment my image using attention unet,
my plan is:
I,ll train the unet model with my images and labels
test the model using some images

I,m stuck at the point:
how to show the segmented images that are received after applying attention unet on test images

plz guide me!

Regards,

Hi
I am trying to convert this model to 3D, but i am facing severe shape mismatch issues,

1. I am using input size as 128,128,128
2. Is there any intution you follow to build the network, that might make it easier for me to convert

Thanks in advance