kid-liet / reg-gan

Breaking the Dilemma of Medical Image-to-image Translation

Get the full paper on Arxiv. This paper has been accepted by NeurIPS 2021 (Spotlight).

Main Reference Environment

1. Linux (Titan RTX)
2. Python (3.6.6)
3. torch (1.9.0+cu111)
4. visdom (0.1.8.9)
5. numpy (1.19.2)
6. skimage (0.15.0)
7. Yaml (5.4.1)
8. cv2 (3.4.2)
9. PIL (8.3.2)

Usage

1. Create dataset
   • train path/A/
   • train path/B/
   • val path/A/
   • val path/B/
2. The default data file form is .npy and normalized to [-1,1].
3. Modify the parameters in the .yaml file as needed:
   • bidirect: whether to use bidirectional network, corresponding to the C or NC mode in the paper.
   • regist: whether the registration network is used, corresponding to the +R mode in the paper.
   • noise_level: set to 0 if you do not want to use noise.
   • port: port parameters of visdom.
4. Default RegGAN mode (bidirect:False regist:True).
5. Start visdom：

python -m visdom.server -p 6019

If other port parameters are used, you need to modify the port in yaml.

6. Train:

python train.py

Trained Weights

We provide Pix2pix, CycleGAN, RegGAN trained weights under the condition of Noise.0: https://drive.google.com/file/d/1xWXB9u6dQ9ZytmgQl_0ph4H_Ivtd41zJ/view?usp=sharing

• Pix2pix_noise0
• CycleGAN_noise0
• RegGAN_noise0

Processed data

We provide some processed data for your convenience： https://drive.google.com/file/d/1PiTzGQEVV7NO4nPaHeQv61WgDxoD76nL/view?usp=sharing

Citation

If you find RegGAN useful in your research, please consider citing:

@inproceedings{
kong2021breaking,
title={Breaking the Dilemma of Medical Image-to-image Translation},
author={Lingke Kong and Chenyu Lian and Detian Huang and ZhenJiang Li and Yanle Hu and Qichao Zhou},
booktitle={Thirty-Fifth Conference on Neural Information Processing Systems},
year={2021},
url={https://openreview.net/forum?id=C0GmZH2RnVR}
}

reg-gan's Issues

Brats slice selection and data processing details

Hi authors,

Thanks for sharing the code and the processed data.
Could you please kindly share with me how you select slices from the original Brats 3D volumes?
I notice that not all slices are included in the processed dataset.

Secondly, do you apply any processing techniques like resampling the 3D resolutions on the volumes?

Thanks.

关于不同模态的数据shape

尊敬的作者您好，我目前正在复现这个代码，但我有一个疑问在apply这个模型在我自己的任务上，如果两个序列的图像层数不同该如何确定生成图像对应的层数，即如何为fake图像找到可供配准的real图像，感谢您一直以来的无私帮助！

关于unpaired datasets 的训练

尊敬的作者，您好！我想请教一下文中所说的刚性配准具体用的哪个算法。“For unpaired datasets, we can conduct rigid registration first in 3D space and then use RegGAN for training.” 这句话的意思是利用刚性配准尽可能的为源域图像找到配对图像后再用于reggan的训练吗？.

About the training details.

Thank for your work, Kid-Liet. Could you please complete the training process and important details? I'm looking forward to doing something about the Reg-GAN.

关于数据集

尊敬的作者您好，我目前尝试使用IXI数据集验证模型有效性，但是注意到您的数据形式为npy，我使用IXI数据集的nii文件转换为npy格式后，输入图片中存在较大到的噪声，请问您是否愿意分享一下您处理数据集的相关代码，感谢您一直以来的无私帮助

processed data

Does the pipeline only support 2D med image?

Hi,

3D data is more common in med imaging. Does this work support 3D volume?

关于您论文中的D.F deformation fields形变图的绘制

尊敬得作者您好，看到您文章中使用了D.F形变图，想请教您一下是如何绘制的

如何将数据归一化[-1,1]（文中提及的）

请问文中提及的归一化数据[-1,1], 是用的什么公式？是 (x-x.mean())/(x.max()-x.min())吗. Thank you in advance.

There are other architectures but not the main code I don't seem to see the code from the RegGAN

数据归一化

您好！请在您的dataset.py中没有看到对数据归一化代码的部分，是不需要吗？

Save the test results

                       Dear Kid-Liet,

I have noticed that there is a "save_deformation" subfunction within CycTrainer.py. Please, what is it for? Besides, how can CycTrainer.py be modified in such a way that the results can be saved during testing. I am sorry, I am not very familiar with Pytorch.

Thank you for your time and help.

Kind regards

Hello, I have a question.

I saw your good reg-net code.

But there's no code to run 'regTrainer.py', so can you upload it T^T?

I'd appreciate it if you upload it.

您好啊, 我想请问一下代码中的Ymal文件各自对应的是做什么任务呢?

比如Unit.yaml等分别是为了实现什么任务呢?

Should network R be used in the test phase?

Great Work! I read your paper and realize your goal is to optimize three networks G, D, and R through L1loss(you named correction loss), SMloss, and adversarial loss.

But there is one thing in the test phase that confuses me. Which of G(x) and R(G(x)) should I use as the final generated image? I noticed a test phase in your code that only use G(x) as the final translated image. Is it correct for Reg-GAN that use G(x) rather than R(G(x)) as the translated image?

If there is some mistake in my understanding of RegGAN, hope you can point it out.

CUDA version

                    Dear author,

I hope you are having a productive day. Congratulations on your great work, and thank you very much for making the source codes publicly accessible! As you stated in the environment requirements, one has to install Cuda 11.1 to be able to run the codes. Unfortunately, I am having an older Cuda version (10.0) on my server, and the codes do not work with this version. Besides, I am running the codes on a server that is being used by many people and I am not the Admin, which makes it quite impossible for me to upgrade the Cuda version. Please, is there any way to adjust the codes in such a way that they can be run on an environment with older Cuda version?

Thank you very much for your time and help!

With kind regards

关于数据集

尊敬的作者您好，我目前尝试使用IXI数据集验证模型有效性，但是注意到您的数据形式为npy，我使用IXI数据集的nii文件转换为npy格式后，输入图片中存在较大到的噪声，请问您是否愿意分享一下您处理数据集的相关代码，感谢您一直以来的无私帮助

关于数据预处理

作者大大你好哇，我用您的代码跑自己的数据，我用的是头部磁共振的数据，但是在加载数据集的代码有点疑问：

[1] read_dicom 和 read_ori_w函数有啥区别？就是read_ori_w调整了窗宽窗位吗？
[2] 我打印了read_dicom的倒数两行，值应该落在-1到1，但是大多都是-1到0的范围，会不会影响网络训练？除以4095是因为dicom的Bitstrore=12，所以取值范围是0-4095吗？

image2=image2/4095 # BitStore=12 2^12=4096 print("read_dicom: image2=", np.min(image2), np.max(image2)) image2 = (image2 - 0.5)/0.5 # 后处理:图像像素值被标准化到-1到1的范围内，为了后续处理或显示做准备。 print("read_dicom: image2=", np.min(image2), np.max(image2))
打印的值是：
read_dicom: image2= 0.0017094017094017094 0.0862026862026862
read_dicom: image2= -0.9965811965811966 -0.8275946275946275

read_dicom: image2= 0.0 0.09401709401709402
read_dicom: image2= -1.0 -0.811965811965812

read_dicom: image2= 0.0009768009768009768 0.06275946275946276
read_dicom: image2= -0.998046398046398 -0.8744810744810745

read_dicom: image2= 0.0 0.09987789987789988
read_dicom: image2= -1.0 -0.8002442002442003

read_dicom: image2= 0.0014652014652014652 0.07814407814407814
read_dicom: image2= -0.9970695970695971 -0.8437118437118437

read_dicom: image2= 0.0014652014652014652 0.09255189255189256
read_dicom: image2= -0.9970695970695971 -0.8148962148962149

[3]在read_ori_w为什么要加10242？center 和width 的值是怎么确定下来的呀？
`
def read_ori_w(file_path):#read_dicom_mw
file_path=file_path.replace('../../../', '../../')

dicom = sitk.ReadImage(file_path)
data1 = np.squeeze(sitk.GetArrayFromImage(dicom))
data=data1+1024

center =50# ds.WindowCenter 50
width = 400#ds.WindowWidth # 400

win_min = (2 * center - width) / 2.0 + 0.5
win_max = (2 * center + width) / 2.0 + 0.5
dFactor = 255.0 / (win_max - win_min)
image = data1 - win_min #sitk读取的数值比pydicom读取的数值小1024

image1 = np.trunc(image * dFactor)#dFactor
image1[image1>255]=255
image1[image1<0]=0
image1=image1/255#np.uint8(image)
image1 = (image1 - 0.5)/0.5

image2=data#sitk读取的数值比pydicom读取的数值小1024
image2[image2<0]=0#-2000->0
image2=image2/4095
image2 = (image2 - 0.5)/0.5

# image1=(image1*2-1)*255
# image2=(image2*2-1)*255
# plt.subplot(2, 2, 1)
# plt.imshow(image1*255, cmap='gray')#,vmin=0,vmax=255
# plt.subplot(2, 2, 2)
# plt.imshow(image2*255, cmap='gray')#,vmin=0,vmax=255
# plt.show()

return image1,image2

`

关于RegGan中无cycle加配准模型的判别器训练问题？

尊敬的作者您好，在复现你的代码的过程中，我注意到即使在无cycle的条件下也需要用判别器去判别生成器生成的效果，但是我不太清除是否需要预训练判别器使其产生判别真实图像与生成图像得能力？代码文件CycTrainer.py中230行的adv_loss = self.config['Adv_lamda'] * self.MSE_loss(pred_fake0, self.target_real)此处用到判别器。

关于BtoA方向的训练

尊敬的作者您好，我已经完成了t1到t2模态的训练，也就是AtoB，但是注意到您的代码中没有关于t2生成t1的具体描述，想请问一下您该如何操作，感谢您的帮助，祝您工作顺利！

are you really using the decay in learning rate?

Hello Author (@Kid-Liet),
Thanks for your clean implementation.

I am curious to know if you are really using the weight decay here in your implementation (NC+R)? if so in which line (can you please point me to that line)? Though I can see you have set decay_epoch=20 but I can not see its usage in CycTrainer.py file.

Also in your paper you have mentioned that the batch size was set too 1 with weight decay 0.0001. what do you mean by this line?

Any reason not using larger batch size?

Hi,

I notice in the paper and also the code that batch size was set to 1 in the experiment. Any reason not using a larger batch size? Thank you!

你好，可否总结一下每个包的版本呢？？最近在配置环境总是遇到不兼容的问题

数据集数量

作者您好，brats2018的训练集有285个病例，z轴数量为155，那么一共能产生44175个npy文件，但是文中提到训练集为8457，请问这个数据集是做了删减吗？或者是什么其它处理？麻烦您了，新手上路困难好多，如果您能告知一下我将万分感激。

Test Phase Output

Hello,
first of all thank you that you shared your code with us.
I am currently trying to run your code on custom dataset. However, I cannot get the results that is expected according to original paper. My dataset consists of RGB images with some noise (input images) and target images that doesn't contain any noise. I am trying to regenerate noisy images to get target images, but test results are really different. Even though I tried so hard, I couldn't find where I make a mistake? Can you help me?
Here is my input A, target B and test result output, respectively;

Can you provided the pretrained weight ?

Hello~Could you provide the pretrained weight of the three different model?

about how to process my own dataset for this project

Can you please describe the features in the dataset processing process, I am training with my own dataset, but the dataset format is not correct. I've converted the png image to npy format, but I can't understand why the original dataset is two-dimensional

WHY Registration after translation?

Dear Author,

I have a question relating your proposed framework when tackling misaligned data.

You proposed to introduce the registration network after translating the image, why you suggest this way? I think misaligned image can be firstly registered to provide aligned data and then perform aligned I2I translation.

I look forward to your answer.

About the dataset used in paper.

作者您好，方便提供一下您论文实验中使用的BraTS2018数据集吗？官方提供的数据为原始的nii数据，希望能直接在您使用的.npy数据集上直接做实验，这样能更好地复现。:relaxed:

关于Ugat_Trainer以及其模型

尊敬的作者您好：
我在工程文件里注意到您没有公布ugat_trainer以及其模型的代码，请问您是否还保存这个模型的代码，希望您可以提供一些帮助。

Complete steps for custom dataset

Hi, thanks for sharing your code, but I am unable to replicate your results. Could you please include the complete sequence of steps for installation of the environment and execution of the inference with your weights and dataset?

Further, not everyone has access to a system with a GPU so it would be great of you could test your code on a platform like Colab.