hanna-xu / murf

MURF

Code for "MURF: Mutually Reinforcing Multi-modal Image Registration and Fusion" (IEEE TPAMI 2023).

Recommended Environment:

python=3.6
tensorflow-gpu=1.14.0
numpy=1.19
scikit-image=0.17.2
pillow=8.2

Task #1: Shared information extraction

To train:

• Download the training data:
  • RGB-IR (created with RoadScene dataset)
  • RGB-NIR (created with VIS-NIR Scene dataset)
  • PET-MRI (created with Harvard dataset)
  • CT-MRI (created with Harvard dataset)
  • or create your training dataset according to it
    
• Run CUDA_VISIBLE_DEVICES=0 python main.py

To test:

• Put the test data in ./test_imgs/
  
• Run CUDA_VISIBLE_DEVICES=0 python test.py
  

Task #2: Multi-scale coarse registration

• This task is based on Task #1, so the code and models in task #1 should be downloaded and prepared in advance.

To train:

• Download the training data: RGB-IR, RGB-NIR, PET-MRI, CT-MRI or create your training dataset.
• Adjust task1_model_path in main.py to the path where you store the model in task #1.
• Run CUDA_VISIBLE_DEVICES=0,1 python main.py
  

In some tasks:

• Put more large-resolution training images in ./large_images_for_training/
• Finetune the trained model with large-resolution images by running CUDA_VISIBLE_DEVICES=0,1 python finetuning.py

To test:

• Prepare test data (one of the two ways):
  • Put the test images in ./test_data/images/ or
  • Put the test data (including images and landmark) in ./test_data/LM/ in .mat format
    
• Run test code:
  • CUDA_VISIBLE_DEVICES=0 python test.py or
  • CUDA_VISIBLE_DEVICES=0,1 python test.py or
  • CUDA_VISIBLE_DEVICES=0,1 python test_w_finetuning.py

Task #3: Fine registration and fusion

To train:

• Download the training data (same as that in Task #1 and the non-rigid deformation is applied subsequently)
• Run CUDA_VISIBLE_DEVICES=0 python main.py

In some tasks:

• Put more large-resolution training images in ./large_images_for_training/
• Finetune the trained model with large-resolution images by running CUDA_VISIBLE_DEVICES=0 python finetuning.py

To test:

• Put the test data in ./test_imgs/
  
• Run CUDA_VISIBLE_DEVICES=0 python test.py
  

The previous version of this work:

@inproceedings{xu2022rfnet,
  title={Rfnet: Unsupervised network for mutually reinforcing multi-modal image registration and fusion},
  author={Xu, Han and Ma, Jiayi and Yuan, Jiteng and Le, Zhuliang and Liu, Wei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19679--19688},
  year={2022}
}

murf's Issues

Can you provide test sets?

问题请教。

你好，关注这篇文章很久了，终于等到代码公布，在学习代码的时候几个问题想请教一下：
1、不同的任务task1，task2，task3用的数据是同一个么？ 比如我跑RGB-NIR 我一直是用part1和part2这两个数据吗？
2、不同的task的用的数据集名称不一样，是从网盘下载数据后改名吗？
3、我看了数据集的图片，好像不是配对的呀。
如有打扰，还请见谅。谢谢。

pytorch

请问有没有基于pytorch的版本

Is this the code for the CVPR2022 paper?

Hello, thank you for the code!
But is this the code for the CVPR2022 work "RFNet: Unsupervised Network for Mutually Reinforcing Multi-modal Image Registration and Fusion"? I notice there are several differences between this code and the paper (for example, the code uses contrast learning instead of the modal transfer net in the paper). Or is this just a code for another work?
Sincerely looking forward to your reply, thank you!

GDrive Link

Hello,

Thanks for your excellent work. Could you please add Google Drive link of your datasets?

Mojtaba

code works finally but loss keep nan

main.py and test.py could run but loss kept nan
well, just offer an env reference here:

cuda 10.0
cudnn 7.6.4
tf-gpu 1.15

is there anyone through this project?
Asking for help

What tools are used to mark the quantitative analysis of registration?

Cannot convert a symbolic Tensor (f2m_net/meshgrid/Size_1:0) to a numpy array.

Environment and Package Version Issue

Sorry to bother you, but i dont see any information about the environment you use.

Could you give me a brief introduction to mainly used packages (e.g. tensorflow) and their versions?