yanivbenny / mrnet

Code for "Multi-scale Abstract Reasoning" paper

MRNet - Multi-scale Reasoning Network

Official repository for:

Yaniv Benny, Niv Pekar, Lior Wolf. "Scale-Localized Abstract Reasoning". CVPR 2021.

Requirements

• python 3.6
• NVIDIA GPU with CUDA 10.0+ capability
• tqdm, PyYaml
• numpy, scipy, matplotlib, scikit-image
• torch==1.7.1, torchvision==0.8.2

Data

• PGM
• RAVEN
• RAVEN-FAIR (Our new version of RAVEN)

Code

Optional:

• To speedup training, try running save_cache.py in advance.
  This script will basically save the dataset after resizing all the images from 160x160 to 80x80 in a separate location so that this won't have to be done during runtime.
  This will reduce a lot of CPU utilization and disk reads during training.
  $ python save_cache.py --data_dir <PATH-TO-DATASETS --dataset <DATASET>
  If you have done this step, add --use_cache to the training command.

To reproduce the results, run:

1. First training
   $ CUDA_VISIBLE_DEVICES=0 python train.py --dataset <DATASET> --data_dir <PATH-TO-DATASETS> --wd <WD> --multihead
2. When first training is done
   $ CUDA_VISIBLE_DEVICES=0 python train.py --dataset <DATASET> --data_dir <PATH-TO-DATASETS> --wd <WD> --recovery --multihead --multihead_mode eprob

• For PGM use WD=0. For RAVEN-like use WD=1e-5.

To run test only, add --recovery --test to the command.

Pretrained models

Download the pretrained models for PGM and RAVEN-FAIR here.
Put the model inside a folder <EXP-DIR>/<EXP-NAME>/save and specify --exp_dir <EXP-DIR> --exp_name <EXP-NAME> --recovery --test

Citation

We thank you for showing interest in our work. If our work was beneficial for you, please consider citing us using:

@inproceedings{benny2021scale,
  title={Scale-localized abstract reasoning},
  author={Benny, Yaniv and Pekar, Niv and Wolf, Lior},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={12557--12565},
  year={2021}
}

If you have any question, please feel free to contact us.

mrnet's Issues

Accuracy on RAVEN-FAIR dataset is higher(98%) than that in the paper(88%)

I downloaded the pre training model you provided on Google disk and generated raven-fair dataset according to the project on GitHub

The accuracy of the first epoch trained from the breakpoint has been significantly improved

The accuracy of the follow-up results is much higher than that in the paper.

The performence of training from scratch on RAVEN-FAIR is similar.

The results of the original Raven dataset training from scratch are consistent with the paper, but raven-fair is directly generated by GitHub, and there should be no problem, so I can't judge where there is a problem.

Therefore, I would like to consult you. Thank you

Reproduction results on PGM dataset are poor

Greetings! I have encountered some problems while using your code to reproduce.

I used your mrnet project on GitHub and trained on RAVEN-FAIR dataset and PGM dataset. RAVEN-FAIR dataset is generated by your another GitHub project, and PGM dataset we used only contains neutral region.

We've tried two ways - training from the beginning and training from your pretrained model.

The results of
training from the beginning in RAVEN-FAIR
pretrained model in RAVEN-FAIR
pretrained model in PGM
are almost the same as those in the paper.
However, the result of training from beginning in PGM dataset is very poor, and the accuracy is only 54.14%. Therefore, I would like to ask you the possible reasons, especially whether there are some problems with parameters?

For the two datasets, except that the WD you mentioned on GitHub is 1e-6 and 1e-5 respectively, we have not modified other parameters and use the default values in the project. The specific parameters when we train PGM dataset are as follows:
beta1=0.9
beta2=0.999
contrast=False
dataset='PGM'
debug=False
dropout=False
early_ stopping=10
epochs=-1
epsilon=1e-08
flip=False
force_ bias=False
img_ size=80
levels='111'
loss_ func='contrast'
lr=0.001
meta_ beta=0.0
model_ name='mrnet'
multihead=True multihead_ mode=None
multihead_ w=1.0
r_ func='dist'
ratio=None, recovery=False
regime=None
relu_ before_ reduce=False
row_ col=True, seed=12345
subset=None
test=False
testname='PGM'
use_ tag=1
wd=1e-06
weighted_ loss=False

I would appreciate your help:）