Regarding pretrained weights, reproducing results about cross_attention_renderer

[yilundu]

Yes the attached network should be trained on the full dataset.

[yilundu]

Hi,

Can you post the error you are getting when loading the pretrained weights as well as the associated command you are running?

Unfortunately, using a small subset of the dataset will get a decent bit poorer results -- but should still result in something that can render intermediate views.

Best,
Yilun

[SunghwanHong]

Thanks for the quick reply!

Sure I will attach the screenshot very soon.
In the meanwhile, so when you trained the network, did you use the full dataset?

[SunghwanHong]

This is what I got when I just ran the code without changing anything and with this command :
python experiment_scripts/eval_realestate10k.py --experiment_name vis_realestate --batch_size 3 --gpus 1 --checkpoint_path logs/realestate/checkpoints/model_current.pth

After I change the --views to 2 (which was 1 for the default value),

I got this error :

I'm guessing it's the number of token size that causes the problem. Maybe I should do something to the image size?

[yilundu]

Can you add the argument --views=2 to the command you are running?

[SunghwanHong]

I still get the same error

Probably because I already changed it to 2 in the codes

[yilundu]

Hmm that's a bit bizarre -- can you confirm that the colab runs normally for you? Can you also try to make a fresh install of the repo? I've never seen the above error before

[SunghwanHong]

actually, I tried to run the training codes, it works perfectly fine.
Hmm.. maybe the weights were trained using higher resolution? I'm not sure.. Because the error is when loading the positional embedding, It should be related to image size

[yilundu]

Hmm what images are you using at the test resolution? I don't think that should be the cause -- in principle the weights should be initialized independent of image resolution. If the model arguments are the same as training, they should be loaded correctly. You can also check to the colab to see how the weights of the model should be loaded.

[SunghwanHong]

Oh right, I will take a look at those parts. Thanks!

One last question is that does 'cvpr2023_wide_baseline_data.tar.gz' file contains poses for subset of ACID and RealEstate or poses for full ACID and RealEstate?

From the codes, it seems that we need .mat pose files, but from the generate_ACID or RealEstate codes, I can't find lines that convert txt -> mat. Maybe I'm missing something.

[yilundu]

Cool thanks! Feel free to send another message if there are any additional problems. And let me know what changes to the code (if any) were needed to get things to run. I can also try running the evaluation code later on my end.

The tar.gz file should have poses for the full ACID and RealEstate.