cassiepython / clipnerf Goto Github PK

Hey Cassie!
Will there be a code release for the ClipNerf. I am interested specifically in part 1 training.

hi,
how to solve this problem?

If I am not missing anything, there currently is no shape editing support as the paper suggests. How would shape editing fit in this pipeline?

Thanks.

Hi, according to your code the original nerf training process, one would randomly choose a batch of rays (say 1024) in the training process and compare this to the ground truth pixel-wise values of the ground truth image sampled by the same set of rays. So the image sent to compute the clip loss is just a batch of random pixels, without any semantic information. Is my understanding correct? And if so, why would it be possible to compare this 'image' to the input prompt?

This is the image sent to the clip loss during your training process.

Hi~ Thanks for your excellent work!

But when I read the open source code, I find it does not match the description in the paper, e.g. first train the disentangled conditional NeRF including the conditional NeRF generator and the deformation network; then we fix the weights of the generator and train the CLIP manipulation parts

I think that it just finetunes pretrained NeRF using clip. Could you please open-source the code described in the paper? Thank you very much.

Thanks for the nice work!
Is there a plan to release the code?
I'm really looking forward to it!

for how long was the model trained until the color of the lego changes?

Hi, in your paper you described the training data of your work are Photoshapes and Carla. However, these dataset is not originally designed to be used in nerf, I was wondering if you could show us the data preprocessing code of Carla so that we could make some comparsion with your work on that. Thank you.

Thanks for the great work. I have a question, since it is a feed-forward framework, I think you may need the text in training. How to get it? I think there are no texts in both of the two datasets. Thanks!

Hi cassie,
thank you so much for your amazing works! i have encountered some problems when running the llff data set. It said that RuntimeError: shape '[60, 60, -1]' is invalid for input of size 3072. how can i fix this problem? Look forward for your replying. thanks again!
Traceback (most recent call last):
File "run_nerf_clip.py", line 885, in
train()
File "run_nerf_clip.py", line 805, in train
rgb_img = rgb.view(sample_scale, sample_scale, -1)
RuntimeError: shape '[60, 60, -1]' is invalid for input of size 3072

I follow the default setting of ClipNerf on one RTX TiTian X with cuda out of memory.

I run successfully standard nerf on TiTian X with nerf-pytorch, which is similar to ColorNerf.

Then I change mse loss to gray mse loss and add Clip loss on nerf-pytorch in the way described in ClipNerf, which also can successfully run on one Titian X.

Is there anything wrong?

Thank you for your time.

Thank you for nice work of your paper and i have huge interest about your work!

In released code, i can not find your idea of Training strategy, like two stage training with Discriminator
Could you please release open-source of the paper??

It seems that the version of my kornia is not the same as the one you use, so can you tell me your kornia version?

Hello!First of all,congratulations on such an amazing paper and thank you very much for making the code public.I have a question regarding the training parameters settings for the red excavator.I set use_alpha,use_feature and use_view parameters. sample_scale=45. I loaded the model you gave with the trained model and trained to 250000 iterations with a single 2080Ti GPU.But the results are not ideal, could you please share the training parameters of red excavator. Thanks you for your attention.

Hello. First of all, congrats for accepting in CVPR 2022 😄

Just minor issue...
For project page link in README.md, it links to your other project (https://cassiepython.github.io/cddfm3d/index).

Again, thank you for your inspirational research 👍🏻

First of all, I'd like to thank you for sharing this amazing work to all!

I'm pretty interested in using your code. However, when I try to run your code, it shows no module named 'clip'.

Here are the files that clip is used.

I was wondering if there's some extra software to install, or you haven't finished cleaning and uploading the code for CLIPNeRF? Thanks a lot!

Thanks for your excellent work on language guided NeRF editting! The results are really amazing!
Since two months have passed since you last answered this quetion, I'm wondering if you are now ready to release the main part of your code.
I'm quite excited to try it out! Looking forward to your reply!

Hi, Cassie,

Thank you so much for your amazing job! I'm wondering if that possible for you to release the full code. I'm really looking forward to trying it! Thanks again!

Clip-NeRF is a great work, I tried to reproduce some results recently, but can not get reasoable results. I can only get the map like this:

Hi, I have encountered a very slow training, it took me about 10 seconds to finish an iteration. The clip process costs the most of the time. My training script is directly copied from your README.md without any changes, and I checked the CLIP model, input image and text are all on cuda. Is this speed normal? Is there any way to speed up this process?