xheon / panoptic-reconstruction Goto Github PK

@xheon Hi, nice work, I want to follow your work and use your generated dataset, when will you release the code and dataset?

In lib/modeling/projection/sparse_projection.py line 38, there is a function self.compute_camera2frustum_transform that is called, but there is no corresponding function within the class definition. Is there somewhere else I can find this code in the repo?

edit - is this function intended to compute the extrinsic matrix?

Thank you for sharing your awesome code. I loved the images generated by you in your paper. Do you happen to have any code that I can use to create similar pictures using the .ply files generated by the network?

Excuse me, where can I find the 2D pre training model?

or, how can I train my 2D model myself?

In front3d_train_3d.yaml,

MODEL:
    FIX2D: True
    PRETRAIN: "/cluster_HDD/gondor/mdahnert/panoptic-release/front3d-mask_depth_r18.pth"

I'd appreciate it if you could provide some idea on this problem. Thank you!

Hi, thank you for your outstanding contribution and your great work!

• Would you tell me about how long it will take to train the whole model?

Thanks a lot!

Hi. I wanted to download the model from link in Table, but I can not open the .pth file's link in Table.
Could you please be kind to update the link?

Hi there, thanks for putting the effort on the code release! I wonder if there is a timeline for when the code will be ready?

Hi Manuel, I'm interested in your recent work "Panoptic 3D Scene Reconstruction from a Single RGB Image", and want to try your released code. However, I cannot find the exact file in the code, such as "resources/front3d/train_list_2d.txt". I know maybe the dataset is big, could you provide me with the processed dataset link in dropbox or google drive or the preprocessed code as your paper said? I'm looking forward to your reply. Thanks a lot.
@xheon

the segmentation data(e.g. segmentation_0007 mapped.npz) is all the zero matries, cannot use it to train other model such as Total3DUnderstanding, i will be very appreciated of anyone could solve this problem~👀🙏🙏

Hi, Thanks for your amazing work! I am wondering where can I find your evaluation code for your new metric?

Dear Author,

The config file you provided 'front3d_train_3d.yaml' is for training the 3D network only.
Can you please share full pipeline training configs? Both 2D networks and 3D networks?

Excuse me, could you please opensource the code which can get camera parameter to capture this image. (same viewpoint with input image)

Excuse me, could you please provide me with the sample IDs corresponding to the images in your paper? I am a Master‘s student interested in your work, but while reading the paper, I could not find any information regarding the specific sample IDs associated with each image.

Hey guys,

I just installed this library, but it wasn't as simple as I would have hoped. Here is my process it might help someone else in the future:

At first you download the github repo:

git clone https://github.com/xheon/panoptic-reconstruction.git
cd panoptic-reconstruction

Depending now on the graphics card you use you have to adapt a few things, I used a 3070 for testing this code, so I needed to use at least CUDA 11. That means I changed the following lines in the environment.yaml. I also used a more recent python version:

-    - python>=3.8
+    - python>=3.9
-    - pytorch>=1.7.1
+    - pytorch=1.10.0
-    - cudatoolkit=10.2
+    - cudatoolkit=11.3

Before now installing the environment you might need to make sure that you have support for OpenEXR, on Ubuntu 20.4 this can be installed via:

sudo apt-get install libopenexr-dev
sudo apt-get install openexr

After that we can just go forward and create the environment:

conda env create --file environment.yaml
conda activate panoptic

Now, you need to install some 3rdparty libs, we start with cocodataset API:

cd .. && mkdir 3rdparty && cd 3rdparty
conda install ipython pip
pip install ninja yacs cython matplotlib tqdm opencv-python
export INSTALL_DIR=$PWD
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install

Afterwards, you need to install cityscapescripts:

cd $INSTALL_DIR
git clone https://github.com/mcordts/cityscapesScripts.git
cd cityscapesScripts/
python setup.py build_ext install

Now, it is time to install apex, which is the most difficult to install from this group, if it fails there is a suggested fix for Ubuntu 20.4 below:

cd $INSTALL_DIR
git clone https://github.com/NVIDIA/apex.git
cd apex
python setup.py install --cuda_ext --cpp_ext

This might fail, because your conda environment does not provide nvidias compiler nvcc, on ubuntu 20.4 you can change this via:

sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt update
sudo apt install cuda-11-3 cuda-runtime-11-3 cuda-demo-suite-11-3 cuda-drivers cuda-drivers-510 nvidia-dkms-510 nvidia-driver-510 nvidia-kernel-common-510 libnvidia-extra-510

After this cuda install, you need to reboot and afterwards set the CUDA_HOME and add the nvcc to your path:

export CUDA_HOME="/usr/local/cuda-11.3"
export PATH=$PATH:/usr/local/cuda-11.3/bin

Now you can run the command again, make sure that you first go to your 3rd_party folder:

cd apex
python setup.py install --cuda_ext --cpp_ext

After installing all these packages, you can now install the maskrcnn-benchmark:

cd ..
git clone https://github.com/facebookresearch/maskrcnn-benchmark.git
cd maskrcnn-benchmark
python setup.py build develop

Finally, you need to add the MinkowskiEngine:

cd ..
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
python setup.py install --blas=openblas --force_cuda

After you installed that package you only need to install the panoptic packge itself:

# Install library
cd ../panoptic-reconstruction/
cd lib/csrc/
python setup.py install

I hope this helps someone.

PS: I am not the maintainer of this package and I can not help you if this doesn't work for you.

Best,
Max

Hi

I was wondering if you could release the code for the generation of the unsigned distance field to try your repository on new datasets?

Thank you

Hi. I wanted to train the depth2d part of the model, but it raises the following error:

Traceback (most recent call last):
  File "/home/student/.conda/envs/panoptic_cp/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/student/.conda/envs/panoptic_cp/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/student/data/student/zhouyingquan/panoptic-reconstruction/tools/train_net.py", line 56, in <module>
    main()
  File "/data/student/data/student/zhouyingquan/panoptic-reconstruction/tools/train_net.py", line 51, in main
    trainer.do_train()
  File "/data/student/data/student/zhouyingquan/panoptic-reconstruction/lib/engine/trainer.py", line 91, in do_train
    losses, results = self.model(images, targets)
  File "/home/student/.conda/envs/panoptic_cp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/student/data/student/zhouyingquan/panoptic-reconstruction/lib/modeling/panoptic_reconstruction.py", line 49, in forward
    depth_losses, depth_results = self.depth2d(image_features["blocks"], depth_targets)
  File "/home/student/.conda/envs/panoptic_cp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/student/data/student/zhouyingquan/panoptic-reconstruction/lib/modeling/depth/depth_prediction.py", line 57, in forward
    valid_masks = torch.stack([(depth.depth_map != 0.0).bool() for depth in depth_target], dim=0)
  File "/data/student/data/student/zhouyingquan/panoptic-reconstruction/lib/modeling/depth/depth_prediction.py", line 57, in <listcomp>
    valid_masks = torch.stack([(depth.depth_map != 0.0).bool() for depth in depth_target], dim=0)
AttributeError: 'Tensor' object has no attribute 'depth_map'

I inspected the code and found the following code in lib/modeling/depth/depth_prediction.py:

    def forward(self, features, depth_target) -> ModuleResult:
        depth_pred, depth_feature = self.model(features)
        depth_return = [DepthMap(p_[0].cpu(), t_.get_intrinsic()) for p_, t_ in zip(depth_pred, depth_target)]
        depth_target = torch.stack([target.get_tensor() for target in depth_target]).float().to(config.MODEL.DEVICE).unsqueeze(1)
        ...
        if self.training:
            valid_masks = torch.stack([(depth.depth_map != 0.0).bool() for depth in depth_target], dim=0)

If my observations are correct, in the third line of the function, depth_targets has become a list of Tensors, so the attribute depth_map is lost. Is this a bug? How can I resolve this?

Hi. I wanted to train the model on my own dataset, but I found my cuda memory runs out when processing occupancy_256 prediction. I tried nn.DataParallel to try to run the model on multiple GPUs, but it raises such an error:

AttributeError: 'MinkowskiConvolution' object has no attribute 'dimension'

I searched for this error and found out it was an unresolved issue of MinkowskiEngine (link here). I wonder how you trained the model on your computer, and could you please be kind to inform other possible solutions to make it work? Thank you!

I have downloaded the 3d weighting mask. In each of the 6081 existing scenes there are some .npz files. I cannot understand each of represent which items in that scene? Are they related to the existing rooms to each or not. Because the number of those .npz files are different from existing items in each scene.

Thank you so much, in advance.

I am a fresh man about cv and I found your project very interesting. So I follow the steps to install the enviroments, I used pycharm to open the code, but I have a very basic question that how can I run this project, I run the train_net.py directly but there are some errors.
I know this are very basicly, but I would be very grateful if you could help me.

Thanks for such an amazing work!!! I am now working on semantic scene completion, and wonder whether you can share your processed matterport3d data?

my conda environment is: cuda11.1+pytorch1.9.0,
when running this command: python tesr_net_single_image_.py
it occurs the error: RuntimeError： expected all tensors to be on the same device, but found at least two device, cuda:0 and cpu!
but i took a look at the test_net_single_image.py, there may be nothing wrong with the code, i don't know how to solve it, i will be very appreciated if any one could help me.

Hi! Thanks so much for sharing the code of this amazing work!

I am trying to train some other models on your dataset. I notice that in the data pre-processing step, you flip the depth map here (the operation of [::-1, ::-1]). However, when I visualize the color and depth in the input, I find that they are not aligned after the depth is flipped. Is there any particular reason that you apply this flipping operation?

Besides, could you help provide the name of each semantic label in your dataset? Thank you!

Hello,

I am really impressed with your paper and thank you for making it available on github.

I am having trouble with generating training data for new datasets, could you please share the data generation code?
I saw isssue #7, I guess I need to send an email?

Thank you.

Hi Manuel,I'm a freshman who just started running code recently and interested in your recent work "Panoptic 3D Scene Reconstruction from a Single RGB Image", and want to try your released code. However, I cannot find the exact pretrain model , such as "panoptic_front3d.pth". Could you provide me with the pretrain model link in google drive or baidu cloud? I'm looking forward to your reply. Thanks a lot. @xheon