sunset1995 / pytorch-layoutnet Goto Github PK

Why?

I have followed the exact steps as mentioned in the readme to visualize the layout for the demo image in the 3D Viewer.

The boundary maps and corner maps outputted by the model are precise as seen below.

However, there is an issue of misaligned axes and hence the output looks something like this.

Following is the exact command used for 3D visualization.
python3 visual_3d_layout.py --ignore_ceiling --img assert/demo.png --layout assert/output/demo_aligned_rgb_cor_id.txt

It looks to me the network training has no flip or rotation, why they are added as evaluation? Thanks

My current laptop has a 4 GB graphics card due to which I am unable to train the model on it. Can you please tell me how should I go about making the changes in order to run it on my machine. Thank you!

Hi

Thank you for the implementation. It's very helpful.

I was wondering why do you start with very low learning rate (1e-6 at warmup) and then change to 0.018 later.

Isn't that supposed to be other way around? Starting with high learning-rate and then lowering it?

python torch2pytorch_data.py 
Traceback (most recent call last):
  File "torch2pytorch_data.py", line 83, in <module>
    cvt2png(os.path.join(DATA_DIR, 'train'), train_pats, train_pano_map)
  File "torch2pytorch_data.py", line 37, in cvt2png
    assert os.path.isfile(th_path), '%s not found !!!' % th_path
AssertionError: /home/anish/Downloads/dataset/halfc/layout/pytorch-layoutnet/data/origin/data/panoContext_img_train.t7 not found !!!

Please see the output from the 3d viewer above. It is outputting some totally random noise for some reason. I'm running it on a macbook.

I can't read any data use ' imgs = torchfile.load(th_path)' , and when I use 'imgs = load_lua(th_path).numpy()',raise warning:ModuleNotFoundError: No module named 'torch.utils.serialization'。 find serialization is not exist

Hi @sunset1995 , I know you are not planning to add support for perspective images for this repo. But do you by any chance know how one should write the perspective image version of function panoEdgeDetection?

Thank you!

Hello
First, thanks for your sharing .
We want to use new indoor panorama samples for training, but how to annotate panorama as shown in the appendix becomes a trouble for us. Could you share the method or some tools to do the work?
Thank you!

Is there any way to use this logic to make a 3d box from single RGB image of indoor room.. (without panorama).

like of this image

Regards

Could you please explain the steps that would be required to implement the Manhattan layout optimiser in the same repo?

I have some features that I extract from the original panorama. I want to calculate the coordinates of those feature points after alignment is done in the preprocessing step. Could you please advise. I did not find a function anywhere.

panoContext_img_train.t7      : 414 examples
Traceback (most recent call last):
  File "torch2pytorch_data.py", line 83, in <module>
    cvt2png(os.path.join(DATA_DIR, 'train'), train_pats, train_pano_map)
  File "torch2pytorch_data.py", line 65, in cvt2png
    imgs = load_lua(th_path).numpy()
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 608, in load_lua
    return reader.read()
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 593, in read
    return self.read_object()
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 523, in wrapper
    result = fn(self, *args, **kwargs)
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 546, in read_object
    return reader_registry[cls_name](self, version)
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 147, in read_tensor
    storage = reader.read()
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 593, in read
    return self.read_object()
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 523, in wrapper
    result = fn(self, *args, **kwargs)
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 546, in read_object
    return reader_registry[cls_name](self, version)
  File "/opt/anaconda3/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 166, in read_storage
    return python_class.from_buffer(reader.f.read(size), 'native')
RuntimeError: $ Torch: not enough memory: you tried to allocate 4GB. Buy new RAM! at /pytorch/aten/src/TH/THGeneral.cpp:204

I wanted to try experimenting with the models with my own panoramic images, however, I had some doubts regarding how to go about capturing these images.

Would a phone camera(I am using iPhone) be appropriate for capturing the images or is some special camera required?
I have noticed that the images taken on the phone are not of the same aspect ratio.
Moreover, it is difficult to capture the whole height of the room from floor to ceiling.
Lastly, the images shot cannot cover the entire perimeter of the room.

Here, is an example image. Resolution: 11612x3628 and hence, the aspect ratio is also completely different than 2:1 used by the model.

Hello :)
I've just started trying to use your repository.
I've done the following steps:

1. I've built the directory tree as you guided.
2. I've ran python torch2pytorch_data.py successfully
3. I've ran python torch2pytorch_pretrained_weight.py --torch_pretrained ckpt/panofull_joint_box_pretrained.t7 --encoder ckpt/pre_full_encoder.pth --edg_decoder ckpt/pre_full_edg_decoder.pth --cor_decoder ckpt/pre_full_cor_decoder.pth
   which in turn resulted in the following error:
   Traceback (most recent call last):
   File "torch2pytorch_pretrained_weight.py", line 2, in
   import torchfile
   ImportError: No module named 'torchfile'

And the solution was to install torchfile :)

I've done the following steps:

1. Built the directory tree as you instructed
2. ran python torch2pytorch_data.py annd python torch2pytorch_pretrained_weight.py with their instructed parameters
   But when i run 'python visual.py --flip --rotate 0.25 0.5 0.75 --img_glob "data/test/img/pano_aaccxxpwmsdgvj.png" --line_glob "data/test/line/pano_aaccxxpwmsdgvj.png" --output_dir assert/'
   I get the following error

Traceback (most recent call last):
File ".../LayoutNet/visual.py", line 55, in
encoder.load_state_dict(torch.load('%s_encoder.pth' % args.path_prefix))
File ".../lib/python3.5/site-packages/torch/serialization.py", line 356, in load
f = open(f, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'ckpt/pre_encoder.pth'

When it comes to Warp floor and ceiling,After creating the thread, there is no result or error. What is the reason? Is it just waiting for a long time? Line 87 of visual_ 3d_ layout.py

    with Pool(n_thread) as p:
        sel = p.map(functools.partial(pt_in_poly, xy_poly), floor_xyz[:, :2])

    return floor_texture[sel], floor_xyz[sel], ceil_texture[sel], ceil_xyz[sel]

Hi

Thank you very much for sharing
I've read through your code (and was very impressed!) and collected several questions about difference from the original implementation.
I'm wondering whether I'm right and the differences exist, and why you have chosen to implement it this way

1. The learning rate that you use is very different. You use ~0.018 (after warmup), while the original work uses 0.0001 (times 5, because they multiply by 5 where the mask is true, while you divide by 5 where it is false). That is a very significant factor of 36... am I missing something? Maybe something is different between torch in lua and pytorch?

2. The original work first trains only on edges, and then trains both on edges and on corners. Why have you chosen to avoid this?

3. You haven't implemented the gamma augmentation (But I'm quite sure that it is not very significant)

4. You use weight decay. Why?

5. You use learning rate decay. Why?

6. The original work clamped the gradients at 10. You have clipped the gradients' norm (Not the individual gradients) at 3. Any reason you have chosen to change this value?

The questions obviously don't criticize your work, just trying to understand the differences and get the best of both worlds...

Thanks

I used One Plus smartphone to click panorama picture of room.And used it as input image and the results turned out to be so different .I used the pre-trained model of PanaStd2d-3D to check the result.

Can someone suggest me how to proceed using smartphone panorama to 3D room layout ?

Thanks in advance

When use torchfile to load the panoContext_img_train.t7, the result is empty。

th_path = "D:\program\pytorch-layoutnet\data\origin\data\panoContext_img_train.t7"
imgs = torchfile.load(th_path)
print(imgs.shape)
(0,)

Hello.
Thanks for your sharing .
I read your paper “ 360-Indoor: Towards Learning Real-World Objects in 360◦ Indoor Equirectangular Images”.But how to annotate panorama as shown in the appendix becomes a trouble for us. Could you share the method or some tools to do the work?

I want use this model to train a model on LSUN dataset. Where is the dataloader for perspective image? Thank you！

Dear author,
First of all, thank you for sharing your work. The project has been incredibly helpful for learning. However, the CKPT file has become invalid. When I click on the provided link, it shows that the file does not exist. Unfortunately, this prevents me from verifying its effectiveness at the moment. I kindly request your assistance in resolving this issue. Thank you!

The corresponding location is as follows:
Download the trained model from here (350M). Put the 3 files extracted from the downloaded zip under ckpt/ folder.