Add depth and object segmentations about matterport3dsimulator

[peteanderson80]

We have added depth outputs. See the README for details, these are based on the undistorted depth images which are not necessarily perfectly aligned to the skyboxes. I haven't got any immediate plans to do segmentations, but happy to take pull requests :)

[peteanderson80]

My expectation is that projecting cnn features to the ground plane would be fine. The location of cnn features in the image is quite coarse anyway (with large receptive fields), and I suspect that depth alignment is mostly off by only a couple of pixels. It might be a bigger issue if the depth maps where being used to generate rgb pointclouds, because the rgb and the xyz might be misaligned a bit.

[suhangpro]

Any plan on when these will be implemented? Thanks for sharing the awesome work!

[jotaf98]

I'd like to second the request for depth/segmentation!

If I understand correctly, the depth images in the Matterport3D dataset are not aligned with the RGB skybox images. On the other hand, the RGB images that are aligned with the depth images result in panoramas with seams (see discussion in readme here, "Matterport3D support" section). Any idea how to handle this?

[peteanderson80]

Hi @jotaf98, I enjoyed your MapNet talk, very cool.

Depth / segm is a priority so we will get this done asap. I think we will want to do some color correction and image blending on top of the RGB stitching, to remove the seams. I guess we can just use any near to state of the art codebase for this - I'm not super familiar with this literature so happy to take suggestions on which method to use...

[srama2512]

Hi @peteanderson80,

Will you be providing an update to incorporate depth map and segmentations soon? If not, can you give me an idea of how easy/hard this is to do and any suggested starting points?

[srama2512]

Thank you for the update! Could you comment on how the imperfect alignment would affect projection of RGB to ground-plane? More specifically, I'm looking to use MapNet on M3D. Will it significantly affect the projection of features from image-plane to ground-plane?

[yellowjs0304]

@peteanderson80
Hi, Thank you for sharing nice work.
Is there any plan about object segmentation?
I really want to do object detection with your simulator.