michaelramamonjisoa / sharpnet

SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation

SharpNet

Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation, by Michaël Ramamonjisoa and Vincent Lepetit.

Link to the paper: here

Installation

Make sure you have installed the following requirements:

• Python3
• Pytorch
• OpenCV
• numpy, imageio, Pillow, matplotlib

Optional (for training):

• TensorboardX
• skimage

Clone the repository and download the trained weights:

• on PBRS
• after finetuning on NYUv2

Weights trained on NYUv2 should be used for depth estimation, however weights trained on synthetic data only provide sharper normals and contours predictions.

git clone https://github.com/MichaelRamamonjisoa/SharpNet.git
cd SharpNet
mkdir models && cd models

Put the trained weights in the models/ directory.

Demo

On your test image

Try the demo.py script to test our network on your image :

python3 demo.py --image $YOURIMAGEPATH \
--cuda CUDA_DEVICE_ID\
--model models/final_checkpoint_NYU.pth \
--normals \
--depth \
--boundary \
--bias \
--scale SCALEFACTOR 

The network was trained using 640x480 images, therefore better results might be observed after rescaling the image with SCALEFACTOR different than 1.

Here is what you can get on your test image:

If you want to display the predictions, use the --display flag.

Live demo

To run the live version of SharpNet, connect a camera and run demo.py with the --live flag.

• Make sure your camera is detected by OpenCV beforehand.
• Press 'R' on your keyboard to switch between normals, depth, contours and RGB
• Press 'T' to save your image and its predictions
• Press 'Q' to terminate the script

Training

Pretrained weights

The PBRS dataset is currently offline due to instructions of SUNCG authors (see this and this). Therefore exact reproduction of our training procedure cannot be done properly.

Finetuning

For finetuning on NYUv2 Depth dataset, you will need the dataset and split:

mkdir datasets
wget -O datasets/nyuv2_splits.mat http://horatio.cs.nyu.edu/mit/silberman/indoor_seg_sup/splits.mat
wget -O datasets/nyu_depth_v2_labeled.mat http://horatio.cs.nyu.edu/mit/silberman/nyu_depth_v2/nyu_depth_v2_labeled.mat

Use the following command to train the network on NYUv2:

python3 train_sharpnet.py --dataset NYU \
--rootdir ../datasets/ \
-b BATCH_SIZE \
--cuda CUDA_DEVICE_ID --cpu NUM_PROCESSES \
--boundary --normals --depth \
--lr 8e-3 --max-epoch 80 \ 
--pretrained-model ../models/final_checkpoint_PBRS.pth \
--bias \
--exp_name CUSTOM_EXPERIMENT_NAME \
--consensus \
--freeze normals,boundary

Set BATCH_SIZE, NUM_PROCESSES based on your GPU and CPU capabilities. A GPU is required for training (multiple GPU is not supported yet).

Work in progress: argument parsing with configuration file.

Please read the paper for details on parameters we used.

Evaluation

NYUv2-OC dataset

You can find our manually annotated dataset used for occluding contour quality evaluation here.

Run the code

Run the evaluation code using:

python3 eval.py \
--rootdir DIR_CONTAINING_ALL_PREDICTIONS \
--cuda CUDA_DEVICE_ID \
--pretrained-model ../models/final_checkpoint_PBRS.pth \
--edges --crop \
--savepath PREDICTION_SAVEPATH \
--low LOW_CANNY_THRESHOLD \
--high HIGH_CANNY_THRESHOLD

Replace all capital FIELDS with appropriate values.

Citation

If you find SharpNet useful in your research, please consider citing:

@article{ramamonjisoa2019sharpnet,
    Title = {SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation},
    Author = {Michael Ramamonjisoa and Vincent Lepetit},
    Journal = {The IEEE International Conference on Computer Vision (ICCV) Workshops},
    Year = {2019}
}

sharpnet's Issues

[Question] Could you explain the normalization scheme you have used for PBRS and NYUV2 ?

In the dataset_manager.py file you normalize the depth map differently for the PBRS dataset and the NYU-V2 datasets.

More precisely, in the PBRS dataset you divide the depth maps by 65535 whereas in the NYUV2 dataset you use 65.535.

Is there a specific reason behind these normalizations ?

Thank you very much

How to get the real absolute depth from your model ?

I found the following code in file 'demo.py'
depth_pred = depth_pred.data.cpu().numpy()[0, 0, ...] * 65535 / 1000

and I think the depth_pred is the real depth in meters, right?

Optimizer and Weight decay

Regarding 6.4 in the paper, do you use actual weight decay or a simple L^2 regularisation term on the weights? Is the optimizer ordinary SGD or something like Adam?

Dear author, is it necessary to have a pretraining model of PBRS for training?

Do you have a code with insertion Stanford Bunny into the scene?

I mean the code of implementation of Figure 1 of https://arxiv.org/pdf/1905.08598.pdf.
Thank you for reply!

[BUG] Evaluation bug in eval.py compute_depth_metrics()

Thank for this wonderful work!
I believe there is a bug in this line of code.

SharpNet/eval.py

Line 56 in 6d69dcd

rmse_log = np.sqrt(np.mean(np.log10(np.clip(input, a_min=1e-12, a_max=1e12)) - np.log10(

May be the RMSElog is natural logarithm instead of log10?

Question: Annotation tool

I noted that you manually annotated 100 NYUv2 images with occlusion contours. Can you kindly tell me which tool did you use for annotation? Thanks a lot!

Strange Output Artefacts

Reference: BSDS500, subset test. Order: 1st row - RGB, Depth, 2nd row - Boundary, Normals

I tried out the BSDS500 dataset (https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html) and got these artefacts always in the left bottom in about 20% of 200 images from the test subset.

Normal ground-truth for NYU dataset?

As shown in the following lines, the only available dataset, NYU, did not load normal (and boundary) GT? I also check and find normal loss (and geo loss) are 0. However, in Fig. 7, normal GTs are showed. Is there way to load normal GT from NYU dataset? Thanks!

https://github.com/MichaelRamamonjisoa/SharpNet/blob/master/utils.py#L141-L142

why can't predict occluding contours?

Hello,
I load the model pre-trained on PBRS, then finetune on NYU. But when I run demo.py, the model can't predict occluding contours, the predicted result is blank. However, the other two task can work normally. I'm confused......

Normalization deph image

In demo.py :

depth_pred = depth_pred.data.cpu().numpy()[0, 0, ...]
depth_pred = scale * cv2.resize(depth_pred* 65535 / 1000, dsize=(image_original.size[0], image_original.size[1]), interpolation=cv2.INTER_LINEAR)

Change to :

depth_pred = depth_pred.data.cpu().numpy()[0, 0, ...]
depth_pred = scale * cv2.resize(depth_pred*255, dsize=(image_original.size[0], image_original.size[1]), interpolation=cv2.INTER_LINEAR)

About the depth-boundary consistency loss

Thank you for sharing your great work.
I have one question about the implementation of your loss function. For depth-boundary consistency loss, I think you realize it as a class there. But I found it isn't the same as the formulation you use in your paper. It seems like one the first term and without a minus sign. I'm a little confused and please help me figure it out.

SharpNet/loss.py

Line 266 in 9206fb0

class DepthBoundaryConsensusLoss(nn.Module):

[Question] Do you have any result or plan on KITTI

Hi, Michael,

Nice idea and great work! I am wondering if you have result of KITTI. Or do you plan to further work on KITTI?

Thanks

What's the --scale option means when run the demo.py?

If the size of my input image is not 640x480, can you tell me how to set the --scale option ？an example？

About your loss function and finetuning

Hi, I have been watching your excellent work recently. However I have not found the loss functions computation in your source code. Could you share them for us to take a look?