nv-tlabs / steal

STEAL - Learning Semantic Boundaries from Noisy Annotations (CVPR 2019)

STEAL

This is the official inference code for:

Devil Is in the Edges: Learning Semantic Boundaries from Noisy Annotations

David Acuna, Amlan Kar, Sanja Fidler

CVPR 2019 [Paper] [Project Page]

License

# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of NVIDIA CORPORATION nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Usage

Clone this repo

git clone https://github.com/nv-tlabs/STEAL
cd STEAL

Install dependencies

This code requires PyTorch 0.4 and python 3+. Please install dependencies by

pip install -r requirments.txt

Download pretrained models

Download the tar of the pretrained models from the Google Drive Folder, save it in 'checkpoints/', and run

cd checkpoints
tar -xvf checkpoints.tar.gz
cd ../

Inference (SBD)

python inference_sbd.py \
    --root_dir_val= ./data/sbd/data_aug/\
    --flist_val= ./data/sbd/data_aug/val_list.txt\
    --output_folder=./output/sbd/ \
    --ckpt=./checkpoints/sbd/model_checkpoint.pt\

Instructions and preprocessing scripts to download SBD and preprocess the dataset can be found here: https://github.com/Chrisding/sbd-preprocess

Inference (Cityscapes)

python inference_cityscapes.py \
    --root_dir_val=./data/cityscapes-preprocess/data_proc \
    --flist_val=./data_proc/val.txt \
    --output_folder=./output/cityscapes/ \
    --ckpt=./checkpoints/cityscapes/model_checkpoint.pt\

Instructions and preprocessing scripts for Cityscapes can be found here: https://github.com/Chrisding/cityscapes-preprocess

Test-NMS: An example of how to apply TEST-NMS using Piotr's Structured Forest matlab toolbox. can be found in utils/edges_nms.m. During training, we optimized for the same set of operations with r=2 (Check paper for more details)

Coarse-to-fine Demo

Checkout the ipython notebook that provides a simple walkthrough demonstrating how to run our model to refine coarsely annotated data.

If you use this code, please cite:

@inproceedings{AcunaCVPR19STEAL,
title={Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations},
author={David Acuna and Amlan Kar and Sanja Fidler},
booktitle={CVPR},
year={2019}
}

steal's Issues

Inference outputs are black images

I am running inference models, and the outputs are blank images.

how to get the result picture like this?

when I use inference_cityscapes.py , it seems it will output different class into different images separately. I want to know how to get a total result like this in the paper?

Could you please provide the training script?

I want to train it on another dataset, thank you very much!

spelling issue in README.md

Hello, thank you for your awesome work.

In the README.md file, the instruction to install the requirements has a typo. It should be:

pip install -r requirements.txt

training code releasing

Thanks for the excellent work, but when will you release the training code?

Output Problem

When I run the inference.py using your provided model and sdb dataset. the output images are almost dark and a few images have clear and thinning edges. I do not know why. Thank you for your reply!

opencv-python version error

ERROR: Could not find a version that satisfies the requirement opencv-python==3.4.0.12 (from -r requirements.txt (line 3)) (from versions: 3.4.2.16, 3.4.2.17, 3.4.3.18, 3.4.4.19, 3.4.5.20, 4.0.0.21, 4.0.1.23, 4.0.1.24, 4.1.0.25)
ERROR: No matching distribution found for opencv-python==3.4.0.12 (from -r requirements.txt (line 3))

How to get the mean value?

STEAL/inference_cityscapes.py

Line 65 in 99f78de

mean_value = (104.008, 116.669, 122.675) # BGR

If I use the model to inference other dataset for example KITTI, should I change the mean value? Or dose it matter for different image size?

AMD gpu or cpu only code

Hi there,
Thanks for the excellent work. I’m new to CV and machine learning in general and was wondering if it is possible to run this code if CUDA is not available, I.e. using cpu only or AMD gpu. Thanks!

Where is the place to download the data?

(venv) (base) suryadi@xtal:/media/suryadi/DATA/learn/STEAL$ python inference_sbd.py \
>     --root_dir_val= ./data/sbd/data_aug/\
>     --flist_val= ./data/sbd/data_aug/val_list.txt\
>     --output_folder=./output/sbd/ \
>     --ckpt=./checkpoints/sbd/model_checkpoint.pt\
> 
usage: inference_sbd.py [-h] [--root_dir_val ROOT_DIR_VAL]
                        [--flist_val FLIST_VAL] [--ckpt CKPT]
                        [--output_folder OUTPUT_FOLDER] [--dataset DATASET]
                        [--n_classes N_CLASSES]
inference_sbd.py: error: unrecognized arguments: ./data/sbd/data_aug/ ./data/sbd/data_aug/val_list.txt

Dear David,

Where to download data?
Is it correct from http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz ?

Thank you very much in advance.

Warmest Regards,
Suryadi

Implementation of thinning layer

Hello, thank you for your excellent work. I've read your paper and i wonder whether you plan to release the code of training, especially the computation details of NMS loss and direction loss. Thank you.

Typo in your readme

Hi! I just noticed that there is a small typo in your readme:
pip install -r requirments.txt

Scripts for benchmarks on SBD and Cityscapes

Hello.

Can you provide the scripts for the benchmarks on SBD re-annotated an Cityscapes? In fact, I took your pre-trained models and do the benchmakrs using the scripts from: https://github.com/Chrisding/seal, but I cannot get the same results as in your paper.

Thank you very much.

How to generate the ground truth normal direction

Hi, thanks for your great work. I was curious about how the ground truth normal direction on boundaries is generated. It will be appreciated if you could share the implementation. Thank you.

pytorch version issue with python 3.8

HI, I am using the "pip install -r requirements.txt" for dependencies installation, but got following error. I have using ubantu with python 3.8.

ERROR: Could not find a version that satisfies the requirement torch==0.4.0 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1)
ERROR: No matching distribution found for torch==0.4.0

Please suggest a solution.

About direction loss

Thx for your work and releasing the test code.
How can I rewrite the direction loss if I didn't install matlab on my computer? Are there any other ways to calculate normal directions of the edge pixels?
Looking forward for your reply.

Are you planning to release training code?