raghudeep / netwarp_public

Semantic Video CNNs through Representation Warping. ICCV 2017

Semantic Video CNNs through Representation Warping

This is the code accompanying the following ICCV 2017 publication:

---

Semantic Video CNNs through Representation Warping.

---

This is developed and maintained by Raghudeep Gadde, Varun Jampani, Peter V. Gehler.

Please visit the project website http://segmentation.is.tue.mpg.de for more details about the paper and overall methodology.

Installation

The code provided in this repository relies on the same installation procedure as the one from Caffe. Before you start with the NetWarp code, please install all the requirements of Caffe by following the instructions from this page first. You will then be able to build Caffe with our code. The repository also contains external code from https://github.com/tikroeger/OF_DIS to compute the optical flow and https://github.com/mcordts/cityscapesScripts to evaluate results on the Cityscapes dataset.

Integration into Caffe

There are mainly two ways for integrating the additional layers provided by our library into Caffe:

• Dowloading a fresh clone of Caffe and patching it with our source files, so that you will be able to test the code with minimal effort.
• Patching an existing copy of Caffe, so that you can integrate our code with your own development on Caffe.

Downloading and Patching

This can be done just by the following commands:

cd $netwarp
mkdir build
cd build
cmake ..

This will configure the project, you may then run:

• for building the project

  make 
  

  This will clone a Caffe version from the main Caffe repository into the build folder and compiles together with our newly added layers.

• for running the tests, including the ones of the NetWarp:

  make runtest
  

  (this follows the same commands as for Caffe)

Notes

• Our code has been tested with revision 691febcb83d6a3147be8e9583c77aefaac9945f8 of Caffe, and this is the version that is cloned. You may change the version by passing the option CAFFE_VERSION on the command line of cmake:

    cmake -DCAFFE_VERSION=some_hash_or_tag ..
  

such as cmake -DCAFFE_VERSION=HEAD ...

• If you want to use your fork instead of the original Caffe repository, you may provide the option CAFFE_REPOSITORY on the cmake command line (it works exactly as for CAFFE_VERSION).

• Any additional command line argument you pass to cmake will be forwarded to Caffe, except for those used directly by our code:

  cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DBOOST_ROOT=../osx/boost_1_60_0/
    -DBoost_ADDITIONAL_VERSIONS="1.60\;1.60.0" ..
  

Patching an existing Caffe version

Automatic CMAKE way

You may patch an existing version of Caffe by providing the CAFFE_SRC on the command line

cd $netwarp
mkdir build
cd build
cmake -DCAFFE_SRC=/your/caffe/local/copy ..

This will add the files of the NetWarp to the source files of the existing Caffe copy, but will also overwrite caffe.proto (a backup is made in the same folder). The command will also create a build folder local to the NetWarp repository (inside the build folder on the previous example): you may use this one or use any previous one, Caffe should automatically use the sources of the NetWarp.

Manual way

The above patching that is performed by cmake is rather a copying of the files from the folder of the netwarp to the corresponding folders of Caffe. Caffe will then add the new files into the project.

Alternatively, you can manually copy all but caffe.proto source files in netwarp folder to the corresponding locations in your Caffe repository. Then, for merging the caffe.proto file of netwarp to your version of the caffe.proto:

1. the copy the lines 409-412 and 1418-1451 in caffe.proto to the corresponding caffe.proto file in the destination Caffe repository.
2. Change the parameter IDs for BNParameter, WarpParameter, and InterpParameter based on the next available LayerParameter ID in your Caffe.

Example Usage

To use the provided code and replicate the results on the Cityscapes val dataset,

Preparing the data

Download leftImg8bit_sequence.zip from the Cityscapes dataset webpage https://www.cityscapes-dataset.com/ in the data folder. If you want to compute the accuracy scores on the val set, also download the gtFine.zip file. Extract content from both the zip files and place them in the data folder following the same directory structure. Also set the CITYSCAPES_DATASET environment variable with the path to the dataset.

export CITYSCAPES_DATASET='$netwarp/data/cityscapes/'

Computing optical flow

Next, compute the optical flow using the following command

export NETWARP_BUILD_DIR='/path/to/build/'
cd $netwarp
python scripts/extract_opticalflow.py VAL 

The above command will compute the optical flow on the Cityscapes val set and save them in the Cityscapes dataset folder.

Get the trained PSPNet-NetWarp model

Execute the below command to download a NetWarp model for PSPNet, trained on Cityscapes train videos.

sh scripts/get_cityscapes_model.sh

This will download the caffemodel in the models folder. (We shall hopefully release more models in the coming days).

Doing the segmentation

You can run the segmentation using the run_netwarp.py python script in the $netwarp/scripts folder which rely on the Python extensions of Caffe.

Syntax for running the segmentation script:

cd $netwarp
python scripts/run_netwarp.py data_split path_to_prototxt path_to_caffemodel path_to_results_dir number_of_gpus_to_use

To run the segmentation on Cityscapes validation set:

python scripts/run_netwarp.py VAL models/pspnet101_cityscapes_conv5_4netwarp_deploy.prototxt models/pspnet101_cityscapes_conv5_4netwarp.caffemodel results/ 2

The above command will save color coded segmentation masks in results/color/ and class indexed segmentation masks suitable for computing IoU using cityscapesScripts in results/index/

Evaluating the results

We provide a python script to compute the Trimap IoU score of the obtained segmentations.

cd $netwarp
python scripts/compute_scores.py VAL path_to_results trimap_width

An example to compute Trimap IoU on the obtained results:

python scripts/compute_scores.py VAL results/index/ 3

Citations

Please consider citing the below paper if you make use of this work and/or the corresponding code:

@inproceedings{gadde2017semantic,
  author = {Gadde, Raghudeep and Jampani, Varun and Gehler, Peter V.},
  title = {Semantic Video CNNs Through Representation Warping},
  booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
  month = {Oct},
  year = {2017}
} 

If you use the dense inverse search based optical flow, please do not forget citing the relavant paper:

@inproceedings{kroeger2016fast,
  title={Fast optical flow using dense inverse search},
  author={Kroeger, Till and Timofte, Radu and Dai, Dengxin and Van Gool, Luc},
  booktitle={European Conference on Computer Vision},
  pages={471--488},
  year={2016},
  organization={Springer}
}

netwarp_public's Issues

run error about cudnn issue

Hi, @raghudeep

When I run:
python scripts/run_netwarp.py data_split models/pspnet101_cityscapes_conv5_4netwarp_deploy.prototxt models/pspnet101_cityscapes_conv5_4netwarp.caffemodel results 1

I got the following error:

F1110 21:45:58.179529 28101 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0)  CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
Aborted (core dumped)

It seems the cudnn issue. I had installed cuda-8.0 and cudnn 6.0 in my system

root@milton-ThinkCentre-M93p:~# cat /usr/local/cuda/version.txt
CUDA Version 8.0.61
root@milton-ThinkCentre-M93p:~# cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR      6
#define CUDNN_MINOR      0
#define CUDNN_PATCHLEVEL 21
--
#define CUDNN_VERSION    (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"

Could you give some suggestion to fix this problem?

THX!

required GPU memory

Hi, @raghudeep,

How much GPU memory is required to run the segmentation? I got "out of memory" error using my Titan-Black 6GB GPU card. BTW, could I run the segmentation by reducing the batch number or other settings?

Thanks!

Could your provide training script

Hi, thanks for your work, I would like to know could you provide the training code, thx.

script filename mismatch

Hi, @raghudeep ,

Here, the script fiilename should be get_netwarp_pspnet_model.sh , right?

non-differentiable case in warp operation

The paper mentioned that when the flow values are integer numbers, the warping function needs to add 0.0001 to avoid the non-differentiable case. However, I cannot find any line in the code that seems to do this, except for a comment "// + 0.00000001" in warper_layer.cpp:62.

To be honest, I am not really sure whether this non-differentiable case really exists. If yes, where is the addition?

frame index

netwarp_public/scripts/run_netwarp.py

Line 100 in 1b84e43

inputs['flo_1'] = flo_array[j][:,:,s_y:e_y,s_x:e_x]

this line should be inputs['flo_1'] = flo_array[j+1][:,:,s_y:e_y,s_x:e_x]
because your current input should be the second flo(because you load two flos once time)

Low IOU result

I follow your instruction to run your model on the cityscapes dataset.
However, I got the following result:
classes IoU nIoU

road : 0.971 nan
sidewalk : 0.788 nan
building : 0.882 nan
wall : 0.583 nan
fence : 0.637 nan
pole : 0.263 nan
traffic light : 0.558 nan
traffic sign : 0.558 nan
vegetation : 0.873 nan
terrain : 0.580 nan
sky : 0.913 nan
person : 0.617 0.516
rider : 0.442 0.360
car : 0.871 0.802
truck : 0.783 0.451
bus : 0.819 0.652
train : 0.756 0.557
motorcycle : 0.506 0.390
bicycle : 0.616 0.459

Score Average : 0.685 0.523

Naming of .png files different from the format mentioned in netwarp_public-master/scripts/extract_opticalflow.py

Hi @raghudeep ,

I just wanted to confirm if the model has been verified on the val and train datasets populated by the cityscapes - leftImg8bit_sequence/ dataset.

I am observing that - in order to calculate the optical flow using extract_opticalflow.py, it needs two consecutive frames or image files - but the val/frankfurt directory does not have any consecutive frames, hence I am getting the following errors:
OpenCV Error: Assertion failed (dims <= 2 && step[0] > 0) in locateROI, file /tmp/binarydeb/ros-kinetic-opencv3-3.3.1/modules/core/src/matrix.cpp, line 991
terminate called after throwing an instance of 'cv::Exception'
what(): /tmp/binarydeb/ros-kinetic-opencv3-3.3.1/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI

Steps followed after using "Downloading and Patching" method to integrate with Caffe:

1. I have populated the following path with - netwarp_public-master/data/cityscapes/leftImg8bit_sequence/ train, val and test directories downloaded from cityscapes website(leftImg8bit_trainvaltest.zip).
2. Run the following commands:
   export NETWARP_BUILD_DIR='/path/to/build/'
   cd <path_to_netwarp>
   python scripts/extract_opticalflow.py VAL

Please let me know if I am going wrong in any step or if some changes need to make to the scripts

Parameter problems

Thansks your code,in your paper,you set two parameters w1 w2 to balance the weights of PSP net module and warp module, However, in your trained caffemodel, the two parametes are still very close to their initializaion values(w1 =1 ,w2= 0, conv5_4_0_w ,conv5_4_1_w in model) .Why ? @raghudeep