xiaochengcike / seg-with-spn

Demo code of the paper: "Learning to Segment Instances in Videos with Spatial Propagation Network", in CVPR'17 Workshop on DAVIS Challenge

Learning to Segment Instances in Videos with Spatial Propagation Network

This paper is available at the 2017 DAVIS Challenge website.

Check our results in this video.

Contact: Jingchun Cheng (chengjingchun at gmail dot com)

Cite the Paper

If you find that our method is useful in your research, please cite:

@article{DAVIS2017-6th,
  author = {J. Cheng and S. Liu and Y.-H. Tsai and W.-C. Hung and S. Gupta and J. Gu and J. Kautz and S. Wang and M.-H. Yang}, 
  title = {Learning to Segment Instances in Videos with Spatial Propagation Network}, 
  journal = {The 2017 DAVIS Challenge on Video Object Segmentation - CVPR Workshops}, 
  year = {2017}
}

About the Code

• The code released here mainly consistes of two parts in the paper: foreground segmentation and instance recognition.

• It contains the parent net for foreground segmentation and training codes for instance recognition networks.

• The matlab_code folder contains a simple version of our CRAF step for segmentation refinement.

Requirements

• Install caffe and pycaffe at http://caffe.berkeleyvision.org/.

• Download the DAVIS 2017 dataset and put it in the data folder.

• Download the pre-trained foreground/background model here and put it in the pretrained folder.

Training

• Train the per-object recognition model.
  cd training
python solve.py PATH_OF_MODEL PATH_OF_SOLVER
  Foe example, on the 'choreography' video for the 1st object, run:
  python solve.py ../pretrained/PN_ResNetF.caffemodel ../ResNetF/testnet_per_obj/choreography/solver_1.prototxt

Testing

• Test the general foreground/backgroung model.
  python infer_test_fgbg.py PATH_OF_MODEL PATH_OF_RESULT VIDEO_NAME
  Foe example, on the 'lions' video, run:
  python infer_test_fgbg.py pretrained/PN_ResNetF.caffemodel results/fgbg lions

• Test the object instance model.
  python infer_test_perobj.py MODEL_ITERATION VIDEO_NAME OBJECT_ID
  For example, on the 'lions' video for the 2nd object, run:
  python infer_test_perobj.py 3000 lions 2

• Run example_CRAF.m in the matlab_code folder for a demo on CRAF segmentation refinement.

Download Our Segmentation Results on 2017 DAVIS Challenge

• General foreground/background segmentation here
• Instance-level object segmentation without refinement here
• Final instance-level object segmentation with refinement here

Note

The model and code are available for non-commercial research purposes only.

• 09/2017: code and model released
• 03/2018: pre-trained model updated