seth-park / text_objseg_caffe

Caffe implementation for Hu et al. Segmentation for Natural Language Expressions in arXiv:1603.06180, 2016 http://ronghanghu.com/text_objseg

Segmentation from Natural Language Expressions

This repository contains the Caffe reimplementation of the following paper:

• R. Hu, M. Rohrbach, T. Darrell, Segmentation from Natural Language Expressions. in arXiv:1603.06180, 2016. (PDF)

@article{hu2016segmentation,
  title={Segmentation from Natural Language Expressions},
  author={Hu, Ronghang and Rohrbach, Marcus and Darrell, Trevor},
  journal={arXiv preprint arXiv:1603.06180},
  year={2016}
}

Project Page: http://ronghanghu.com/text_objseg

Installation

1. Install Caffe following the instructions here.
2. Download this repository or clone with Git, and then cd into the root directory of the repository.

Training and evaluation on ReferIt Dataset

Download dataset and VGG network

Download ReferIt dataset:

./referit/referit-dataset/download_referit_dataset.sh

Download the caffemodel for VGG-16 network parameters trained on ImageNET 1000 classes.

Training

You may need to add the repository root directory to Python's module path:

export PYTHONPATH=/path/to/text_objseg_caffe/:$PYTHONPATH

Build training batches for bounding boxes:

python referit/build_training_batches_det.py

Build training batches for segmentation:

python referit/build_training_batches_seg.py

Configure the config.py file in the directory det_model and train the language-based bounding box localization model:

python det_model/train_det_model.py

Configure the config.py file in the directory seg_low_res_model and train the low resolution language-based segmentation model (from the previous bounding box localization model):

python seg_low_res_model/train_low_res_model.py

Configure the config.py file in the directory seg_model and train the high resolution language-based segmentation model (from the previous low resolution segmentation model):

python seg_model/train_seg_model.py

Evaluation

You may need to add the repository root directory to Python's module path:

export PYTHONPATH=path/to/text_objseg_caffe:$PYTHONPATH

Configure the test_config.py file in the directory seg_model and run evaluation for the high resolution language-based segmentation model:

python seg_model/test_seg_model.py

This should reproduce the results in the paper. You may also evaluate the language-based bounding box localization model:

python det_model/test_det_model.py

The results can be compared to this paper.

Demo

There is a demo that you can try! Run the demo in ./demo/text_objseg_demo.ipynb with Jupyter Notebook (IPython Notebook).

text_objseg_caffe's Issues

Unknown layer type: L2Normalize

Hi Seth:
Thank you for your great work.
When I train the detection model. I got the following error:

I0322 11:15:13.847193 15945 net.cpp:129] Top shape: 50 1000 (50000)
I0322 11:15:13.847195 15945 net.cpp:137] Memory required for data: 5770873800
I0322 11:15:13.847198 15945 layer_factory.hpp:77] Creating layer img_l2norm
F0322 11:15:13.847214 15945 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: L2Normalize (known types: AbsVal, Accuracy, ArgMax, BNLL, BatchNorm, BatchReindex, Bias, Concat, ContrastiveLoss, Convolution, Crop, Data, Deconvolution, Dropout, DummyData, ELU, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, Input, LRN, LSTM, LSTMUnit, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Parameter, Pooling, Power, Python, RNN, ReLU, Reduction, Reshape, SPP, Scale, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, WindowData)
*** Check failure stack trace: ***
Aborted (core dumped)

I am using official version of caffe and successfully compiled the caffe with pycaffe and anaconda. and followed your steps.But I still got the error. Is there anything I am missing?
My system is Ubuntu16.04 LTS with Nvidia Cuda8.0 Titan Xp
I appreciate if you can help me.

Cannot copy param 0 weights from layer 'fc6'

when I running train_det_model.py，I meet this question:
Cannot copy param 0 weights from layer 'fc6'; shape mismatch. Source param shape is 1024 512 3 3 (4718592); target param shape is 4096 25088 (102760448). To learn this layer's parameters from scratch rather than copying from a saved net, rename the layer.
how can i solve this problem?

Image input channel order

Hi @Seth-Park . It seems from https://github.com/Seth-Park/text_objseg_caffe/blob/master/seg_model/referit_data_provider.py#L47 that in the data-providing Python layer, the image is in RGB order. However, shouldn't image input to Caffe be in BGR order? Is there any channel-switching step that I missed?
Thank you very much!

L2Normalize Layer?

Hi @Seth-Park . Thank you for the code. I was wondering which Caffe version are you using? The network structure uses L2Normalize layer, which I cannot find in the BVLC Caffe.
Thanks!