halbielee / sec_pytorch

PyTorch implementation of "Seed, Expand, Constrain: Three Principles for Weakly-Supervised Image Segmentation", ECCV2016

Seed, Expand, Constrain: Three Principles for Weakly-Supervised Image Segmentation

PyTorch implementation of "Seed, Expand, Constrain: Three Principles for Weakly-Supervised Image Segmentation", ECCV2016

This is not the official repository for this paper. For the official, please see the following links.

• Paper : https://arxiv.org/abs/1603.06098

• Official code : caffe implmentation

Introduction

This is a work that proposes a new composite loss function for training convolutional neural network for the task of weakly-supervised semantic segmentation. Three novel loss functions are introduced:

• Seeding loss
• Expasion loss
• Contrain-to-boundary loss

Updates

19 Jul, 2020: upload PascalVOC pretrained model

02 Jan, 2020: upload COCO implementation

11 Nov, 2019: Initial upload

Prerequisites

• Python 3.6
• PyTorch >= 1.0.0
• Torchvision >= 0.2.2
• PIL
• opencv-python (OpenCV for Python)
• tqdm
• tensorboardX

Fully connected CRF wrapper (requires the Eigen3 Package)

apt-get install libeigen3-dev

# this should be done after download the source..
pip install CRF/

Data & Model Preparation

Pascal VOC 2012 dataset (VOC2012) is used for this implementation.

• Download VOC2012 from here
• You can see the detail of VOC2012 in http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html

We use ImageNet pretrained model which is coverted from Caffe.

• Download the pretrained model from here
• You can convert it on your own. Please see here for more details.

Execution

• download the source code & localization cue preparation

git clone https://github.com/halbielee/SEC_pytorch.git
cd SEC_pytorch 

# localizatio-cue preparation
gzip -kd datalist/PascalVOC/localization_cues.pickle.gz

• train

# Before executing this, please set the appropriate dataset path
bash script/train.sh

• test (generate the prediction map)

# Before executing this, please set the appropriate dataset path and other options..
bash script/test_multiprocess.sh

• evaluation (calculate the performance)

# Before executing this, please set the appropriate prediction_map / gt_map path
bash script/evaluation.sh

Performance

We evaluate the PyTorch implementation with hyperparms which the author provided without any other tuning.

Method	Dataset	Backbone	mIOU	Download
SEC	VOC2012-val	DeepLab-LargeFOV	50.6049	⬇️
SEC	VOC2012-val	DeepLab-LargeFOV	49.6978	⬇️

Segmentation Result

Origin : Prediction : Ground Truth

{: widths="100%")

sec_pytorch's Issues

Question about training from scratch

First of all, Thank you for your hard work. I am familiar with Pytorch, so your work is very helpful.
I am curious about the condition that you got the performance (mIoU 50.6049 and mIoU 49.6978) you mentioned. When I download the weight and evaluate it, I get the same result as the performance you mentioned.

meanIOU : 50.6049

background : 82.1013
aeroplane : 59.3787
bicycle : 25.7055
bird : 61.6386
boat : 26.7922
bottle : 40.3784
bus : 66.6121
car : 63.3243
cat : 75.6848
chair : 22.2849
cow : 54.4826
diningtable : 28.8335
dog : 66.1762
horse : 57.7841
motorbike : 63.0711
person : 53.3926
pottedplant : 32.1166
sheep : 61.7726
sofa : 32.0557
train : 44.8119
tvmonitor : 44.3050

However, if I try to train from scratch, I cannot get the same result. This is the result of testing by saving the checkpoint when the loss is minimum without changing the hyperparameter setting.

meanIOU : 10.8535

background : 74.5545
aeroplane : 12.2275
bicycle : 0.0005
bird : 0.0000
boat : 1.5335
bottle : 0.1878
bus : 0.6897
car : 0.0151
cat : 8.4732

The learning loss is shown below. Even if I change the hyperparameters, the sum of losses converges to 6.xx (seed loss 2.xx, expand loss 4.xx, constrain loss 0.5xx). It seems that training is not working in my case.

What I'm curious about is your learning loss decreasing well? What is the saving checkpoint criteria? (minimal loss or something?)

How to generate loc-cues file, would u provide more details or release related code?

As mentioned above.

localization_cues of COCO

Can you tell me how do you get the localization_cues.pickle.gz of COCO ? Thanks.

The result of training always generates all-0 value maps

I don't know why the model after being trained in train.script always generates all-0 value maps, but I test the best model you provide shows the result is right...

I encountered a problem while running.

After I run test_multiprocess.py, the generated pred_images are all black. What is the reason for this?

_

_