yzou2 / cbst

Code for <Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training> in ECCV18

Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training

By Yang Zou*, Zhiding Yu*, Vijayakumar Bhagavatula, Jinsong Wang (* indicates equal contribution).

Update

• 2019.10.10: check out our new paper "Confidence Regularized Self-Training" (ICCV 2019, Oral), which investigates confidence regularization in self-training systematically. The pytorch code based on CBST are released.
• 2018.11.11: source domain training code for GTA-5 and SYNTHIA uploaded
• 2018.10.14: code release for GTA-5 to Cityscapes and SYNTHIA to Cityscapes

Contents

0. Introduction
1. Citation and license
2. Requirements
3. Setup
4. Usage
5. Results
6. Note

Introduction

This repository contains the self-training based methods described in the ECCV 2018 paper "Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training". Self-training (ST), Class-balanced self-training (CBST) with Spatial Priors (CBST-SP) are implemented. CBST is the core algorithm for the 1st and 3rd winner of Domain Adaptation of Semantic Segmentation Challenge in CVPR 2018 Workshop on Autonomous Driving (WAD).

Requirements:

The code is tested in Ubuntu 16.04. It is implemented based on MXNet 1.3.0 and Python 2.7.12. For GPU usage, the maximum GPU memory consumption is about 7GB in a single NVIDIA TiTan Xp.

Citation

If you use this code, please cite:

@inproceedings{zou2018unsupervised,
  title={Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training},
  author={Zou, Yang and Yu, Zhiding and Kumar, BVK Vijaya and Wang, Jinsong},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  pages={289--305},
  year={2018}
}


@InProceedings{Zou_2019_ICCV,
author = {Zou, Yang and Yu, Zhiding and Liu, Xiaofeng and Kumar, B.V.K. Vijaya and Wang, Jinsong},
title = {Confidence Regularized Self-Training},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}

The model and code are available for non-commercial (NC) research purposes only. If you modify the code and want to redistribute, please include the CC-BY-NC-SA-4.0 license.

Results:

0. GTA2city:

   Case	mIoU	Road	Sidewalk	Build	Wall	Fence	Pole	Traffic Light	Traffic Sign	Veg.	Terrain	Sky	Person	Rider	Car	Truck	Bus	Train	Motor	Bike
   Source	35.4	70.0	23.7	67.8	15.4	18.1	40.2	41.9	25.3	78.8	11.7	31.4	62.9	29.8	60.1	21.5	26.8	7.7	28.1	12.0
   ST	41.5	88.0	20.4	80.4	25.5	19.7	41.3	42.6	20.2	86.0	3.5	64.6	65.4	25.4	83.3	31.7	44.3	0.6	13.4	3.7
   CBST	45.2	86.8	46.7	76.9	26.3	24.8	42.0	46.0	38.6	80.7	15.7	48.0	57.3	27.9	78.2	24.5	49.6	17.7	25.5	45.1
   CBST-SP	46.2	88.0	56.2	77.0	27.4	22.4	40.7	47.3	40.9	82.4	21.6	60.3	50.2	20.4	83.8	35.0	51.0	15.2	20.6	37.0

1. SYNTHIA2City:

   Case	mIoU	Road	Sidewalk	Build	Wall	Fence	Pole	Traffic Light	Traffic Sign	Veg.	Sky	Person	Rider	Car	Bus	Motor	Bike
   Source	29.2	32.6	21.5	46.5	4.8	0.1	26.5	14.8	13.1	70.8	60.3	56.6	3.5	74.1	20.4	8.9	13.1
   ST	32.2	38.2	19.6	70.2	3.9	0.0	31.9	17.6	17.2	82.4	68.3	63.1	5.3	78.4	11.2	0.8	7.5
   CBST	42.5	53.6	23.7	75.0	12.5	0.3	36.4	23.5	26.3	84.8	74.7	67.2	17.5	84.5	28.4	15.2	55.8

Setup

We assume you are working in cbst-master folder.

0. Datasets:

• Download GTA-5 dataset. Since GTA-5 contains images with different resolutions, we recommend resize all images to 1052x1914.
• Download Cityscapes.
• Download SYNTHIA-RAND-CITYSCAPES.
• Put downloaded data in "data" folder.

1. Source pretrained models:

• Download source model trained in GTA-5.
• Download source model trained in SYNTHIA.
• For ImageNet pre-traine model, download model in dropbox, provided by official ResNet-38 repository.
• Put source trained and ImageNet pre-trained models in "models/" folder

2. Spatial priors

• Download Spatial priors from GTA-5. Spatial priors are only used in GTA2Cityscapes. Put the prior_array.mat in "spatial_prior/gta/" folder.

Usage

0. Set the PYTHONPATH environment variable:

cd cbst-master
export PYTHONPATH=PYTHONPATH:./

1. Self-training for GTA2Cityscapes:

• CBST-SP:

python issegm/solve_AO.py --num-round 6 --test-scales 1850 --scale-rate-range 0.7,1.3 --dataset gta --dataset-tgt cityscapes --split train --split-tgt train --data-root DATA_ROOT_GTA5 --data-root-tgt DATA_ROOT_CITYSCAPES --output gta2city/cbst-sp --model cityscapes_rna-a1_cls19_s8 --weights models/gta_rna-a1_cls19_s8_ep-0000.params --batch-images 2 --crop-size 500 --origin-size-tgt 2048 --init-tgt-port 0.15 --init-src-port 0.03 --seed-int 0 --mine-port 0.8 --mine-id-number 3 --mine-thresh 0.001 --base-lr 1e-4 --to-epoch 2 --source-sample-policy cumulative --self-training-script issegm/solve_ST.py --kc-policy cb --prefetch-threads 2 --gpus 0 --with-prior True

2. Self-training for SYNTHIA2City:

• CBST:

python issegm/solve_AO.py --num-round 6 --test-scales 1850 --scale-rate-range 0.7,1.3 --dataset synthia --dataset-tgt cityscapes --split train --split-tgt train --data-root DATA_ROOT_SYNTHIA --data-root-tgt DATA_ROOT_CITYSCAPES --output syn2city/cbst --model cityscapes_rna-a1_cls16_s8 --weights models/synthia_rna-a1_cls16_s8_ep-0000.params --batch-images 2 --crop-size 500 --origin-size 1280 --origin-size-tgt 2048 --init-tgt-port 0.2 --init-src-port 0.02 --max-src-port 0.06 --seed-int 0 --mine-port 0.8 --mine-id-number 3 --mine-thresh 0.001 --base-lr 1e-4 --to-epoch 2 --source-sample-policy cumulative --self-training-script issegm/solve_ST.py --kc-policy cb --prefetch-threads 2 --gpus 0 --with-prior False

• To run the code, you need to set the data paths of source data (data-root) and target data (data-root-tgt) by yourself. Besides that, you can keep other argument setting as default.
• For CBST, set "--kc-policy cb" and "--with-prior False". For ST, set "--kc-policy global" and "--with-prior False".
• We use a small class patch mining strategy to mine the patches including small classes. To turn off small class mining, set "--mine-port 0.0".

4. Evaluation

• Test in Cityscapes for model compatible with GTA-5 (Initial source trained model as example)

python issegm/evaluate.py --data-root DATA_ROOT_CITYSCAPES --output val/gta-city --dataset cityscapes --phase val --weights models/gta_rna-a1_cls19_s8_ep-0000.params --split val --test-scales 2048 --test-flipping --gpus 0 --no-cudnn

• Test in Cityscapes for model compatible with SYNTHIA (Initial source trained model as example)

python issegm/evaluate.py --data-root DATA_ROOT_CITYSCAPES --output val/syn-city --dataset cityscapes16 --phase val --weights models/synthia_rna-a1_cls16_s8_ep-0000.params --split val --test-scales 2048 --test-flipping --gpus 0 --no-cudnn

• Test in GTA-5

python issegm/evaluate.py --data-root DATA_ROOT_GTA --output val/gta --dataset gta --phase val --weights models/gta_rna-a1_cls19_s8_ep-0000.params --split train --test-scales 1914 --test-flipping --gpus 0 --no-cudnn

• Test in SYNTHIA

python issegm/evaluate.py --data-root DATA_ROOT_SYNTHIA --output val/synthia --dataset synthia --phase val --weights models/synthia_rna-a1_cls16_s8_ep-0000.params --split train --test-scales 1280 --test-flipping --gpus 0 --no-cudnn

5. Train in source domain

• Train in GTA-5

python issegm/train_src.py --gpus 0,1,2,3 --split train --data-root DATA_ROOT_GTA --output gta_train --model gta_rna-a1_cls19_s8 --batch-images 16 --crop-size 500 --scale-rate-range 0.7,1.3 --weights models/ilsvrc-cls_rna-a1_cls1000_ep-0001.params --lr-type fixed --base-lr 0.0016 --to-epoch 30 --kvstore local --prefetch-threads 16 --prefetcher process --cache-images 0 --backward-do-mirror --origin-size 1914

• Train in SYNTHIA

python issegm/train_src.py --gpus 0,1,2,3 --split train --data-root DATA_ROOT_SYNTHIA --output synthia_train --model synthia_rna-a1_cls16_s8 --batch-images 16 --crop-size 500 --scale-rate-range 0.7,1.3 --weights models/ilsvrc-cls_rna-a1_cls1000_ep-0001.params --lr-type fixed --base-lr 0.0016 --to-epoch 50 --kvstore local --prefetch-threads 16 --prefetcher process --cache-images 0 --backward-do-mirror --origin-size 1280

• Train in Cityscapes, please check the official ResNet-38 repository.

Note

• This code is based on ResNet-38.
• Due to the randomness, the self-training results may slightly vary in each run. Usually the best results will be obtained in 2nd/3rd round. For training in source domain, the best model usually appears during the first 30 epoches. Optimal model appearing in initial stage is also possible.

Contact: [email protected]

cbst's Issues

pretrained file corrupted

I found that your source model of gta is corrupted. Can you check it and provide the completed one?
https://www.dropbox.com/s/idnnk398hf6u3x9/gta_rna-a1_cls19_s8_ep-0000.params?dl=0

About the Results

@yzou2

Are your results of cityscapes from validation set or test set?

About label definition of SYNTHIA

As the label maps in labels_synthia.py and labels_cityscapes_synthia.py show, there is 16 classes in both training and testing phase for SYNTHIA16 settings. And the results in paper show SYNTHIA13 is just evaluated among 13 classes based on SYNTHIA16 results.
I'm just making sure that for SYNTHIA16 and SYNTHIA13, there are 16 classes in training and testing, not 19 or 13. Am I right?

Training Problem

Hi, thanks for sharing code! As I follow the tutorial setting , when I run the gta2Cityscapes self-training code, it reports the error as follow. May I get some helpful information from you?

Prior_array.mat file generation

Hi @yzou2 , Can you provide script for prior array generation. Thanks!!

File mine_id.npy and mine_id_priority.npy

I was trying to run the code solve_AO.py . When the subprocess calls the solve_ST.py, it asks for these specific files which I cannot find anywhere in the repo. Can I ask how can I get these files?

unable to download SYNTHIA-RAND-CITYSCAPES dataset

Hi, I am encountering difficulties in downloading the dataset from the provided links on this website: https://synthia-dataset.net/downloads/. I have tried accessing the download links multiple times, but unfortunately, they seem to be inactive or not working properly.
Could you please provide guidance on how I can download the dataset?

VGG16 code

Thanks for sharing the code! I am just wondering if you could also share the code for vgg16 structure?

Pytorch version?

Hi, first of all congratulations for the great work. Do you plan to release a Pytorch version of your implementation? Do you have any estimated release date? Thanks in advance

BDD-V

Could you provide the link of BDD-V dataset as your discription in the paper ?

About the finetuning of self-training

I used your command of
"python issegm/solve_AO.py --num-round 6 --test-scales 1850 --scale-rate-range 0.7,1.3 --dataset gta --dataset-tgt cityscapes --split train --split-tgt val --data-root DATA_ROOT_GTA5 --data-root-tgt DATA_ROOT_CITYSCAPES --output gta2city/cbst --model cityscapes_rna-a1_cls19_s8 --weights models/gta_rna-a1_cls19_s8_ep-0000.params --batch-images 2 --crop-size 500 --origin-size-tgt 2048 --init-tgt-port 0.15 --init-src-port 0.03 --seed-int 0 --mine-port 0.8 --mine-id-number 3 --mine-thresh 0.001 --base-lr 1e-4 --to-epoch 2 --source-sample-policy cumulative --self-training-script issegm/solve_ST.py --kc-policy cb --prefetch-threads 2 --gpus 0 --with-prior False"
in GTA-to-Cityscapes setting and find your code use validation set(cityscapes) instead of training set(cityscapes) to produce pseudo-labels.
Does this accord with your experiment in eccv paper?

SYNTHIA-RAND-CITYSCAPES dataset

Hi, I downloaded SYNTHIA-RAND-CITYSCAPES from the link you provided, but found that the foder doesn't include image and labels. Do you know why? Thanks!

simple_bind error

Hello,

Thanks for your great work, I am trying to running your code.
When I try to run

when goes into line 806 in solver_ST.py， I meet this error:

Have you ever meet same error? or do you have any suggestions to debug?

Save confidence scores of target domain as class-wise vectors

Hi，@yzou2，saving confidence scores of target domain as class-wise vectors as the type of numpy， Isn’t it too big? In my experiment, the cpu utilization reached 200% and it was never unable to continue.

About label format of GTA

@Chrisding @yzou2, what is the expected format of labels for GTA5 training, source only script? I used the labels.py script in order to transform themt o cityscapes format (valid labels 0-18) and the rest 255. Nevertheless, when i start training i got loss values with NaNs.

About the basenet

Sorry to bother you again,I am very interested about your ECCV PAPERDomain Adaptation for Semantic Segmentation via Class-Balanced Self-Training.I found that you have choosed both FCN8s-VGG16，and RESNET-38 as basenet, and this project is based on the RESNET. If I want to reproduce the same result base in FCN8s-VGG16，how can I change your code? It's very pleasant if you can answer me more explicitly since I am a freshman.