jd730 / eopsn

[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021)

Official PyTorch implementation for EOPSN.

We propose open-set panoptic segmentation task and propose a new baseline called EOPSN. The code is based on Detectron2

---

Usage

First, install requirements.

pip install -r requirements.txt

Then, install PyTorch 1.5+ and torchvision 0.6+:

conda install -c pytorch pytorch torchvision

Finally, you need to install Detectron2. To prevent version conflict, I recommand to install via included detectron2 folders. Regarding installation issue caused from detectron2, please refer to here.

cd detectron2
pip install -e ./

Data preparation

Download and extract COCO 2017 train and val images with annotations from http://cocodataset.org. We expect the directory structure to be the following:

datasets/coco
  annotations/  # annotation json files
  train2017/    # train images
  val2017/      # val images

To convert closed-set panoptic segmentation to open-set panoptic segmentation, run:

python prepare_unknown.py

The default setting is K=20, you can change here.

Training

To train EOPSN on a single node with 8 gpus for 30,000 iterations run:

python train_net.py --config configs/EOPSN_K20.yaml --num-gpus 8

Note that it requires pre-trained models (Void-suppression). Please download from Goolge Drive.

To train baseline (train) on a single node with 8 gpus for 45,000 iterations run:

python train_net.py --config configs/baseline_K20.yaml --num-gpus 8

If you want to log using WandB, you can add --wandb flag.

Evaluation

To evaluate EOPSN on COCO val5k with a single GPU run:

python train_net.py --config configs/EOPSN_K20.yaml --num-gpus 8 --resume --eval-only

Citations

@inproceedings{hwang2021exemplar,
    author = {Hwang, Jaedong and Oh, Seoung Wug and Lee, Joon-Young and Han, Bohyung},
    title = {Exemplar-Based Open-Set Panoptic Segmentation Network},
    booktitle = {CVPR},
    year = {2021},
}   

License

EOPSN is released under the CC BY-NC-SA 4.0 license. Please see the LICENSE file for more information. The detectron2 part is released under the Apache 2.0 license. Please see the detectron2/LICENSE file for more information.

Contributing

We actively welcome your pull requests!

eopsn's Issues

train problems

How to solve the problem of insufficient video memory? Thanks!

About the pre-trained model.

Hi @jd730, thanks for sharing your wonderful work.
In the README, you said the training requires "void-suppression" pre-trained model. However, I find that in the config file, you still use the standard pre-trained model here.
If I want to train your EOPSN, which pre-trained model should I use?
Thx.

Adoption to cityscapes

Hi,
thanks for providing the code to your method. I am interested in trying this out on another dataset, namely cityscapes. I already know that I would have to adapt the dataloading to cityscapes, what I am still unsure about is which pretrained model I would need.

In this issue you mention that training needs a model pretrained with void-supression, the weights of which you provide. How would I go about pretraining such a model on cityscapes? Is there code in this repository for pretraining on COCO, which I could adapt to cityscapes?

test problems

请问是否有训练好的模型能直接让我们跑测试？

Problems of reproducing the results of EOPSN

Hi,

Thanks for the interesting paper and open-sourced code.

Recently, I ran the EOPSN method on K20 setting folllowing the given guideline (w/o any editing) and I found the results of unknown things are quite different from the reported one.

Unk	PQ	SQ	RQ
EOPSN reported	11.3	73.8	15.3
EOPSN reproduced	15.6	79.2	19.6

From the table, it seems that the released code achieves a much better improvement than the reported one. However, when I further inspect the predictions of class-wise unkown things, it seems that EOPSN's unkown recognition is dominated by the "car" class and other unkown classes are rarely detected. Moreover, the reproduced results may not support the visualization results in Fig5 since several unkown classes are shown to be detected, e.g., stop sign, keyboard, banana, and toilet. So, could you please release the EOPSN checkpoint which supports the reported results? Thanks a lot.

BTW, I found that the training of EOPSN requires the pre-trained model of Void-Suppression, but the current released codebase only contains the void-train method. I wonder could you please release the void-suppression code for better reproduction? Thanks again.

FYI, the predictions of class-wise unkown things on Void-Suppression method are as follows and the results are identical to the reported ones in the paper

Infinity losses during training

When running the training, I noticed that sometimes some losses (especially the box regression loss) can become Infinity. This seems to happen especially once many exemplars have been mined. I was wondering if you encountered this issue as well, and if you know of any fixes.