hsulee-lx / standardized-max-logits

Official PyTorch implementation of paper: Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation (ICCV 2021 Oral Presentation)

SML (ICCV 2021, Oral) : Official Pytorch Implementation

This repository provides the official PyTorch implementation of the following paper:

Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation
Sanghun Jung* (KAIST AI), Jungsoo Lee* (KAIST AI), Daehoon Gwak (KAIST AI)
Sungha Choi (LG AI Research), and Jaegul Choo (KAIST AI) (*: equal contribution)
ICCV 2021 (Oral)

Paper: arxiv

Youtube Video (English): Youtube

Abstract: Identifying unexpected objects on roads in semantic segmentation (e.g., identifying dogs on roads) is crucial in safety-critical applications. Existing approaches use images of unexpected objects from external datasets or require additional training (e.g., retraining segmentation networks or training an extra network), which necessitate a non-trivial amount of labor intensity or lengthy inference time. One possible alternative is to use prediction scores of a pre-trained network such as the max logits (i.e., maximum values among classes before the final softmax layer) for detecting such objects. However, the distribution of max logits of each predicted class is significantly different from each other, which degrades the performance of identifying unexpected objects in urban-scene segmentation. To address this issue, we propose a simple yet effective approach that standardizes the max logits in order to align the different distributions and reflect the relative meanings of max logits within each predicted class. Moreover, we consider the local regions from two different perspectives based on the intuition that neighboring pixels share similar semantic information. In contrast to previous approaches, our method does not utilize any external datasets or require additional training, which makes our method widely applicable to existing pre-trained segmentation models. Such a straightforward approach achieves a new state-of-the-art performance on the publicly available Fishyscapes Lost & Found leaderboard with a large margin.

Code Contributors

Sanghun Jung [Website] [LinkedIn] [Google Scholar] (KAIST AI)
Jungsoo Lee [Website] [LinkedIn] [Google Scholar] (KAIST AI)

Concept Video

Click the figure to watch the youtube video of our paper!

Pytorch Implementation

Installation

Clone this repository.

git clone https://github.com/shjung13/Standardized-max-logits.git
cd Standardized-max-logits
pip install -r requirements.txt

Cityscapes data directory

cityscapes
 └ leftImg8bit_trainvaltest
   └ leftImg8bit
     └ train
     └ val
     └ test
 └ gtFine_trainvaltest
   └ gtFine
     └ train
     └ val
     └ test

OoD data directory

Fishyscapes (OoD Dataset)
 └ leftImg8bit_trainvaltest
   └ leftImg8bit
     └ val
 └ gtFine_trainvaltest
   └ gtFine
     └ val

How to Run

Train the segmentation model

Please download resnet101 network pretrained on ImageNet dataset from this link and place it in pretrained/resnet101-imagenet.pth path. Then run the below command.

CUDA_VISIBLE_DEVICES=0,1 ./scripts/train_r101_os8.sh

Obtain statistics from training samples

CUDA_VISIBLE_DEVICES=0 ./scripts/calc_stat_r101_os8.sh

Evaluate on Out-of-Distribution dataset

Download the pretrained model here and after creating "<Directory Home>/pretrained", place it under the folder.

CUDA_VISIBLE_DEVICES=0 python eval.py --ood_dataset_path <path_to_OoD_dataset>

Quantitative / Qualitative Evaluation

Fishyscapes Learboard

Identified OoD pixels (colored white)

Fishyscapes Leaderboard

Our result is also available at fishyscapes.com.

Citation

@InProceedings{Jung_2021_ICCV,
    author    = {Jung, Sanghun and Lee, Jungsoo and Gwak, Daehoon and Choi, Sungha and Choo, Jaegul},
    title     = {Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {15425-15434}
}

Acknowledgments

We deeply appreciate Hermann Blum and FishyScapes team for their sincere help in providing the baseline performances and helping our team to update our model on the FishyScapes Leaderboard. Our pytorch implementation is heavily derived from NVIDIA segmentation and RobustNet. Thanks to the NVIDIA implementations.