- ADer is an open source visual Anomaly Detection toolbox based on PyTorch, which supports multiple popular AD datasets and approaches.
- We reproduce popular AD methods under the Multi-class Unsupervised Anomaly Detection (MUAD) by default.
- We hope it can bring convenience to your research and application.
- ๐ฅ We have released several reproduced models, configuration files, and training logs in our benchmark paper ๐ฒ Paper | Results & CFGs
- ๐ฅ COCO-AD and powerful InvAD is released ๐ฒ Paper | Project | Code
- ๐ฅ Plain ViT based ViTAD is released ๐ฒ Paper | Project | Code
- ๐ฅ Real-IAD is released: a new large-scale challenging industrial AD dataset ๐ฒ Paper | Project | Code
- ๐ Multi-/Single-class Training and Testing
- ๐ Convenient and flexible way to implement a new approach, refer to here.
- Reproduced popular methods in ADer Benchmark:
- ๐Augmentation-based
- ๐Embedding-based
- ๐Reconstruction-based
- ViTAD, arXiv'23
- InvAD, arXiv'24
- InvAD-lite, arXiv'24
- DiAD, AAAI'24: See tripartite implementation in this website
- MambaAD, arXiv'24
- RD, CVPR'22
- ๐Hybrid
- UniAD, NeurIPS'22: download in
TIMM
tomodel/pretrain
- RD++, CVPR'23
- DesTSeg, CVPR'23
- UniAD, NeurIPS'22: download in
- ๐ (Opt.) DDP Training
- By default, the weights used by different methods are automatically downloaded (
model.kwargs.pretrained=True
,model.kwargs.checkpoint_path=''
). If you prefer to specify offline weights, you can download the model weights tomodel/pretrain
and modify the settings tomodel.kwargs.pretrained=False
,model.kwargs.checkpoint_path='model/pretrain/xxx.pth'
.
-
Clone this repo:
git clone https://github.com/zhangzjn/ader.git && cd ader
-
Prepare general experimental environment
pip3 install timm==0.8.15dev0 mmselfsup pandas transformers openpyxl imgaug numba numpy tensorboard fvcore accimage Ninja pip3 install mmdet==2.25.3 pip3 install --upgrade protobuf==3.20.1 scikit-image faiss-gpu pip3 install adeval pip3 install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118 pip3 install fastprogress geomloss FrEIA mamba_ssm adeval fvcore==0.1.5.post20221221 (or) conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia
Please refer to Datasets Description for preparing visual AD datasets as needed.
- Real-IAD: A new large-scale challenging industrial AD dataset, containing 30 classes with totally 151,050 images; 2,000 โผ 5,000 resolution; 0.01% ~ 6.75% defect proportions; 1:1 ~ 1:10 defect ratio.
- COCO-AD: Large-scale and general-purpose challenging AD-adapted dataset.
- [MVTec AD](data/README.md/###MVTec AD): Most popular AD dataset.
- VisA: Popular AD dataset.
- Uni-Medical: Unified medical AD dataset.
- (Opt.) [MVTec 3D-AD](data/README.md/###MVTec 3D-AD): Improved 3D version of MVTec AD.
- (Opt.) Cifar10 & Cifar100: For one-class-train, one-class-test, and unified settings.
- (Opt.) Tiny-ImageNet: A larger one-class dataset.
- Check
data
andmodel
settings for the config fileconfigs/METHOD/METHOD_CFG.py
- Train with single GPU example:
CUDA_VISIBLE_DEVICES=0 python run.py -c configs/METHOD/METHOD_cfg.py -m train
- Train with multiple GPUs (DDP) in one node:
export nproc_per_node=8
export nnodes=1
export node_rank=0
export master_addr=YOUR_MACHINE_ADDRESS
export master_port=12315
python -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs/METHOD/METHOD_CFG.py -m train
.
- Modify
trainer.resume_dir
to resume training. - Single-class Unsupervised AD (SUAD, not recommend currently)
- one
GPU-0
for training onegrid
class inmvtec
dataset:CUDA_VISIBLE_DEVICES=0 python3 run.py -c configs/vitad/single_cls/vitad_mvtec_bs16.py -m train data.cls_names=grid trainer.checkpoint=runs/vitad/single_class/vitad_mvtec_bs16/grid
- one
GPU-0
for training all classes serially inmvtec
dataset:python3 runs_single_class.py -d mvtec -c configs/vitad/single_cls/vitad_mvtec_bs16.py -n 1 -m -1 -g 0
$GPU_NUM
GPUs for training all classes parallelly inmvtec
dataset::python3 runs_single_class.py -d mvtec -c configs/vitad/single_cls/vitad_mvtec_bs16.py -n $GPU_NUM -m 1
- results will be saved in default dir:
runs/vitad/single_cls/vitad_mvtec_bs16
- one
- Modify
trainer.resume_dir
ormodel.kwargs['checkpoint_path']
- Test with single GPU example:
CUDA_VISIBLE_DEVICES=0 python run.py -c configs/METHOD/METHOD_cfg.py -m test
- Test with multiple GPUs (DDP) in one node:
python -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs/METHOD/METHOD_CFG.py -m test
.
- Add a model config
cfg_model_MODEL_NAME
toconfigs/__base__
- Add configs to
configs/MODEL_NAME/CFG.py
for training and testing. - Add a model implementation file
model/MODEL_NAME.py
- Add a trainer implementation file
trainer/MODEL_NAME_trainer.py
- (Optional) Add specific files to
data
,loss
,optim
, etc.
- Red metrics are recommended for comprehensive evaluations.
SubscriptsI
,R
, andP
representimage-level
,region-level
, andpixel-level
, respectively. - The following results are derived from the original paper. Since the benchmark re-runs all experiments on the same
L40S
platform, slight differences in the results are reasonable.
Method | mAU-ROCI | mAPI | mF1-maxI | mAU-PROR | mAU-ROCP | mAPP | mF1-maxP | mF1P/.2/.8 | mAccP/.2/.8 | mIoUP/.2/.8 | mIoU-maxP | mADI | mADP | mAD.2/.8 | mAD | Download |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
UniAD | 97.5 | 99.1 | 97.3 | 90.7 | 97.0 | 45.1 | 50.4 | 22.4 | 37.5 | 13.9 | 34.2 | 98.0 | 64.1 | 24.6 | 82.4 | log & weight |
DiAD | 97.2 | 99.0 | 96.5 | 90.7 | 96.8 | 52.6 | 55.5 | 19.5 | 40.7 | 12.0 | 21.3 | 97.6 | 68.3 | 24.1 | 84.0 | log & weight |
ViTAD | 98.3 | 99.4 | 97.3 | 91.4 | 97.7 | 55.3 | 58.7 | 30.9 | 40.8 | 20.4 | 42.6 | 98.3 | 70.6 | 30.7 | 85.4 | log & weight |
InvAD | 98.9 | 99.6 | 98.1 | 94.1 | 98.2 | 57.6 | 60.1 | 34.6 | 46.9 | 23.0 | 43.7 | 98.9 | 72.0 | 34.8 | 86.7 | log & weight |
InvAD-lite | 98.2 | 99.2 | 97.2 | 97.3 | 55.0 | 58.1 | 92.7 | 32.6 | 47.1 | 21.3 | 41.7 | 98.2 | 68.6 | 33.7 | 85.4 | log & weight |
Method | mAU-ROCI | mAPI | mF1-maxI | mAU-PROR | mAU-ROCP | mAPP | mF1-maxP | mF1P/.2/.8 | mAccP/.2/.8 | mIoUP/.2/.8 | mIoU-maxP | mADI | mADP | mAD.2/.8 | mAD | Download |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
UniAD | 88.8 | 90.8 | 85.8 | 85.5 | 98.3 | 33.7 | 39.0 | 17.9 | 47.1 | 10.9 | 25.7 | 88.4 | 57.0 | 25.3 | 74.5 | log & weight |
DiAD | 86.8 | 88.3 | 85.1 | 75.2 | 96.0 | 26.1 | 33.0 | 13.2 | 46.2 | 8.0 | 16.2 | 86.7 | 51.7 | 22.5 | 70.1 | log & weight |
ViTAD | 90.5 | 91.7 | 86.3 | 85.1 | 98.2 | 36.6 | 41.1 | 21.6 | 38.2 | 13.5 | 27.6 | 89.5 | 58.7 | 24.4 | 75.6 | log & weight |
InvAD | 95.5 | 95.8 | 92.1 | 92.5 | 98.9 | 43.1 | 47.0 | 28.0 | 45.6 | 17.9 | 32.7 | 94.5 | 63.0 | 30.5 | 80.7 | log & weight |
InvAD-lite | 94.9 | 95.2 | 90.8 | 98.6 | 40.3 | 44.3 | 92.5 | 25.8 | 44.2 | 16.1 | 30.0 | 93.6 | 59.0 | 28.7 | 79.5 | log & weight |
Method | mAU-ROCI | mAPI | mF1-maxI | mAU-PROR | mAU-ROCP | mAPP | mF1-maxP | mF1P/.2/.8 | mAccP/.2/.8 | mIoUP/.2/.8 | mIoU-maxP | mADI | mADP | mAD.2/.8 | mAD | Download |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
UniAD | 56.2 | 49.0 | 61.8 | 31.7 | 65.4 | 12.9 | 19.4 | 6.6 | 26.3 | 3.7 | 11.1 | 55.7 | 32.6 | 12.2 | 42.3 | log & weight |
DiAD | 59.0 | 53.0 | 63.2 | 30.8 | 68.1 | 20.5 | 14.2 | 9.6 | 31.1 | 6.1 | 11.6 | 58.4 | 34.2 | 15.6 | 44.1 | log & weight |
ViTAD | 69.3 | 60.4 | 64.9 | 41.0 | 78.3 | 27.9 | 31.9 | 12.4 | 37.4 | 7.2 | 19.8 | 64.9 | 46.0 | 19.0 | 53.4 | log & weight |
InvAD | 65.9 | 57.8 | 64.1 | 44.9 | 73.2 | 19.7 | 25.4 | 12.4 | 37.5 | 7.1 | 15.2 | 62.6 | 39.4 | 19.0 | 50.1 | log & weight |
InvAD-lite | 64.7 | 56.7 | 63.5 | 70.5 | 18.3 | 23.4 | 38.2 | 11.2 | 34.2 | 6.4 | 13.8 | 61.6 | 26.6 | 17.3 | 47.9 | log & weight |
Method | mAU-ROCI | mAPI | mF1-maxI | mAU-PROR | mAU-ROCP | mAPP | mF1-maxP | mF1P/.2/.8 | mAccP/.2/.8 | mIoUP/.2/.8 | mIoU-maxP | mADI | mADP | mAD.2/.8 | mAD | Download |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
UniAD | 78.9 | 93.4 | 91.4 | 88.1 | 96.5 | 21.2 | 28.0 | 12.2 | 43.6 | 7.0 | 16.8 | 87.9 | 48.6 | 20.9 | 71.1 | log & weight |
DiAD | 84.6 | 94.8 | 95.6 | 87.8 | 96.4 | 25.3 | 32.3 | 5.0 | 71.4 | 2.6 | 5.4 | 91.7 | 51.3 | 26.3 | 73.8 | log & weight |
ViTAD | 79.0 | 93.1 | 91.8 | 91.6 | 98.2 | 27.3 | 33.3 | 17.2 | 45.3 | 10.0 | 20.5 | 88.0 | 52.9 | 24.1 | 73.5 | log & weight |
InvAD | 86.1 | 95.8 | 93.2 | 94.7 | 98.8 | 37.8 | 42.5 | 22.0 | 50.5 | 13.2 | 27.5 | 91.7 | 59.7 | 28.6 | 78.4 | log & weight |
InvAD-lite | 85.3 | 95.2 | 93.0 | 98.6 | 37.2 | 41.4 | 94.1 | 21.6 | 55.3 | 12.9 | 26.5 | 91.2 | 57.6 | 30.0 | 77.8 | log & weight |
Method | mAU-ROCI | mAPI | mF1-maxI | mAU-PROR | mAU-ROCP | mAPP | mF1-maxP | mF1P/.2/.8 | mAccP/.2/.8 | mIoUP/.2/.8 | mIoU-maxP | mADI | mADP | mAD.2/.8 | mAD | Download |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
UniAD | 78.5 | 75.2 | 76.6 | 96.4 | 37.6 | 40.2 | 85.0 | 13.3 | 37.8 | 8.0 | 26.8 | 76.8 | 54.3 | 19.7 | 69.9 | log & weight |
DiAD | 85.1 | 84.5 | 81.2 | 85.4 | 95.9 | 38.0 | 35.6 | 19.0 | 57.6 | 11.4 | 25.0 | 83.6 | 56.5 | 29.3 | 72.2 | log & weight |
ViTAD | 82.2 | 81.0 | 80.1 | 97.2 | 49.9 | 49.6 | 86.1 | 18.6 | 36.5 | 11.7 | 35.1 | 81.1 | 61.9 | 22.3 | 75.2 | log & weight |
InvAD | 82.2 | 79.6 | 80.6 | 97.4 | 47.5 | 47.1 | 89.6 | 21.8 | 45.2 | 13.8 | 33.3 | 80.8 | 61.4 | 26.9 | 74.9 | log & weight |
InvAD-lite | 79.5 | 78.3 | 79.1 | 96.4 | 40.1 | 40.4 | 85.5 | 18.3 | 40.5 | 11.2 | 27.6 | 79.0 | 55.3 | 23.3 | 71.3 | log & weight |
Method | mAU-ROCI | mAPI | mF1-maxI | mAU-PROR | mAU-ROCP | mAPP | mF1-maxP | mF1P/.2/.8 | mAccP/.2/.8 | mIoUP/.2/.8 | mIoU-maxP | mADI | mADP | mAD.2/.8 | mAD | Download |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
UniAD | 82.9 | 80.8 | 74.4 | 97.4 | 22.9 | 30.3 | 86.4 | 10.5 | 35.0 | 6.0 | 18.3 | 79.4 | 46.5 | 17.2 | 67.9 | log & weight |
DiAD | 75.6 | 66.4 | 69.9 | 58.1 | 88.0 | 2.9 | 7.1 | 2.9 | 41.9 | 1.5 | 3.7 | 70.6 | 32.7 | 15.4 | 52.6 | log & weight |
ViTAD | 82.7 | 80.1 | 73.7 | 97.3 | 24.2 | 32.3 | 83.9 | 13.4 | 27.3 | 7.7 | 19.6 | 78.8 | 46.8 | 16.1 | 67.8 | log & weight |
InvAD | 89.0 | 86.4 | 79.6 | 98.4 | 30.7 | 37.6 | 91.9 | 17.7 | 36.1 | 10.4 | 23.5 | 85.0 | 53.4 | 21.4 | 73.4 | log & weight |
InvAD-lite | 87.2 | 85.1 | 77.8 | 98.1 | 31.6 | 37.9 | 91.6 | 17.6 | 36.8 | 10.3 | 23.7 | 83.3 | 53.7 | 21.6 | 72.7 | log & weight |
If you use this toolbox or benchmark in your research, please cite our related works.
@article{ader,
title={ADer: A Comprehensive Benchmark for Multi-class Visual Anomaly Detection},
author={Jiangning Zhang and Haoyang He and Zhenye Gan and Qingdong He and Yuxuan Cai and Zhucun Xue and Yabiao Wang and Chengjie Wang and Lei Xie and Yong Liu},
journal={arXiv preprint arXiv:2406.03262},
year={2024}
}
@inproceedings{realiad,
title={Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection},
author={Wang, Chengjie and Zhu, Wenbing and Gao, Bin-Bin and Gan, Zhenye and Zhang, Jianning and Gu, Zhihao and Qian, Shuguang and Chen, Mingang and Ma, Lizhuang},
booktitle={CVPR},
year={2024}
}
@article{vitad,
title={Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detection},
author={Zhang, Jiangning and Chen, Xuhai and Wang, Yabiao and Wang, Chengjie and Liu, Yong and Li, Xiangtai and Yang, Ming-Hsuan and Tao, Dacheng},
journal={arXiv preprint arXiv:2312.07495},
year={2023}
}
@article{invad,
title={Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark},
author={Jiangning Zhang and Chengjie Wang and Xiangtai Li and Guanzhong Tian and Zhucun Xue and Yong Liu and Guansong Pang and Dacheng Tao},
journal={arXiv preprint arXiv:2404.10760},
year={2024}
}
@article{mambaad,
title={MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection},
author={He, Haoyang and Bai, Yuhu and Zhang, Jiangning and He, Qingdong and Chen, Hongxu and Gan, Zhenye and Wang, Chengjie and Li, Xiangtai and Tian, Guanzhong and Xie, Lei},
year={2024}
}