trellixvulnteam / mmseg-combine-cls_rir1

该mmseg为处理ocr版面分割所用的主要框架，对其进行一些修改： 环境配置 : cuda 10.0，gcc 4.9.2， torch 1.3， torchvision 0.4.1 mmcv 0.3.1 mmcv-full 1.3.7

• 分层多尺度

  1. 文件目录303行见注释 Mmseg/mmseg/models/segmentors/encoder_decoder.py
  
  
  

• mmseg分类与分割并行

  1. 在voc格式数据集中指定位置，需要有cls-label.json。包含了每张图对应的标签，通用文档-0，ppt-1，格式如下。
  
  
  
  
  
  2. 在Mmseg/mmseg/datasets/custom.py中对cls-label.json 进行读取并保存，用于计算指标，保存方式如下
  
  
  3. 训练时需要用到cls-label，(在200行)准备训练数据时将cls-label加入到img-metas中去
  
  
  4. 测试时需要对分类结果进行accuracy指标输出。在318行编写了cls-acc函数用于计算精度
  
  
  5. 在/Mmseg/mmseg/models/segmentors/encoder_decoder.py中，设置cls-head和lossfn用于计算分类的out和loss，由于分类与分割公用backbone，所以decode-head的输入通道数即为cls-head中全连接层的输入维度。
  
  
  6. 在160行进行分类loss，acc的计算并更新到total-loss中用于整体梯度更新
  
  
  7. 在测试过程中增加返回值cls-out，同时更改所用到的所有测试流程函数
  
  
  8. Mmseg/mmseg/apis/test.py中保存整体cls的预测label，最后回到datasets进行计算指标
  
  
  

• 已尝试内容

  1. 模型：
  • deeplabv3+(resnet)
  • deeplabv3+(resnet+dcn)
  • deeplabv3+(resnest)
  • vit
  • swin
  • setr
  • hrnet
  • 对比上述模型v3+（resnet）在ocr版面分割任务中前向速度最快，指标也最高
  2. 方法：
  • 分层多尺度：见上面1（有效）
  • 标签腐蚀：固定中心坐标不变，将每一个目标的mask覆盖区域进行收缩处理，原图不进行处理。（有效）
  
  
  
  • booststrap：标签腐蚀的升级版，将mask进行随机大幅度裁剪而非小尺度收缩，帮助网络学习目标特征，之后对原图进行背景均化（有效）
  • booststrap前
  
  
  
  • booststrap后
  
  
  

• trace流程

  • 在Mmseg/trace_tools下
    • trace_cpu.py ===== 将pth模型转换为pt模型 --model_path： pth模型地址 --dst-trace-path：想要保存的pt模型文件 --config-path: pth模型对应的config文件 -- img-path： 任意图片
    • trace-cpu-valid.py ===== 验证pth模型与pt模型是否一致 --model-path： pt模型地址 --pytorch-model-path： pth模型地址 --config-path：pth模型对应的config -- img-path：任意图片 --output-path：可视化输出目录 --cls-dict： 分类标签对应内容

• 官方Mmsegmentation指引

Documentation: https://mmsegmentation.readthedocs.io/

English | 简体中文

Introduction

MMSegmentation is an open source semantic segmentation toolbox based on PyTorch. It is a part of the OpenMMLab project.

The master branch works with PyTorch 1.3+.

Major features

• Unified Benchmark

  We provide a unified benchmark toolbox for various semantic segmentation methods.

• Modular Design

  We decompose the semantic segmentation framework into different components and one can easily construct a customized semantic segmentation framework by combining different modules.

• Support of multiple methods out of box

  The toolbox directly supports popular and contemporary semantic segmentation frameworks, e.g. PSPNet, DeepLabV3, PSANet, DeepLabV3+, etc.

• High efficiency

  The training speed is faster than or comparable to other codebases.

License

This project is released under the Apache 2.0 license.

Changelog

v0.14.1 was released in 06/16/2021. Please refer to changelog.md for details and release history.

Benchmark and model zoo

Results and models are available in the model zoo.

Supported backbones:

• ResNet (CVPR'2016)
• ResNeXt (CVPR'2017)
• HRNet (CVPR'2019)
• ResNeSt (ArXiv'2020)
• MobileNetV2 (CVPR'2018)
• MobileNetV3 (ICCV'2019)
• [Vision Transformer (ICLR'2021)]

Supported methods:

• FCN (CVPR'2015/TPAMI'2017)
• UNet (MICCAI'2016/Nat. Methods'2019)
• PSPNet (CVPR'2017)
• DeepLabV3 (ArXiv'2017)
• Mixed Precision (FP16) Training (ArXiv'2017)
• PSANet (ECCV'2018)
• DeepLabV3+ (CVPR'2018)
• UPerNet (ECCV'2018)
• NonLocal Net (CVPR'2018)
• EncNet (CVPR'2018)
• Semantic FPN (CVPR'2019)
• DANet (CVPR'2019)
• APCNet (CVPR'2019)
• EMANet (ICCV'2019)
• CCNet (ICCV'2019)
• DMNet (ICCV'2019)
• ANN (ICCV'2019)
• GCNet (ICCVW'2019/TPAMI'2020)
• Fast-SCNN (ArXiv'2019)
• OCRNet (ECCV'2020)
• DNLNet (ECCV'2020)
• PointRend (CVPR'2020)
• CGNet (TIP'2020)
• SETR (CVPR'2021)

Installation

Please refer to get_started.md for installation and dataset_prepare.md for dataset preparation.

Get Started

Please see train.md and inference.md for the basic usage of MMSegmentation. There are also tutorials for customizing dataset, designing data pipeline, customizing modules, and customizing runtime. We also provide many training tricks.

A Colab tutorial is also provided. You may preview the notebook here or directly run on Colab.

Citation

If you find this project useful in your research, please consider cite:

@misc{mmseg2020,
    title={{MMSegmentation}: OpenMMLab Semantic Segmentation Toolbox and Benchmark},
    author={MMSegmentation Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmsegmentation}},
    year={2020}
}

Contributing

We appreciate all contributions to improve MMSegmentation. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

MMSegmentation is an open source project that welcome any contribution and feedback. We wish that the toolbox and benchmark could serve the growing research community by providing a flexible as well as standardized toolkit to reimplement existing methods and develop their own new semantic segmentation methods.

Projects in OpenMMLab

• MMCV: OpenMMLab foundational library for computer vision.
• MMClassification: OpenMMLab image classification toolbox and benchmark.
• MMDetection: OpenMMLab detection toolbox and benchmark.
• MMDetection3D: OpenMMLab's next-generation platform for general 3D object detection.
• MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark.
• MMAction2: OpenMMLab's next-generation action understanding toolbox and benchmark.
• MMTracking: OpenMMLab video perception toolbox and benchmark.
• MMPose: OpenMMLab pose estimation toolbox and benchmark.
• MMEditing: OpenMMLab image and video editing toolbox.
• MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding.
• MMGeneration: A powerful toolkit for generative models.