tvphu213 / data-competition

Table of contents

1. Introduction
2. Dataset
3. Model & Metrics
4. How to Run
   • Quickstart
   • Install
   • Training
   • Evaluation
   • Detection

DATA COMPETITION

The COVID-19 pandemic, which is caused by the SARS-CoV-2 virus, is still continuing strong, infecting hundreds of millions of people and killing millions. Face masks reduce transmission by preventing aerosols and droplets from spreading too far into the atmosphere. As a result, there is a growing demand for automated systems that can detect whether people are not wearing masks or are wearing masks incorrectly. This competition was designed in order to solve the problem mentioned above. This competition is unlike any other that has come before it. With a fixed model, participants will receive model code and configuration code that organizers use to train models. The candidate's task is to use data processing and generation techniques to improve the model's performance, then submit the dataset to the organizing team for training and evaluation on the private test set. The winner is the team with the highest score on the private test set.

Dataset

• A dataset of 1100 images will be sent to you. This is an object detection dataset consisting of employee images at the office. The dataset has been assigned 3 labels by us which are no mask, mask, and incorrect mask, with the numbers 0,1,2 corresponding to each.

• The dataset has been divided into three parts for you: train, valid, and public test. We have prepared a private test to be able to evaluate the candidate's model. This private test will be made public after the contest ends. In the public test, you can get a basic idea of the private test. Download the dataset here

• To improve the model's performance, you can re-label it and employ data augmentation to generate more images (up to 3000).

The number of each label in each part is shown below:

	No mask	Mask	incorrect mask
Train	308	882	51
Val	97	190	9
Public_test	47	95	13

Model & Metrics

• The challenge is defined as object detection challenge. In the competition, We use YOLOv5s and also use a pre-trained model trained with easy mask dataset to greatly reduce training time.

• We fix all hyperparameters of the model and do not use any augmentation tips in the source code. Therefore, each participant need to build the best possible dataset by relabeling incorrect labels, splitting train/val, augmentation tips, adding new dataset, etc.

• In training process, Early Stopping method with patience setten to 100 iterations is used to keep track of validation set's [email protected]. Detail about [email protected] metric:

[email protected] = [email protected] = 0.2 * AP50_w + 0.3 * AP50_nw + 0.5 * AP50_wi

Where,
AP50_w: AP50 on valid mask boxes
AP50_nw: AP50 on non-mask boxes
AP50_wi: AP50 on invalid mask boxes

• The [email protected] metric is also used as the main metric to evaluate participant's submission on private testing set.

How to Run

QuickStart

Click the image below

Install requirements

• All requirements are included in requirements.txt

• Run the script below to clone and install all requirements

git clone https://github.com/fsoft-ailab/Data-Competition
cd Data-Competition
pip3 install -r requirements.txt

Training

• Put your dataset into the Data-Competition folder. The structure of dataset folder is followed as folder structure below:

folder-name
├── images
│   ├── train
│   │   ├── train_img1.jpg
│   │   ├── train_img2.jpg
│   │   └── ...
│   │   
│   └── val
│       ├── val_img1.jpg
│       ├── val_img2.jpg
│       └── ...
│   
└── labels
    ├── train
    │   ├── train_img1.txt
    │   ├── train_img2.txt
    │   └── ...
    │   
    └── val
        ├── val_img1.txt
        ├── val_img2.txt
        └── ...

• Change relative paths to train and val images folder in config/data_cfg.yaml file

• train_cfg.yaml where we set up the model during training. You should not change such hyperparameters because it will result in incorrect results. The training results are saved in the results/train/<version_name>.

• Run the script below to train the model. Specify particular name to identify your experiment:

python3 train.py --batch-size 64 --device 0 --name <version_name> 

Note: If you get out of memory error, you can decrease batch-size to multiple of 2 as 32, 16.

Evaluation

• Run script below to evaluate on particular dataset.
• The --task's value is only one of train, val, or test, respectively evaluating on the training set, validation set, or public testing set.
• Note: Specify relative path to images folder which you evaluate in config/data_cfg.yaml file.

python3 val.py --weights <path_to_weight> --task test --name <version_name> --batch-size 64 --device 0
                                                 val
                                                 train

• Results are saved at results/evaluate/<task>/<version_name>.

Detection

• You can use this script to make inferences on particular folder

• Results are saved at <save_dir>.

python3 detect.py --weights <path_to_weight> --source <path_to_folder> --dir <save_dir> --device 0

• You can find more default arguments at detect.py

References

• Our source code is based on Ultralytics's implementation: https://github.com/ultralytics/yolov5
• ScaledYOLOV4: https://github.com/WongKinYiu/ScaledYOLOv4