megvii-research / motrv2

[CVPR2023] MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors

MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors

This repository is an official implementation of MOTRv2.

Introduction

TL; DR. MOTRv2 improve MOTR by utilizing YOLOX to provide detection prior.

Abstract. In this paper, we propose MOTRv2, a simple yet effective pipeline to bootstrap end-to-end multi-object tracking with a pretrained object detector. Existing end-to-end methods, e.g. MOTR and TrackFormer are inferior to their tracking-by-detection counterparts mainly due to their poor detection performance. We aim to improve MOTR by elegantly incorporating an extra object detector. We first adopt the anchor formulation of queries and then use an extra object detector to generate proposals as anchors, providing detection prior to MOTR. The simple modification greatly eases the conflict between joint learning detection and association tasks in MOTR. MOTRv2 keeps the end-to-end feature and scales well on large-scale benchmarks. MOTRv2 achieves the top performance (73.4% HOTA) among all existing methods on the DanceTrack dataset. Moreover, MOTRv2 reaches state-of-the-art performance on the BDD100K dataset. We hope this simple and effective pipeline can provide some new insights to the end-to-end MOT community.

News

• 2023.02.28 MOTRv2 is accepted to CVPR 2023.
• 2022.11.18 MOTRv2 paper is available on arxiv.
• 2022.10.27 Our DanceTrack challenge tech report is released [arxiv] [ECCVW Challenge].
• 2022.10.05 MOTRv2 achieved the 1st place in the 1st Multiple People Tracking in Group Dance Challenge.

Main Results

DanceTrack

HOTA	DetA	AssA	MOTA	IDF1	URL
69.9	83.0	59.0	91.9	71.7	model

Visualization

SORT-like SoTA	MOTRv2

Installation

The codebase is built on top of Deformable DETR and MOTR.

Requirements

• Install pytorch using conda (optional)

  conda create -n motrv2 python=3.7
  conda activate motrv2
  conda install pytorch=1.8.1 torchvision=0.9.1 cudatoolkit=10.2 -c pytorch

• Other requirements

  pip install -r requirements.txt

• Build MultiScaleDeformableAttention

  cd ./models/ops
  sh ./make.sh

Usage

Dataset preparation

1. Download YOLOX detection from here.
2. Please download DanceTrack and CrowdHuman and unzip them as follows:

/data/Dataset/mot
├── crowdhuman
│   ├── annotation_train.odgt
│   ├── annotation_trainval.odgt
│   ├── annotation_val.odgt
│   └── Images
├── DanceTrack
│   ├── test
│   ├── train
│   └── val
├── det_db_motrv2.json

You may use the following command for generating crowdhuman trainval annotation:

cat annotation_train.odgt annotation_val.odgt > annotation_trainval.odgt

Training

You may download the coco pretrained weight from Deformable DETR (+ iterative bounding box refinement), and modify the --pretrained argument to the path of the weight. Then training MOTR on 8 GPUs as following:

./tools/train.sh configs/motrv2.args

Inference on DanceTrack Test Set

# run a simple inference on our pretrained weights
./tools/simple_inference.sh ./motrv2_dancetrack.pth

# Or evaluate an experiment run
# ./tools/eval.sh exps/motrv2/run1

# then zip the results
zip motrv2.zip tracker/ -r

Acknowledgements

• MOTR
• ByteTrack
• YOLOX
• OC-SORT
• DanceTrack
• BDD100K

motrv2's Issues

YOLOX version

Hi, is the performance listed in your paper achieved by yolox-x?

This can get HOTA 69.9?

because i look for the training steps you provided and the use of yolox ，but can't get the HOTA of 69.9 ,So I want to ask if the code you provided can get 69.9, Thanks!!!

about YOLOX

您好，请问YOLOX是需要按照所给链接里的说明进行安装么

No checkpoint

Below error obtained while running bash tools/eval_dance.sh exps/motrv2/run1

Traceback (most recent call last):
File "submit_dance.py", line 194, in
checkpoint = torch.load(args.resume, map_location='cpu')
File "/home/bajrangi/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/serialization.py", line 579, in load
with _open_file_like(f, 'rb') as opened_file:
File "/home/bajrangi/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/serialization.py", line 230, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/home/bajrangi/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/serialization.py", line 211, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'checkpoint.pth'

• popd
  ~/MOTRv2-main

We downloaded the dance track model into the current working directory and given the same path, we are getting this error.

Metrics Calculation

Hi,

Thanks for sharing your great work. I am performing the experiments and unable to get metrics , can you tell me how I will able to get the metrics ?

file not found

I have downloaded crowdhuman dataset ,but in that DataSet ,282555,bd4210009cb2ad0b.jpg,284193,20d2d000d1596356.jpg files are not found.
can you provided the updated link of dataset of crowdhuman if any. so that i can solve my issue, or suggest me an appropriate solution.

您好，我无法在12g显存的2080ti上训练，只能在v100或3090上训练

问题如题目

如何使用到视频文字追踪中

我目前正研究密集文字追踪任务，我将MOTRv2的代码替换原来的MOTR代码部分，但是效果不如原来motr的优秀，鲁棒性非常差，很多文字仍然检测不到。请问我应该从哪些方面去排查呢？

Is there no code to make a demo video?

I want to see the performance of the model intuitively.

motr.py中iterative bounding box refinement是否多余？

请教下，我调试代码时发现，在decoder中已经做了iterative bounding box refinement，但是motr.py中又做了一次，这个实现是否重复了？
deformable_transformer_plus.py +429

# hack implementation for iterative bounding box refinement
if self.bbox_embed is not None:
    tmp = self.bbox_embed[lid](output)
    if reference_points.shape[-1] == 4:
        new_reference_points = tmp + inverse_sigmoid(reference_points)
        new_reference_points = new_reference_points.sigmoid()
    else:
        assert reference_points.shape[-1] == 2
        new_reference_points = tmp
        new_reference_points[..., :2] = tmp[..., :2] + inverse_sigmoid(reference_points)
        new_reference_points = new_reference_points.sigmoid()
    reference_points = new_reference_points.detach()

if self.return_intermediate:
    intermediate.append(output)
    intermediate_reference_points.append(reference_points)

motr.py +542

for lvl in range(hs.shape[0]):
    if lvl == 0:
        reference = init_reference
    else:
        reference = inter_references[lvl - 1]
    reference = inverse_sigmoid(reference)
    outputs_class = self.class_embed[lvl](hs[lvl])
    tmp = self.bbox_embed[lvl](hs[lvl])
    if reference.shape[-1] == 4:
        tmp += reference
    else:
        assert reference.shape[-1] == 2
        tmp[..., :2] += reference
    outputs_coord = tmp.sigmoid()
    outputs_classes.append(outputs_class)
    outputs_coords.append(outputs_coord)

This is my current problem now, just says subprocess.CalledProcessError.

• PY_ARGS=
• set -o pipefail
  ++ echo configs/motrv2.args
  ++ sed -e s/configs/exps/g
  ++ sed -e 's/.args$//g'
• OUTPUT_BASE=exps/motrv2
• mkdir -p exps/motrv2
  ++ seq 100
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run1
• OUTPUT_DIR=exps/motrv2/run1
• mkdir exps/motrv2/run1
• break
• echo 'Backing up to log dir: exps/motrv2/run1'
  Backing up to log dir: exps/motrv2/run1
• rmpyc
  ++ find -name pycache
• rm -rf
  ++ find -name '*.pyc'
• rm -rf
• cp -r models datasets util main.py engine.py submit_dance.py configs/motrv2.args exps/motrv2/run1
• echo ' ...Done'
  ...Done
  ++ cat configs/motrv2.args
• args='--meta_arch motr
  --dataset_file e2e_dance
  --epoch 5
  --with_box_refine
  --lr_drop 4
  --lr 2e-4
  --lr_backbone 2e-5
  --pretrained mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth
  --batch_size 128
  --sample_mode random_interval
  --sample_interval 10
  --sampler_lengths 5
  --merger_dropout 0
  --dropout 0
  --random_drop 0.1
  --fp_ratio 0.3
  --query_interaction_layer QIMv2
  --query_denoise 0.05
  --num_queries 10
  --append_crowd
  --det_db det_db_motrv2.json
  --use_checkpoint '
• pushd exps/motrv2/run1
  ~/MOTRv2/exps/motrv2/run1 ~/MOTRv2
• trap cleanup EXIT
• echo 'Logging git status'
  Logging git status
• git status
• git rev-parse HEAD
• git diff
• echo
• echo ' ...Done'
  ...Done
• OMP_NUM_THREADS=1
• python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --meta_arch motr --dataset_file e2e_dance --epoch 5 --with_box_refine --lr_drop 4 --lr 2e-4 --lr_backbone 2e-5 --pretrained mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth --batch_size 128 --sample_mode random_interval --sample_interval 10 --sampler_lengths 5 --merger_dropout 0 --dropout 0 --random_drop 0.1 --fp_ratio 0.3 --query_interaction_layer QIMv2 --query_denoise 0.05 --num_queries 10 --append_crowd --det_db det_db_motrv2.json --use_checkpoint --output_dir .
• tee -a output.log
  | distributed init (rank 2): env://
  | distributed init (rank 0): env://
  | distributed init (rank 1): env://
  | distributed init (rank 3): env://
  Traceback (most recent call last):
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
  "main", mod_spec)
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/runpy.py", line 85, in _run_code
  exec(code, run_globals)
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
  main()
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
  cmd=cmd)
  subprocess.CalledProcessError: Command '['/data1/dcy/anaconda3/envs/motrv2/bin/python', '-u', 'main.py', '--meta_arch', 'motr', '--dataset_file', 'e2e_dance', '--epoch', '5', '--with_box_refine', '--lr_drop', '4', '--lr', '2e-4', '--lr_backbone', '2e-5', '--pretrained', 'mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth', '--batch_size', '128', '--sample_mode', 'random_interval', '--sample_interval', '10', '--sampler_lengths', '5', '--merger_dropout', '0', '--dropout', '0', '--random_drop', '0.1', '--fp_ratio', '0.3', '--query_interaction_layer', 'QIMv2', '--query_denoise', '0.05', '--num_queries', '10', '--append_crowd', '--det_db', 'det_db_motrv2.json', '--use_checkpoint', '--output_dir', '.']' died with <Signals.SIGSEGV: 11>.
• cleanup
• echo 'Packing source code'
  Packing source code
• rmpyc
  ++ find -name pycache
• rm -rf ./pycache ./datasets/pycache ./models/pycache ./models/ops/functions/pycache ./models/ops/modules/pycache ./models/structures/pycache ./util/pycache
  ++ find -name '*.pyc'
• rm -rf
• echo ' ...Done'
  ...Done

多类型目标跟踪

你好，我想问一下，这个项目可不可以支持多类型的目标跟踪？如果可以，需要怎么配置？

Question about evaluation scrips.

I couldn't find evaluation related code or files to evaluate on Dancetrack. Could you please provide it? Looking forward to your reply, thanks.

Do you use detection or track bbox for proposal generation?

Hi, I'm a little confused how to generate det_bbox.json for Dancetrack?
The first question is what is the yolox weight? Is it obtained by pretraining Bytetrack on Dancetrack dataset? Do you use other datasets to train Bytetrack?
The second question is do you use the detection bbox of yolox or use the track bbox of Bytetrack? I am wondering whether track bbox will be better?

attn_mask not work

MOTRv2/models/motr.py

Line 672 in be49b73

frame_res = self._forward_single_image(frame, tmp, gtboxes)

当gtboxes为none或为噪声的GT， frame_res['pred_logits']值差别很大。

How to solve this problem?

The following is my problem:
(motrv2) dcy@ubuntu:~/MOTRv2$ bash tools/train.sh configs/motrv2.args

• PY_ARGS=
• set -o pipefail
  ++ echo configs/motrv2.args
  ++ sed -e s/configs/exps/g
  ++ sed -e 's/.args$//g'
• OUTPUT_BASE=exps/motrv2
• mkdir -p exps/motrv2
  ++ seq 100
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run1
  run1
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run2
  run2
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run3
  run3
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run4
• OUTPUT_DIR=exps/motrv2/run4
• mkdir exps/motrv2/run4
• break
• echo 'Backing up to log dir: exps/motrv2/run4'
  Backing up to log dir: exps/motrv2/run4
• rmpyc
  ++ find -name pycache
• rm -rf
  ++ find -name '*.pyc'
• rm -rf
• cp -r models datasets util main.py engine.py eval_detr.py seqmap submit_dance.py configs/motrv2.args exps/motrv2/run4
  cp: cannot stat 'eval_detr.py': No such file or directory
  cp: cannot stat 'seqmap': No such file or directory
• echo ' ...Done'
  ...Done
  ++ cat configs/motrv2.args
• args='--meta_arch motr
  --dataset_file e2e_dance
  --epoch 5
  --with_box_refine
  --lr_drop 4
  --lr 2e-4
  --lr_backbone 2e-5
  --pretrained mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth
  --batch_size 128
  --sample_mode random_interval
  --sample_interval 10
  --sampler_lengths 5
  --merger_dropout 0
  --dropout 0
  --random_drop 0.1
  --fp_ratio 0.3
  --query_interaction_layer QIMv2
  --query_denoise 0.05
  --num_queries 10
  --append_crowd
  --det_db det_db_motrv2.json
  '-use_checkpoint
• pushd exps/motrv2/run4
  ~/MOTRv2/exps/motrv2/run4 ~/MOTRv2
• trap cleanup EXIT
• echo 'Logging git status'
  Logging git status
• git status
• git rev-parse HEAD
• git diff
• echo
• echo ' ...Done'
  ...Done
• tee -a output.log
• python -m torch.distributed.run --nproc_per_node=6 --use_env main.py --meta_arch $'motr\r' --dataset_file $'e2e_dance\r' --epoch $'5\r' $'--with_box_refine\r' --lr_drop $'4\r' --lr $'2e-4\r' --lr_backbone $'2e-5\r' --pretrained $'mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth\r' --batch_size $'128\r' --sample_mode $'random_interval\r' --sample_interval $'10\r' --sampler_lengths $'5\r' --merger_dropout $'0\r' --dropout $'0\r' --random_drop $'0.1\r' --fp_ratio $'0.3\r' --query_interaction_layer $'QIMv2\r' --query_denoise $'0.05\r' --num_queries $'10\r' $'--append_crowd\r' --det_db $'det_db_motrv2.json\r' $'--use_checkpoint\r' --output_dir .
  /data1/dcy/anaconda3/envs/motrv2/bin/python: No module named torch.distributed.run
• cleanup
• echo 'Packing source code'
  Packing source code
• rmpyc
  ++ find -name pycache
• rm -rf
  ++ find -name '*.pyc'
• rm -rf
• echo ' ...Done'
  ...Done
  (motrv2) dcy@ubuntu:~/MOTRv2$ bash tools/train.sh configs/motrv2.args
• PY_ARGS=
• set -o pipefail
  ++ echo configs/motrv2.args
  ++ sed -e s/configs/exps/g
  ++ sed -e 's/.args$//g'
• OUTPUT_BASE=exps/motrv2
• mkdir -p exps/motrv2
  ++ seq 100
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run1
  run1
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run2
  run2
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run3
  run3
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run4
  run4
• continue
• for RUN in $(seq 100)
• ls exps/motrv2
• grep run5
• OUTPUT_DIR=exps/motrv2/run5
• mkdir exps/motrv2/run5
• break
• echo 'Backing up to log dir: exps/motrv2/run5'
  Backing up to log dir: exps/motrv2/run5
• rmpyc
  ++ find -name pycache
• rm -rf
  ++ find -name '*.pyc'
• rm -rf
• cp -r models datasets util main.py engine.py eval_detr.py seqmap submit_dance.py configs/motrv2.args exps/motrv2/run5
  cp: cannot stat 'eval_detr.py': No such file or directory
  cp: cannot stat 'seqmap': No such file or directory
• echo ' ...Done'
  ...Done
  ++ cat configs/motrv2.args
• args='--meta_arch motr
  --dataset_file e2e_dance
  --epoch 5
  --with_box_refine
  --lr_drop 4
  --lr 2e-4
  --lr_backbone 2e-5
  --pretrained mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth
  --batch_size 128
  --sample_mode random_interval
  --sample_interval 10
  --sampler_lengths 5
  --merger_dropout 0
  --dropout 0
  --random_drop 0.1
  --fp_ratio 0.3
  --query_interaction_layer QIMv2
  --query_denoise 0.05
  --num_queries 10
  --append_crowd
  --det_db det_db_motrv2.json
  '-use_checkpoint
• pushd exps/motrv2/run5
  ~/MOTRv2/exps/motrv2/run5 ~/MOTRv2
• trap cleanup EXIT
• echo 'Logging git status'
  Logging git status
• git status
• git rev-parse HEAD
• git diff
• echo
• echo ' ...Done'
  ...Done
• python -m torch.distributed.launch --nproc_per_node=6 --use_env main.py --meta_arch $'motr\r' --dataset_file $'e2e_dance\r' --epoch $'5\r' $'--with_box_refine\r' --lr_drop $'4\r' --lr $'2e-4\r' --lr_backbone $'2e-5\r' --pretrained $'mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth\r' --batch_size $'128\r' --sample_mode $'random_interval\r' --sample_interval $'10\r' --sampler_lengths $'5\r' --merger_dropout $'0\r' --dropout $'0\r' --random_drop $'0.1\r' --fp_ratio $'0.3\r' --query_interaction_layer $'QIMv2\r' --query_denoise $'0.05\r' --num_queries $'10\r' $'--append_crowd\r' --det_db $'det_db_motrv2.json\r' $'--use_checkpoint\r' --output_dir .
• tee -a output.log
  usage: Deformable DETR training and evaluation script [-h] [--lr LR]
  [--lr_backbone_names LR_BACKBONE_NAMES [LR_BACKBONE_NAMES ...]]
  [--lr_backbone LR_BACKBONE]
  [--lr_linear_proj_names LR_LINEAR_PROJ_NAMES [LR_LINEAR_PROJ_NAMES ...]]
  [--lr_linear_proj_mult LR_LINEAR_PROJ_MULT]
  [--batch_size BATCH_SIZE]
  [--weight_decay WEIGHT_DECAY]
  [--epochs EPOCHS]
  [--lr_drop LR_DROP]
  [--save_period SAVE_PERIOD]
  [--lr_drop_epochs LR_DROP_EPOCHS [LR_DROP_EPOCHS ...]]
  [--clip_max_norm CLIP_MAX_NORM]
  [--meta_arch META_ARCH]
  [--sgd]
  [--with_box_refine]
  [--two_stage]
  [--accurate_ratio]
  [--frozen_weights FROZEN_WEIGHTS]
  [--num_anchors NUM_ANCHORS]
  [--backbone BACKBONE]
  [--enable_fpn]
  [--dilation]
  [--position_embedding {sine,learned}]
  [--position_embedding_scale POSITION_EMBEDDING_SCALE]
  [--num_feature_levels NUM_FEATURE_LEVELS]
  [--enc_layers ENC_LAYERS]
  [--dec_layers DEC_LAYERS]
  [--dim_feedforward DIM_FEEDFORWARD]
  [--hidden_dim HIDDEN_DIM]
  [--dropout DROPOUT]
  [--nheads NHEADS]
  [--num_queries NUM_QUERIES]
  [--dec_n_points DEC_N_POINTS]
  [--enc_n_points ENC_N_POINTS]
  [--decoder_cross_self]
  [--sigmoid_attn]
  [--crop] [--cj]
  [--extra_track_attn]
  [--loss_normalizer]
  [--max_size MAX_SIZE]
  [--val_width VAL_WIDTH]
  [--filter_ignore]
  [--append_crowd]
  [--masks]
  [--no_aux_loss]
  [--mix_match]
  [--set_cost_class SET_COST_CLASS]
  [--set_cost_bbox SET_COST_BBOX]
  [--set_cost_giou SET_COST_GIOU]
  [--mask_loss_coef MASK_LOSS_COEF]
  [--dice_loss_coef DICE_LOSS_COEF]
  [--cls_loss_coef CLS_LOSS_COEF]
  [--bbox_loss_coef BBOX_LOSS_COEF]
  [--giou_loss_coef GIOU_LOSS_COEF]
  [--focal_alpha FOCAL_ALPHA]
  [--dataset_file DATASET_FILE]
  [--gt_file_train GT_FILE_TRAIN]
  [--gt_file_val GT_FILE_VAL]
  [--coco_path COCO_PATH]
  [--coco_panoptic_path COCO_PANOPTIC_PATH]
  [--remove_difficult]
  [--output_dir OUTPUT_DIR]
  [--device DEVICE]
  [--seed SEED]
  [--resume RESUME]
  [--start_epoch N]
  [--eval] [--vis]
  [--num_workers NUM_WORKERS]
  [--pretrained PRETRAINED]
  [--cache_mode]
  [--mot_path MOT_PATH]
  [--det_db DET_DB]
  [--input_video INPUT_VIDEO]
  [--data_txt_path_train DATA_TXT_PATH_TRAIN]
  [--data_txt_path_val DATA_TXT_PATH_VAL]
  [--img_path IMG_PATH]
  [--query_interaction_layer QUERY_INTERACTION_LAYER]
  [--sample_mode SAMPLE_MODE]
  [--sample_interval SAMPLE_INTERVAL]
  [--random_drop RANDOM_DROP]
  [--fp_ratio FP_RATIO]
  [--merger_dropout MERGER_DROPOUT]
  [--update_query_pos]
  [--sampler_steps [SAMPLER_STEPS [SAMPLER_STEPS ...]]]
  [--sampler_lengths [SAMPLER_LENGTHS [SAMPLER_LENGTHS ...]]]
  [--exp_name EXP_NAME]
  [--memory_bank_score_thresh MEMORY_BANK_SCORE_THRESH]
  [--memory_bank_len MEMORY_BANK_LEN]
  [--memory_bank_type MEMORY_BANK_TYPE]
  [--memory_bank_with_self_attn]
  [--use_checkpoint]
  [--query_denoise QUERY_DENOISE]
  --use_checkpointraining and evaluation script: error: unrecognized arguments: --with_box_refine
  usage: Deformable DETR training and evaluation script [-h] [--lr LR]
  [--lr_backbone_names LR_BACKBONE_NAMES [LR_BACKBONE_NAMES ...]]
  [--lr_backbone LR_BACKBONE]
  [--lr_linear_proj_names LR_LINEAR_PROJ_NAMES [LR_LINEAR_PROJ_NAMES ...]]
  [--lr_linear_proj_mult LR_LINEAR_PROJ_MULT]
  [--batch_size BATCH_SIZE]
  [--weight_decay WEIGHT_DECAY]
  [--epochs EPOCHS]
  [--lr_drop LR_DROP]
  [--save_period SAVE_PERIOD]
  [--lr_drop_epochs LR_DROP_EPOCHS [LR_DROP_EPOCHS ...]]
  [--clip_max_norm CLIP_MAX_NORM]
  [--meta_arch META_ARCH]
  [--sgd]
  [--with_box_refine]
  [--two_stage]
  [--accurate_ratio]
  [--frozen_weights FROZEN_WEIGHTS]
  [--num_anchors NUM_ANCHORS]
  [--backbone BACKBONE]
  [--enable_fpn]
  [--dilation]
  [--position_embedding {sine,learned}]
  [--position_embedding_scale POSITION_EMBEDDING_SCALE]
  [--num_feature_levels NUM_FEATURE_LEVELS]
  [--enc_layers ENC_LAYERS]
  [--dec_layers DEC_LAYERS]
  [--dim_feedforward DIM_FEEDFORWARD]
  [--hidden_dim HIDDEN_DIM]
  [--dropout DROPOUT]
  [--nheads NHEADS]
  [--num_queries NUM_QUERIES]
  [--dec_n_points DEC_N_POINTS]
  [--enc_n_points ENC_N_POINTS]
  [--decoder_cross_self]
  [--sigmoid_attn]
  [--crop] [--cj]
  [--extra_track_attn]
  [--loss_normalizer]
  [--max_size MAX_SIZE]
  [--val_width VAL_WIDTH]
  [--filter_ignore]
  [--append_crowd]
  [--masks]
  [--no_aux_loss]
  [--mix_match]
  [--set_cost_class SET_COST_CLASS]
  [--set_cost_bbox SET_COST_BBOX]
  [--set_cost_giou SET_COST_GIOU]
  [--mask_loss_coef MASK_LOSS_COEF]
  [--dice_loss_coef DICE_LOSS_COEF]
  [--cls_loss_coef CLS_LOSS_COEF]
  [--bbox_loss_coef BBOX_LOSS_COEF]
  [--giou_loss_coef GIOU_LOSS_COEF]
  [--focal_alpha FOCAL_ALPHA]
  [--dataset_file DATASET_FILE]
  [--gt_file_train GT_FILE_TRAIN]
  [--gt_file_val GT_FILE_VAL]
  [--coco_path COCO_PATH]
  [--coco_panoptic_path COCO_PANOPTIC_PATH]
  [--remove_difficult]
  [--output_dir OUTPUT_DIR]
  [--device DEVICE]
  [--seed SEED]
  [--resume RESUME]
  [--start_epoch N]
  [--eval] [--vis]
  [--num_workers NUM_WORKERS]
  [--pretrained PRETRAINED]
  [--cache_mode]
  [--mot_path MOT_PATH]
  [--det_db DET_DB]
  [--input_video INPUT_VIDEO]
  [--data_txt_path_train DATA_TXT_PATH_TRAIN]
  [--data_txt_path_val DATA_TXT_PATH_VAL]
  [--img_path IMG_PATH]
  [--query_interaction_layer QUERY_INTERACTION_LAYER]
  [--sample_mode SAMPLE_MODE]
  [--sample_interval SAMPLE_INTERVAL]
  [--random_drop RANDOM_DROP]
  [--fp_ratio FP_RATIO]
  [--merger_dropout MERGER_DROPOUT]
  [--update_query_pos]
  [--sampler_steps [SAMPLER_STEPS [SAMPLER_STEPS ...]]]
  [--sampler_lengths [SAMPLER_LENGTHS [SAMPLER_LENGTHS ...]]]
  [--exp_name EXP_NAME]
  [--memory_bank_score_thresh MEMORY_BANK_SCORE_THRESH]
  [--memory_bank_len MEMORY_BANK_LEN]
  [--memory_bank_type MEMORY_BANK_TYPE]
  [--memory_bank_with_self_attn]
  [--use_checkpoint]
  [--query_denoise QUERY_DENOISE]
  --use_checkpointraining and evaluation script: error: unrecognized arguments: --with_box_refine
  usage: Deformable DETR training and evaluation script [-h] [--lr LR]
  [--lr_backbone_names LR_BACKBONE_NAMES [LR_BACKBONE_NAMES ...]]
  [--lr_backbone LR_BACKBONE]
  [--lr_linear_proj_names LR_LINEAR_PROJ_NAMES [LR_LINEAR_PROJ_NAMES ...]]
  [--lr_linear_proj_mult LR_LINEAR_PROJ_MULT]
  [--batch_size BATCH_SIZE]
  [--weight_decay WEIGHT_DECAY]
  [--epochs EPOCHS]
  [--lr_drop LR_DROP]
  [--save_period SAVE_PERIOD]
  [--lr_drop_epochs LR_DROP_EPOCHS [LR_DROP_EPOCHS ...]]
  [--clip_max_norm CLIP_MAX_NORM]
  [--meta_arch META_ARCH]
  [--sgd]
  [--with_box_refine]
  [--two_stage]
  [--accurate_ratio]
  [--frozen_weights FROZEN_WEIGHTS]
  [--num_anchors NUM_ANCHORS]
  [--backbone BACKBONE]
  [--enable_fpn]
  [--dilation]
  [--position_embedding {sine,learned}]
  [--position_embedding_scale POSITION_EMBEDDING_SCALE]
  [--num_feature_levels NUM_FEATURE_LEVELS]
  [--enc_layers ENC_LAYERS]
  [--dec_layers DEC_LAYERS]
  [--dim_feedforward DIM_FEEDFORWARD]
  [--hidden_dim HIDDEN_DIM]
  [--dropout DROPOUT]
  [--nheads NHEADS]
  [--num_queries NUM_QUERIES]
  [--dec_n_points DEC_N_POINTS]
  [--enc_n_points ENC_N_POINTS]
  [--decoder_cross_self]
  [--sigmoid_attn]
  [--crop] [--cj]
  [--extra_track_attn]
  [--loss_normalizer]
  [--max_size MAX_SIZE]
  [--val_width VAL_WIDTH]
  [--filter_ignore]
  [--append_crowd]
  [--masks]
  [--no_aux_loss]
  [--mix_match]
  [--set_cost_class SET_COST_CLASS]
  [--set_cost_bbox SET_COST_BBOX]
  [--set_cost_giou SET_COST_GIOU]
  [--mask_loss_coef MASK_LOSS_COEF]
  [--dice_loss_coef DICE_LOSS_COEF]
  [--cls_loss_coef CLS_LOSS_COEF]
  [--bbox_loss_coef BBOX_LOSS_COEF]
  [--giou_loss_coef GIOU_LOSS_COEF]
  [--focal_alpha FOCAL_ALPHA]
  [--dataset_file DATASET_FILE]
  [--gt_file_train GT_FILE_TRAIN]
  [--gt_file_val GT_FILE_VAL]
  [--coco_path COCO_PATH]
  [--coco_panoptic_path COCO_PANOPTIC_PATH]
  [--remove_difficult]
  [--output_dir OUTPUT_DIR]
  [--device DEVICE]
  [--seed SEED]
  [--resume RESUME]
  [--start_epoch N]
  [--eval] [--vis]
  [--num_workers NUM_WORKERS]
  [--pretrained PRETRAINED]
  [--cache_mode]
  [--mot_path MOT_PATH]
  [--det_db DET_DB]
  [--input_video INPUT_VIDEO]
  [--data_txt_path_train DATA_TXT_PATH_TRAIN]
  [--data_txt_path_val DATA_TXT_PATH_VAL]
  [--img_path IMG_PATH]
  [--query_interaction_layer QUERY_INTERACTION_LAYER]
  [--sample_mode SAMPLE_MODE]
  [--sample_interval SAMPLE_INTERVAL]
  [--random_drop RANDOM_DROP]
  [--fp_ratio FP_RATIO]
  [--merger_dropout MERGER_DROPOUT]
  [--update_query_pos]
  [--sampler_steps [SAMPLER_STEPS [SAMPLER_STEPS ...]]]
  [--sampler_lengths [SAMPLER_LENGTHS [SAMPLER_LENGTHS ...]]]
  [--exp_name EXP_NAME]
  [--memory_bank_score_thresh MEMORY_BANK_SCORE_THRESH]
  [--memory_bank_len MEMORY_BANK_LEN]
  [--memory_bank_type MEMORY_BANK_TYPE]
  [--memory_bank_with_self_attn]
  [--use_checkpoint]
  [--query_denoise QUERY_DENOISE]
  --use_checkpointraining and evaluation script: error: unrecognized arguments: --with_box_refine
  usage: Deformable DETR training and evaluation script [-h] [--lr LR]
  [--lr_backbone_names LR_BACKBONE_NAMES [LR_BACKBONE_NAMES ...]]
  [--lr_backbone LR_BACKBONE]
  [--lr_linear_proj_names LR_LINEAR_PROJ_NAMES [LR_LINEAR_PROJ_NAMES ...]]
  [--lr_linear_proj_mult LR_LINEAR_PROJ_MULT]
  [--batch_size BATCH_SIZE]
  [--weight_decay WEIGHT_DECAY]
  [--epochs EPOCHS]
  [--lr_drop LR_DROP]
  [--save_period SAVE_PERIOD]
  [--lr_drop_epochs LR_DROP_EPOCHS [LR_DROP_EPOCHS ...]]
  [--clip_max_norm CLIP_MAX_NORM]
  [--meta_arch META_ARCH]
  [--sgd]
  [--with_box_refine]
  [--two_stage]
  [--accurate_ratio]
  [--frozen_weights FROZEN_WEIGHTS]
  [--num_anchors NUM_ANCHORS]
  [--backbone BACKBONE]
  [--enable_fpn]
  [--dilation]
  [--position_embedding {sine,learned}]
  [--position_embedding_scale POSITION_EMBEDDING_SCALE]
  [--num_feature_levels NUM_FEATURE_LEVELS]
  [--enc_layers ENC_LAYERS]
  [--dec_layers DEC_LAYERS]
  [--dim_feedforward DIM_FEEDFORWARD]
  [--hidden_dim HIDDEN_DIM]
  [--dropout DROPOUT]
  [--nheads NHEADS]
  [--num_queries NUM_QUERIES]
  [--dec_n_points DEC_N_POINTS]
  [--enc_n_points ENC_N_POINTS]
  [--decoder_cross_self]
  [--sigmoid_attn]
  [--crop] [--cj]
  [--extra_track_attn]
  [--loss_normalizer]
  [--max_size MAX_SIZE]
  [--val_width VAL_WIDTH]
  [--filter_ignore]
  [--append_crowd]
  [--masks]
  [--no_aux_loss]
  [--mix_match]
  [--set_cost_class SET_COST_CLASS]
  [--set_cost_bbox SET_COST_BBOX]
  [--set_cost_giou SET_COST_GIOU]
  [--mask_loss_coef MASK_LOSS_COEF]
  [--dice_loss_coef DICE_LOSS_COEF]
  [--cls_loss_coef CLS_LOSS_COEF]
  [--bbox_loss_coef BBOX_LOSS_COEF]
  [--giou_loss_coef GIOU_LOSS_COEF]
  [--focal_alpha FOCAL_ALPHA]
  [--dataset_file DATASET_FILE]
  [--gt_file_train GT_FILE_TRAIN]
  [--gt_file_val GT_FILE_VAL]
  [--coco_path COCO_PATH]
  [--coco_panoptic_path COCO_PANOPTIC_PATH]
  [--remove_difficult]
  [--output_dir OUTPUT_DIR]
  [--device DEVICE]
  [--seed SEED]
  [--resume RESUME]
  [--start_epoch N]
  [--eval] [--vis]
  [--num_workers NUM_WORKERS]
  [--pretrained PRETRAINED]
  [--cache_mode]
  [--mot_path MOT_PATH]
  [--det_db DET_DB]
  [--input_video INPUT_VIDEO]
  [--data_txt_path_train DATA_TXT_PATH_TRAIN]
  [--data_txt_path_val DATA_TXT_PATH_VAL]
  [--img_path IMG_PATH]
  [--query_interaction_layer QUERY_INTERACTION_LAYER]
  [--sample_mode SAMPLE_MODE]
  [--sample_interval SAMPLE_INTERVAL]
  [--random_drop RANDOM_DROP]
  [--fp_ratio FP_RATIO]
  [--merger_dropout MERGER_DROPOUT]
  [--update_query_pos]
  [--sampler_steps [SAMPLER_STEPS [SAMPLER_STEPS ...]]]
  [--sampler_lengths [SAMPLER_LENGTHS [SAMPLER_LENGTHS ...]]]
  [--exp_name EXP_NAME]
  [--memory_bank_score_thresh MEMORY_BANK_SCORE_THRESH]
  [--memory_bank_len MEMORY_BANK_LEN]
  [--memory_bank_type MEMORY_BANK_TYPE]
  [--memory_bank_with_self_attn]
  [--use_checkpoint]
  [--query_denoise QUERY_DENOISE]
  usage: Deformable DETR training and evaluation script [-h] [--lr LR]
  [--lr_backbone_names LR_BACKBONE_NAMES [LR_BACKBONE_NAMES ...]]
  [--lr_backbone LR_BACKBONE]
  [--lr_linear_proj_names LR_LINEAR_PROJ_NAMES [LR_LINEAR_PROJ_NAMES ...]]
  [--lr_linear_proj_mult LR_LINEAR_PROJ_MULT]
  [--batch_size BATCH_SIZE]
  [--weight_decay WEIGHT_DECAY]
  [--epochs EPOCHS]
  [--lr_drop LR_DROP]
  [--save_period SAVE_PERIOD]
  [--lr_drop_epochs LR_DROP_EPOCHS [LR_DROP_EPOCHS ...]]
  [--clip_max_norm CLIP_MAX_NORM]
  [--meta_arch META_ARCH]
  [--sgd]
  [--with_box_refine]
  [--two_stage]
  [--accurate_ratio]
  [--frozen_weights FROZEN_WEIGHTS]
  [--num_anchors NUM_ANCHORS]
  [--backbone BACKBONE]
  [--enable_fpn]
  [--dilation]
  [--position_embedding {sine,learned}]
  [--position_embedding_scale POSITION_EMBEDDING_SCALE]
  [--num_feature_levels NUM_FEATURE_LEVELS]
  [--enc_layers ENC_LAYERS]
  [--dec_layers DEC_LAYERS]
  [--dim_feedforward DIM_FEEDFORWARD]
  [--hidden_dim HIDDEN_DIM]
  [--dropout DROPOUT]
  [--nheads NHEADS]
  [--num_queries NUM_QUERIES]
  [--dec_n_points DEC_N_POINTS]
  [--enc_n_points ENC_N_POINTS]
  [--decoder_cross_self]
  [--sigmoid_attn]
  [--crop] [--cj]
  [--extra_track_attn]
  [--loss_normalizer]
  [--max_size MAX_SIZE]
  [--val_width VAL_WIDTH]
  [--filter_ignore]
  [--append_crowd]
  [--masks]
  [--no_aux_loss]
  [--mix_match]
  [--set_cost_class SET_COST_CLASS]
  [--set_cost_bbox SET_COST_BBOX]
  [--set_cost_giou SET_COST_GIOU]
  [--mask_loss_coef MASK_LOSS_COEF]
  [--dice_loss_coef DICE_LOSS_COEF]
  [--cls_loss_coef CLS_LOSS_COEF]
  [--bbox_loss_coef BBOX_LOSS_COEF]
  [--giou_loss_coef GIOU_LOSS_COEF]
  [--focal_alpha FOCAL_ALPHA]
  [--dataset_file DATASET_FILE]
  [--gt_file_train GT_FILE_TRAIN]
  [--gt_file_val GT_FILE_VAL]
  [--coco_path COCO_PATH]
  [--coco_panoptic_path COCO_PANOPTIC_PATH]
  [--remove_difficult]
  [--output_dir OUTPUT_DIR]
  [--device DEVICE]
  [--seed SEED]
  [--resume RESUME]
  [--start_epoch N]
  [--eval] [--vis]
  [--num_workers NUM_WORKERS]
  [--pretrained PRETRAINED]
  [--cache_mode]
  [--mot_path MOT_PATH]
  [--det_db DET_DB]
  [--input_video INPUT_VIDEO]
  [--data_txt_path_train DATA_TXT_PATH_TRAIN]
  [--data_txt_path_val DATA_TXT_PATH_VAL]
  [--img_path IMG_PATH]
  [--query_interaction_layer QUERY_INTERACTION_LAYER]
  [--sample_mode SAMPLE_MODE]
  [--sample_interval SAMPLE_INTERVAL]
  [--random_drop RANDOM_DROP]
  [--fp_ratio FP_RATIO]
  [--merger_dropout MERGER_DROPOUT]
  [--update_query_pos]
  [--sampler_steps [SAMPLER_STEPS [SAMPLER_STEPS ...]]]
  [--sampler_lengths [SAMPLER_LENGTHS [SAMPLER_LENGTHS ...]]]
  [--exp_name EXP_NAME]
  [--memory_bank_score_thresh MEMORY_BANK_SCORE_THRESH]
  [--memory_bank_len MEMORY_BANK_LEN]
  [--memory_bank_type MEMORY_BANK_TYPE]
  [--memory_bank_with_self_attn]
  [--use_checkpoint]
  [--query_denoise QUERY_DENOISE]
  usage: Deformable DETR training and evaluation script [-h] [--lr LR]
  [--lr_backbone_names LR_BACKBONE_NAMES [LR_BACKBONE_NAMES ...]]
  [--lr_backbone LR_BACKBONE]
  [--lr_linear_proj_names LR_LINEAR_PROJ_NAMES [LR_LINEAR_PROJ_NAMES ...]]
  [--lr_linear_proj_mult LR_LINEAR_PROJ_MULT]
  [--batch_size BATCH_SIZE]
  [--weight_decay WEIGHT_DECAY]
  [--epochs EPOCHS]
  [--lr_drop LR_DROP]
  [--save_period SAVE_PERIOD]
  [--lr_drop_epochs LR_DROP_EPOCHS [LR_DROP_EPOCHS ...]]
  [--clip_max_norm CLIP_MAX_NORM]
  [--meta_arch META_ARCH]
  [--sgd]
  [--with_box_refine]
  [--two_stage]
  [--accurate_ratio]
  [--frozen_weights FROZEN_WEIGHTS]
  [--num_anchors NUM_ANCHORS]
  [--backbone BACKBONE]
  [--enable_fpn]
  [--dilation]
  [--position_embedding {sine,learned}]
  [--position_embedding_scale POSITION_EMBEDDING_SCALE]
  [--num_feature_levels NUM_FEATURE_LEVELS]
  [--enc_layers ENC_LAYERS]
  [--dec_layers DEC_LAYERS]
  [--dim_feedforward DIM_FEEDFORWARD]
  [--hidden_dim HIDDEN_DIM]
  [--dropout DROPOUT]
  [--nheads NHEADS]
  [--num_queries NUM_QUERIES]
  [--dec_n_points DEC_N_POINTS]
  [--enc_n_points ENC_N_POINTS]
  [--decoder_cross_self]
  [--sigmoid_attn]
  [--crop] [--cj]
  [--extra_track_attn]
  [--loss_normalizer]
  [--max_size MAX_SIZE]
  [--val_width VAL_WIDTH]
  [--filter_ignore]
  [--append_crowd]
  [--masks]
  [--no_aux_loss]
  [--mix_match]
  [--set_cost_class SET_COST_CLASS]
  [--set_cost_bbox SET_COST_BBOX]
  [--set_cost_giou SET_COST_GIOU]
  [--mask_loss_coef MASK_LOSS_COEF]
  [--dice_loss_coef DICE_LOSS_COEF]
  [--cls_loss_coef CLS_LOSS_COEF]
  [--bbox_loss_coef BBOX_LOSS_COEF]
  [--giou_loss_coef GIOU_LOSS_COEF]
  [--focal_alpha FOCAL_ALPHA]
  [--dataset_file DATASET_FILE]
  [--gt_file_train GT_FILE_TRAIN]
  [--gt_file_val GT_FILE_VAL]
  [--coco_path COCO_PATH]
  [--coco_panoptic_path COCO_PANOPTIC_PATH]
  [--remove_difficult]
  [--output_dir OUTPUT_DIR]
  [--device DEVICE]
  [--seed SEED]
  [--resume RESUME]
  [--start_epoch N]
  [--eval] [--vis]
  [--num_workers NUM_WORKERS]
  [--pretrained PRETRAINED]
  [--cache_mode]
  [--mot_path MOT_PATH]
  [--det_db DET_DB]
  [--input_video INPUT_VIDEO]
  [--data_txt_path_train DATA_TXT_PATH_TRAIN]
  [--data_txt_path_val DATA_TXT_PATH_VAL]
  [--img_path IMG_PATH]
  [--query_interaction_layer QUERY_INTERACTION_LAYER]
  [--sample_mode SAMPLE_MODE]
  [--sample_interval SAMPLE_INTERVAL]
  [--random_drop RANDOM_DROP]
  [--fp_ratio FP_RATIO]
  [--merger_dropout MERGER_DROPOUT]
  [--update_query_pos]
  [--sampler_steps [SAMPLER_STEPS [SAMPLER_STEPS ...]]]
  [--sampler_lengths [SAMPLER_LENGTHS [SAMPLER_LENGTHS ...]]]
  [--exp_name EXP_NAME]
  [--memory_bank_score_thresh MEMORY_BANK_SCORE_THRESH]
  [--memory_bank_len MEMORY_BANK_LEN]
  [--memory_bank_type MEMORY_BANK_TYPE]
  [--memory_bank_with_self_attn]
  [--use_checkpoint]
  [--query_denoise QUERY_DENOISE]
  --use_checkpointraining and evaluation script: error: unrecognized arguments: --with_box_refine
  --use_checkpointraining and evaluation script: error: unrecognized arguments: --with_box_refine
  --use_checkpointraining and evaluation script: error: unrecognized arguments: --with_box_refine
  Traceback (most recent call last):
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
  "main", mod_spec)
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/runpy.py", line 85, in _run_code
  exec(code, run_globals)
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
  main()
  File "/data1/dcy/anaconda3/envs/motrv2/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
  cmd=cmd)
  subprocess.CalledProcessError: Command '['/data1/dcy/anaconda3/envs/motrv2/bin/python', '-u', 'main.py', '--meta_arch', 'motr\r', '--dataset_file', 'e2e_dance\r', '--epoch', '5\r', '--with_box_refine\r', '--lr_drop', '4\r', '--lr', '2e-4\r', '--lr_backbone', '2e-5\r', '--pretrained', 'mot/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth\r', '--batch_size', '128\r', '--sample_mode', 'random_interval\r', '--sample_interval', '10\r', '--sampler_lengths', '5\r', '--merger_dropout', '0\r', '--dropout', '0\r', '--random_drop', '0.1\r', '--fp_ratio', '0.3\r', '--query_interaction_layer', 'QIMv2\r', '--query_denoise', '0.05\r', '--num_queries', '10\r', '--append_crowd\r', '--det_db', 'det_db_motrv2.json\r', '--use_checkpoint\r', '--output_dir', '.']' died with <Signals.SIGSEGV: 11>.

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

• cleanup
• echo 'Packing source code'
  Packing source code
• rmpyc
  ++ find -name pycache
• rm -rf ./pycache ./datasets/pycache ./models/pycache ./models/ops/functions/pycache ./models/ops/modules/pycache ./models/structures/pycache ./util/pycache
  ++ find -name '*.pyc'
• rm -rf
• echo ' ...Done'
  ...Done

Training log

Hi,

Thanks for sharing your great work! I am reproducing the training process of motrv2, could you please provide the training log in your server to help me debug the training code?

Train on BrackishMOT

I want to train on BrackishMOT dataset, but there have been a wrong. Please help me how to train on BrackishMOT? Looking forward to your reply.

BDD100K weights

Hi, it seems to me that in the paper you don't refer to what weights of YOLOX you used for BDD100K, could you give me more details about that?

Multi Class Support Plan

Hi Author,

Thank you very much for your amazing work, I want to know when to support multi class tracking？

det_ db_ Is motrv2.json

Hello, thanks for your great work! Excuse me,the result of det_ db_ Is motrv2.json obtained by using YOLOX? In addition, is there no code for this part in this project? I'd appreciate it if you can show more details.

Question about multi-class

Can it support multi-class training?

Data Preprocessing Issue

Thank you for sharing a great repository.

I suspect that there may be a problem with preprocessing the dataset. When creating det_db_0.json, which extracts the bounding boxes in advance through the BytetrackInference repository, I found that the json does not contain all image information due to the sampler.

If I am approaching this incorrectly, please let me know.

score threshold and update_score_threshold

I found the score threshold and update_score_threshold is set to 0.5 in submit_dance.py, then the detections under 0.5 confidence will directly be removed.
So why it set to be 0.5?
Is there any hyper-parameter during training influence them?

About Backbone

你好，我想问一下，如何在不使用torchvision.models里面写好的网络框架，而是自定义一个CNN作为这个模型的backbone？我在自己编写CNN的时候需要注意些什么？

两个不存在的文件？

执行训练脚本时有这么两句：
cp -r models datasets util main.py engine.py eval_detr.py seqmap submit_dance.py $1 $OUTPUT_DIR
其中，eval_detr.py和seqmap不存在，请问我要怎么解决？

det_db_motrv2.json

你好，我想问一下你这个项目里面要用的det_db_motrv2.json是怎么生成的？我在这里期待你的回复，谢谢。

num_classes设置

加载pretrain模型r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth时会出现报错：
No param refine_embed.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset.

请问在哪里更改num_classes设置？

Questions about Multi-gpu training

Why I try to train it with 4 V100, it occured CUDA OUT OF MEMORY during epoch 0. I set nproc_per_node=4 in train.sh, is there anything wrong?

why batch_size only support 1 per GPU

Hi, Deformable Detr origin git can support different batch_size.

Why MOTR V2 only support batch_size ==1 ?

HOTA metric

Hi, I am trying to use a MOTRv2 on a custom dataset and am currently trying to create metrics to assess its performance. I see in your code where you calculate MOTA but I don't see where HOTA is calculated. I am wondering how you calculate HOTA as this seems to be the standard.

BDD100K release

Hi,
I am an Italian student at the University of Salerno who is doing a thesis in autonomous vehicle driving concerning transformer-based tracking techniques. While studying the state-of-the-art I came across your paper. I would be very grateful if it would be possible to have your model trained on BDD100K. How soon will it be published?
Thank you in advance.

nondeterministic of MSDeformAttn backward

MSDeformAttn backward is a bit nondeterministic with floating point types. When trained multiple times, the final tracking performance varies greatly. Anyone Know Why, and How to improve

Training，cuda version problems

It might be due to my graphics card not supporting CUDAToolkit 10.2, resulting in the "RuntimeError: Unrecognized tensor type ID: Autocast" error. How can I modify it to prevent the occurrence of this issue?

Question about 2D conf for query generation

Thanks for sharing the great work!

Could you also share the idea or the reason why you choose 2D confs to generate the query embedding?

Looking forward to your reply.

Where is the code about "Track Query Alignment on MOT17"?

Hi, Author,

  I want to reproduce the paper table 9. Effect of track query alignment on MOT17 validation set.

  But I can not found the detail code in your git repo.

Question about track and object queries

Hi, really nice work!

I am curious how the transformer is able to differentiate between the track and object queries. I undestand that you use TAN to update track queries but this does not process the object queries. How is the network able to figure out that a track query needs to track an object and has priority over the object queries? Does the network learn to produce decoder output embeddings that code for "track queries".

My second question is why you don't use the attention_mask to prevent infromation leakage from the general predictions to the noised track queries. I assume if the noised track queries see the rest of track queries, it will compromise it's ability to track objects due to the NMS that is performed by the self attention module in the decoder.

MOTRv2/models/motr.py

Line 530 in be49b73

attn_mask[:n_dt, n_dt:] = True

Thanks,
Owen

user_checkpoint和CheckpointFunction的作用是什么？

如题，调试了一遍代码，没看懂他们的作用，为什么不计算梯度，又为什么重写backward，求解。

Do you tune YOLOX provided in ByteTrack with other tracker?

I previously thought that the yolox weight is from training yolox with ByteTrack tracker on DanceTrack training set, which according to the results in this link you provided in the paper: [https://github.com/DanceTrack/DanceTrack] as you provided in the paper ( 47.7 HOTA)

But it seems that you use OC-SORT tracker rather than ByteTrack tracker? Do you mean you use the model that have 55.1 HOTA results on DanceTrack test set?

Have you tried other yolox weights such as the yolox pretrained with ByterTrack tracker(47.7 HOTA)? Will it influence the overall performance a lot?

Looking forward to your reply, thanks!

Originally posted by @wenyuqing in #34 (comment)

MOT17

Hi, could you please share the model weights trained on MOT17?

Question about "make sure you set the correct --num_classes for your own dataset"

Hi, I try to run the training code with command ./tools/train.sh configs/motrv2.args and there are some warnings when loading the pretrained detr weights as follows:
loaded /home/code/mot/MOTRv2/detr_pretrain/r50_deformable_detr_plus_iterative_bbox_refinement-checkpoint.pth Drop parameter transformer.reference_points.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. Drop parameter transformer.reference_points.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. Skip loading parameter class_embed.0.weight, required shapetorch.Size([1, 256]), loaded shapetorch.Size([91, 256]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.0.weight shape=torch.Size([91, 256]) Skip loading parameter class_embed.0.bias, required shapetorch.Size([1]), loaded shapetorch.Size([91]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.0.bias shape=torch.Size([91]) Skip loading parameter class_embed.1.weight, required shapetorch.Size([1, 256]), loaded shapetorch.Size([91, 256]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.1.weight shape=torch.Size([91, 256]) Skip loading parameter class_embed.1.bias, required shapetorch.Size([1]), loaded shapetorch.Size([91]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.1.bias shape=torch.Size([91]) Skip loading parameter class_embed.2.weight, required shapetorch.Size([1, 256]), loaded shapetorch.Size([91, 256]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.2.weight shape=torch.Size([91, 256]) Skip loading parameter class_embed.2.bias, required shapetorch.Size([1]), loaded shapetorch.Size([91]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.2.bias shape=torch.Size([91]) Skip loading parameter class_embed.3.weight, required shapetorch.Size([1, 256]), loaded shapetorch.Size([91, 256]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.3.weight shape=torch.Size([91, 256]) Skip loading parameter class_embed.3.bias, required shapetorch.Size([1]), loaded shapetorch.Size([91]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.3.bias shape=torch.Size([91]) Skip loading parameter class_embed.4.weight, required shapetorch.Size([1, 256]), loaded shapetorch.Size([91, 256]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.4.weight shape=torch.Size([91, 256]) Skip loading parameter class_embed.4.bias, required shapetorch.Size([1]), loaded shapetorch.Size([91]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.4.bias shape=torch.Size([91]) Skip loading parameter class_embed.5.weight, required shapetorch.Size([1, 256]), loaded shapetorch.Size([91, 256]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.5.weight shape=torch.Size([91, 256]) Skip loading parameter class_embed.5.bias, required shapetorch.Size([1]), loaded shapetorch.Size([91]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. load class_embed: class_embed.5.bias shape=torch.Size([91]) Skip loading parameter query_embed.weight, required shapetorch.Size([10, 256]), loaded shapetorch.Size([300, 512]). If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.self_attn.in_proj_weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.self_attn.in_proj_bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.self_attn.out_proj.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.self_attn.out_proj.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear1.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear1.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear2.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear2.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear_feat1.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear_feat1.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear_feat2.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.linear_feat2.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.norm_feat.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.norm_feat.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.norm1.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.norm1.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.norm2.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param track_embed.norm2.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param position.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param yolox_embed.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. No param refine_embed.weight.If you see this, your model does not fully load the pre-trained weight. Please make sure you set the correct --num_classes for your own dataset. Start training

Does this matter? Should I change any configurations or code?

Thanks!

Could you provide the GT proposal file?

Hi, thanks for your work. I'm really interested in the effect of the detection proposals. Have you tried different kind of proposals? Will the performance be better when the proposal is more accurate? What if you use the groundtruth proposals? If it's convinient could you please provide the GT proposal?
Looking forward to your quick reply! Thanks.

Experiment on more datasets

Thanks for your good jobs. Would you provide more results on other datasets such as MOT17 or MOT20. And how can I do if I want to train on private datasets.

How to apply the ensemble of multiple models?

Hi, thanks for your great work with simple idea but excellent performance!

I want to ask that how do you use an ensemble of 4 models?
You mean, 4 tracking models? or 4 detection models?

I'd appreciate it if you can show more details.

这个好落地吗？

Question about tensorrt model

Can MOTRv2 model be converted to tensorrt?
Can you share the pth trained with MOT20?

Questions about label losses for disappear tracks

Dear authors,

I have a problem about labels loss in MOTR-v2.

In https://github.com/megvii-research/MOTRv2/blob/main/models/motr.py#L147, the default gt value of labels is set to 1, for J == -1(disappear track), the labels value will keep as 1 and used for following training. Should the default value be self.num_classes?

Thanks!

Speed compared with bytetrack or oc-sort？

Cannot set batch_size>1?

When I set batch_size=2 to train my model, it told me "AttributeError: 'list' object has no attribute 'requires_grad'."So how to solve it to enlarge batch_size?I want to save my time on training model.

Question about pretrained weight

Can I change the pretrained weight by myself? I wanna train my own model with pretrained model using YOLOX result called best_ckpt.pth.