nvlabs / pamtri

PAMTRI: Pose-Aware Multi-Task Learning for Vehicle Re-Identification (ICCV 2019) - Official PyTorch Implementation

PAMTRI: Pose-Aware Multi-Task Learning for Vehicle Re-Identification

This repo contains the official PyTorch implementation of PAMTRI: Pose-Aware Multi-Task Learning for Vehicle Re-Identification Using Highly Randomized Synthetic Data, ICCV 2019.

[Paper] [Poster]

Introduction

We address the problem of vehicle re-identification using multi-task learning and embeded pose representations. Since manually labeling images with detailed pose and attribute information is prohibitive, we train the network with a combination of real and randomized synthetic data.

The proposed framework consists of two convolutional neural networks (CNNs), which are shown in the figure below. Top: The pose estimation network is an extension of high-resolution network (HRNet) for predicting keypoint coordinates (with confidence/visibility) and generating heatmaps/segments. Bottom: The multi-task network uses the embedded pose information from HRNet for joint vehicle re-identification and attribute classification.

Getting Started

Environment

The code was developed and tested with Python 3.6 on Ubuntu 16.04, using a NVIDIA GeForce RTX 2080 Ti GPU card. Other platforms or GPU card(s) may work but are not fully tested.

Code Structure

Please refer to the README.md in each of the following directories for detailed instructions.

• PoseEstNet directory: The modified version of HRNet for vehicle pose estimation. The code for training and testing, keypoint labels, and pre-trained models are provided.

• MultiTaskNet directory: The multi-task network for joint vehicle re-identification and attribute classification using embedded pose representations. The code for training and testing, attribute labels, predicted keypoints, and pre-trained models are provided.

References

Please cite these papers if you use this code in your research:

@inproceedings{Tang19PAMTRI,
  author = {Zheng Tang and Milind Naphade and Stan Birchfield and Jonathan Tremblay and William Hodge and Ratnesh Kumar and Shuo Wang and Xiaodong Yang},
  title = { {PAMTRI}: {P}ose-aware multi-task learning for vehicle re-identification using highly randomized synthetic data},
  booktitle = {Proc. of the International Conference on Computer Vision (ICCV)},
  pages = {211-–220},
  address = {Seoul, Korea},
  month = oct,
  year = 2019
}

@inproceedings{Tang19CityFlow,
  author = {Zheng Tang and Milind Naphade and Ming-Yu Liu and Xiaodong Yang and Stan Birchfield and Shuo Wang and Ratnesh Kumar and David Anastasiu and Jenq-Neng Hwang},
  title = {City{F}low: {A} city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification},
  booktitle = {Proc. of the Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages = {8797–-8806},
  address = {Long Beach, CA, USA},
  month = jun,
  year = 2019
}

License

Code in the repository, unless otherwise specified, is licensed under the NVIDIA Source Code License.

Contact

For any questions please contact Zheng (Thomas) Tang.

pamtri's Issues

Hello! Could you please provide the synthetic data used in the paper?

Hello! Could you please provide the synthetic data used in the paper?

Hello, I don’t seem to see the operation of multiplying the mask with the feature map to get the local features. Can you tell me where this operation is?

Is there a way to change the size of the mask image, can the mask size not be 64*64? Is it possible to not output the mask map during testing? And concatenate them, because this can improve the performance of the network, if so,I hope to tell me the location of the output image in the code during the test, thank you very much!

CityFlow Keypoint Annotation

Thanks for your contribution！Will the annotation files for CityFlow Keypoints also be released?

Thank you very much. I tested the veri data set in your way, but the result of the test is only the marked part of the test data. If I want to test the veri data set or other data sets, such as the part of the data that is not marked, I want How to set in the yaml file (hope to use veri data as an example)

Hello, I would like to ask whether the keypoints are marked by yourself or by the code

Hello, when I reproduce your open source code, I use the test command provided in the readme to run test.py for testing. Why does the following error occur?

Hello, when I reproduce your open source code, I use the test command provided in the readme to run test.py for testing. Why does the following error occur?
The data set uses the veri data set provided in your link. The training can be trained, but the following error is reported as soon as the test arrives. Can you tell me the reason?
### 1：
=> loading model from models/veri/pose_hrnet/w32_256x256_adam_lr1e-3/model_best.pth
=> load 1060 samples
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
Traceback (most recent call last):
File "tools/test.py", line 148, in
main()
File "tools/test.py", line 140, in main
final_output_dir, tb_log_dir)
File "/home/wangsiling/PAMTRI/PoseEstNet/tools/../lib/core/function.py", line 121, in validate
for i, (input, target, target_weight, meta) in enumerate(val_loader):
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in next
return self._process_next_batch(batch)
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
raise batch.exc_type(batch.exc_msg)
cv2.error: Traceback (most recent call last):
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/wangsiling/PAMTRI/PoseEstNet/tools/../lib/dataset/JointsDataset.py", line 86, in getitem
data_numpy = cv2.cvtColor(data_numpy, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(3.4.1) /io/opencv/modules/imgproc/src/color.cpp:11115: error: (-215) scn == 3 || scn == 4 in function cvtColor

### 2：
=> load 9068 samples
=> load 1060 samples
Epoch: [0][0/142] Time 25.574s (25.574s) Speed 2.5 samples/s Data 17.410s (17.410s) Loss 0.00103 (0.00103) Accuracy 0.025 (0.025)
Epoch: [0][100/142] Time 0.749s (1.010s) Speed 85.5 samples/s Data 0.001s (0.196s) Loss 0.00033 (0.00058) Accuracy 0.899 (0.690)
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
Traceback (most recent call last):
File "tools/train.py", line 223, in
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
main()
File "tools/train.py", line 193, in main
OpenCV(3.4.1) Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /io/opencv/modules/imgproc/src/color.cpp, line 11115
final_output_dir, tb_log_dir, writer_dict
File "/home/wangsiling/PAMTRI/PoseEstNet/tools/../lib/core/function.py", line 121, in validate
for i, (input, target, target_weight, meta) in enumerate(val_loader):
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in next
return self._process_next_batch(batch)
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
cv2.error: Traceback (most recent call last):
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/wangsiling/anaconda3/envs/HRNet/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/wangsiling/PAMTRI/PoseEstNet/tools/../lib/dataset/JointsDataset.py", line 86, in getitem
data_numpy = cv2.cvtColor(data_numpy, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(3.4.1) /io/opencv/modules/imgproc/src/color.cpp:11115: error: (-215) scn == 3 || scn == 4 in function cvtColor

Just Infer data without annotations

Hey you,

nice implementation! Is there a way to infer new images without having the annotations?

Thanks in advance!
Timo

In PoseEstNet, how to change the test data set through settings during testing

Thank you very much. I carefully watched the function that changes the size of the mask image and found that there are not only the size of the mask image, but also the sizes of the 13 segments. When the size of the mask image is changed, the segments also need to be changed size? Could you please tell me which sentence to change the size of the mask in the function you pointed out to me? Many thanks

Hello, I want to ask, how did you get such accurate initial keypoints values through the lua network

Special thanks, but why would an error be reported when the size of the mask is changed to 24*24 through the code of https://github.com/NVlabs/PAMTRI/blob/master/PoseEstNet/lib/core/function.py#L397 , Tell me RuntimeError: The size of tensor a (4096) must match the size of tensor b (576) at non-singleton dimension 1