jarr0d / vitaa

ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language

ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language

We provide the code for reproducing experiment results of ViTAA

• ECCV2020 conference paper: pdf.
• If this work is helpful for your research, please cite ViTAA

@misc{wang2020vitaa,
    title={ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language},
    author={Zhe Wang and Zhiyuan Fang and Jun Wang and Yezhou Yang},
    year={2020},
    eprint={2005.07327},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

Benchmark

CUHK-PEDES

Method	Features	R@1	R@5	R@10
GNA-RNN	global	19.05	-	53.64
CMCE	global	25.94	-	60.48
PWM-ATH	global	27.14	49.45	61.02
Dual Path	global	44.40	66.26	75.07
CMPM+CMPC	global	49.37	-	79.27
MIA	global+region	53.10	75.00	82.90
GALM	global+keypoint	54.12	75.45	82.97
ViTAA	global+attribute	55.97	75.84	83.52

Data preparation

1. Download CUHK-PEDES dataset and save it anywhere you like (e.g. ~/datasets/cuhkpedes/).
2. Download text_attribute_graph (GoogleDrive / BaiduYun(code: vbss)) which are the text phrases parsed from the sentences, and save it in (e.g. ~/datasets/cuhkpedes/).
3. Use the provided Human Parsing Network to generate the attribute segmentations, and save it in (e.g. ~/datasets/cuhkpedes/).
4. Run the script in tools/cuhkpedes/convert_to_json to generate the json files as annotations.

python tools/cuhkpedes/convert_to_json.py --datadir ~/datasets/cuhkpedes/ --outdir datasets/cuhkpedes/annotations

Your datasets directory should look like this:

ViTAA
-- configs
-- tools
-- vitaa
-- datasets
   |-- cuhkpedes
   |   |-- annotations
   |   |   |-- test.json
   |   |   |-- train.json
   |   |   |-- val.json
   |   |-- imgs
   |   |   |-- cam_a
   |   |   |-- cam_b
   |   |   |--  ...
   |   |-- segs
   |   |   |-- cam_a
   |   |   |-- cam_b
   |   |   |--  ...

Training

# single-gpu training
python tools/train_net.py --config-file configs/cuhkpedes/bilstm_r50_seg.yaml

# multi-gpu training
We provide the code for distributed training but they haven't been tested

Note: We train ViTAA with batch_size=64 on one Tesla V100 GPU. If your GPU doesn't support such batch size, please follow the Linear Scaling Rule to adjust the configuration.

Testing

# single-gpu testing
python tools/test.py --config-file configs/cuhkpedes/bilstm_r50_seg.yaml --checkpoint-file output/cuhkpedes/...

Human Parsing Network

We separately provide the code of our Human Parsing Network because we think it might be a useful tool for the community.

Acknowledgement

Our codes is based on maskrcnn-benchmark, great thanks to their work.

vitaa's Issues

Pretrained weights

@Jarr0d hi thanks for opesourcing the code based , can you please share ur pretrained weights on google drive or one drive

Thanks in adavance

How does HRNet implement ot generate attributes?

Hello, thanks for your great work! I just want to know the details about HRNet implementation, beacause I can't find anything to do with HRNet in your codebase.

Looking forward your reply, thank your very lot.

Is this a code bug?

In the function "local_align_loss" of the code "vitaa/models/embeddings/seg_head/loss.py", should this line "forward_k_idx = rank1[i, :topK]" be modified as "forward_k_idx = rank1[j, :topK]"?

how to generate `text_attribute_graph`

Thanks for sharing the code, I wonder how to generate text_attribute_graph? You didn't mention much about this in your paper. Could you please share the method? Because I want to transfer your method to other datasets for experiment, but I don't know how to generate this file. Thanks again.

请问CUHK-PEDES数据集怎么下载的？发邮件也没用呀？

请问CUHK-PEDES数据集怎么下载的？发邮件也没用呀？

CUHK-PEDES dataset

I tried running the code at https://github.com/Jarr0d/Human-Parsing-Network to create attribute segmentation on the CUHK-PEDES dataset through the Human Parsing Network, but it could not be done because the format of the dataset was different.
Has anyone solved this problem?

Out of memory

When I run train_net, error occurs like:
return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 7.79 GiB total capacity; 6.48 GiB already allocated; 11.12 MiB free; 6.52 GiB reserved in total by PyTorch)
I was wondering whether the error is the one that my GPU is not supported by batch_size=64 as your mentioned. My GPU is 2080 with 16G memory. How can I fix it, is any configuration instruction I can use?
Thanx for your help.

cannot import files

Hi, thanks for your work!

I found these lines maybe redundant, which leads to an error:

ViTAA/vitaa/data/datasets/__init__.py

Lines 4 to 5 in 0bd7638

	from .market1501 import Market1501Dataset
	from .dukemtmc import DukeMTMCDataset

What' more, it would be great if you can provide a requirements.txt. It took me a little while to chose the version of PyTorch and skimage.

File download link failed

Hello, the download link of the file "text_attribute_graph" is invalid. Is there any new link? Or please send a copy to my email: [email protected]. Thank you very much.

how to conduct multi-gpu training

When I entered this command called “python -m torch.distributed.launch --nproc_per_node=4 tools/train_net.py --config-file configs/cuhkpedes/bilstm_r50_seg.yaml ”, The following error occurred:
subprocess.CalledProcessError: Command '['/home/whh/anaconda3/envs/hpn/bin/python', '-u', 'tools/train_net.py', '--local_rank=0', '--config-file', 'configs/cuhkpedes/bilstm_r50_seg.yaml']' returned non-zero exit status 1.

This problem has not been solved yet. What command should I enter to conduct multi-gpu training?

how to deal it？

../aten/src/ATen/native/cuda/Loss.cu:271: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [0,0,0] Assertion t >= 0 && t < n_classes failed.

Bug of to_bgr255?

ViTAA/vitaa/data/transforms/transforms.py

Lines 94 to 105 in 0bd7638

	class Normalize(object):
	def __init__(self, mean, std, to_bgr255=True):
	self.mean = mean
	self.std = std
	self.to_bgr255 = to_bgr255
	def __call__(self, image, seg=None):
	if self.to_bgr255:
	image = image[[2, 1, 0]] * 255
	image = F.normalize(image, mean=self.mean, std=self.std)
	seg = 255 * seg[0] if seg is not None else None
	return image, seg

I guess to_bgr255 is used for Caffe2. This should be set to False.

ImportError: cannot import name 'datasets'

when I run 'python tools/train_net.py --config-file configs/cuhkpedes/bilstm_r50_seg.yaml', The following errors occurred:
Traceback (most recent call last):
File "tools/train_net.py", line 23, in
from vitaa.data import make_data_loader
File "./vitaa/data/init.py", line 2, in
from .build import make_data_loader
File "./vitaa/data/build.py", line 7, in
from . import datasets as D
ImportError: cannot import name 'datasets'
According to these problems, I can't find datasets in "./vitaa/data"
How to solve this problem?