zju3dv / smap Goto Github PK

After I ran commad below.

set -t generate_result
set -d test
exps/stage3_root2/bash test.sh
Then, I have obtained the json file "stage3_root2_generate_result_test_.json".

After getting the json files I tried to run refinet/test.py scripy by using bash exps/refinet/test.sh.
But the code did not run so I changed it to make it run and got the results below.

The result is root error of mean 49%. But in p2p_dataset.py the code seems to be making
the distance between 3d joints and relative to the root joint.

Is exps/refinet/test.py caculating PCKroot? or PCKrel.
Could you please give some more detailed explanation about the code in the readme??

Thanks

I have read your paper carefully. It mentioned that the batch size is set to 32, but I only see solver. IMG_PER_GPU = 2 in train.py. Is this a change in the code for a GPU training? Thanks a lot for your time and reading

Thanks for your share.
The evaluation metrics used in your paper are very reasonable, and I want to use them to test my own model. Could you share the relevant test codes?

nice work! could you release the dataset config or processing file of Panoptic and Human3.6M.Thanks.

Thank you for sharing your amazing work. I have tested SMAP on some custom videos, but the results are not as good as the ones in your demo videos. If possible, it would be very helpful if you can share some video links of videos that you used in your demo. Then I can test whether there is something wrong with my code.

Besides, I currently used OneEuroFilter for temporal smoothing, but I'd like to know if you use other methods since the results in your demo video are really amazing.

Thank you so much for your help and time.

with torch==1.4.0 and torchvision==0.5.0
when i run the test.sh, an error occured.
it seems that the model you provided is not comparable with the torch==1.4.0
however,when i update the torch to version 1.5.0, i can not install the dapalib package.
have you met this, and how should i do to solve this problem, thanks.

error are as follows:
Traceback (most recent call last):
File "test.py", line 225, in
main()
File "test.py", line 210, in main
state_dict = torch.load(model_file, map_location=lambda storage, loc: storage)
File "/home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/serialization.py", line 527, in load
with _open_zipfile_reader(f) as opened_zipfile:
File "/home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/serialization.py", line 224, in init
super(_open_zipfile_reader, self).init(torch.C.PyTorchFileReader(name_or_buffer))
RuntimeError: version <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fa59fec9193 in /home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1f5b (0x7fa5320e29eb in /home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7fa5320e3c04 in /home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #3: + 0x6c53a6 (0x7fa5a09e13a6 in /home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x2961c4 (0x7fa5a05b21c4 in /home/zhuguotao/software/anaconda3/envs/pose/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #38: __libc_start_main + 0xe7 (0x7fa5aee19b97 in /lib/x86_64-linux-gnu/libc.so.6)

Thanks for this great repo. I'm trying to run the code from pretrained SMAP. I followed the instructions same as Run inference on custom images but I'm getting following error (attached error screenshot) in stage3_root2 test.sh

My environment is torch1.5.0, torchvision0.6, and gcc 7.5c.

I am looking forward to hear from you!

First of all, thank you for providing the code that can be used for research, your work is very meaningful!
I learned from reading the source code that the visualization method is based on Matplotlib. I want to know how the visualization method in your paper is implemented?
If possible, can you provide some effective code, whether it is Matplotlib or other platforms.
Thank you!

Firstly,Thanks for your code sharing!But I have some confusion about your referred Depth-Aware Part Association.I don't find this part in your code.especislly**,you mentioned that you sort joints from near to far according to the predicted root depth,and your association process starts with the root and proceeds along the skeleton tree successively.**I don't understand your detailed algorithm process.Would you please provide more detailed explanation about this part and give a hit on your relative code position?

Thanks for sharing your great work!
I have a question about your experimental details on CMU Panoptic dataset. I see in the code that the V of the "bodys" field is divided into not labeled, locked, and visible. And the confidence of CMUP is between 0 and 1. I'm not sure how to deal with the confidence in the CMUP for SMAP train or test.

Hi， I want to test the results on the MUPOTS dataset. The script settings are as follows

and in convert.py I use the K from MuPoTs-3D
then set eval_mode = 1;is_relative = 0; in mupots-smap.m, The result is very poor! Could you tell me what went wrong?

The pred_2d and pred_3d has 15 joints which has no head and spine joints.

1. Did you change the (pred_2d and pred_3d) 15 joints to MuPoTs 17 joints (adding head and spine manually)
   before putting results "pred_2d" and "pred_3d"
   into the Moon's mpii_mupots_multiperson_eval.m code??
   Do I need to add head and spine in array 15,16?

2. In matlab code,
   should I set the safe_traversal_order = [2,1,10,11,12,4,5,6,13,14,15,7,8,9,3]
   and num_joints = 15?

Thanks for the release of your excellent work, could you please provide the model's weights trained with the Panoptic CMU dataset as the first table in your paper presents results with this dataset?

Thanks in advance

I found that in the paper, the MUCO dataset and CMU dataset were trained separately. So which dataset is the pretraining model downloaded from Google Driver based on? Can I directly test the CMU dataset to reproduce the results in the paper?

could you tell me how to convert coco to the format you mentioned in lib/preprocess/data_format.md. no 3d keypoints is available.

Hello.
Thank you for your great work.

Could you provide the Human3.6M or MuPoTS
Dataset code to match the results in the paper?!

The instruction"python setup.py install" error:
running install
running bdist_egg
running egg_info
writing dapalib.egg-info/PKG-INFO
writing dependency_links to dapalib.egg-info/dependency_links.txt
writing top-level names to dapalib.egg-info/top_level.txt
reading manifest file 'dapalib.egg-info/SOURCES.txt'
writing manifest file 'dapalib.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
Traceback (most recent call last):
File "setup.py", line 19, in
cmdclass = {'build_ext': BuildExtension}
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 172, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 158, in call_command
self.run_command(cmdname)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/command/install_lib.py", line 11, in run
self.build()
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/command/install_lib.py", line 107, in build
self.run_command('build_ext')
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 78, in run
_build_ext.run(self)
File "/home/dell/anaconda3/envs/torch11/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 257, in build_extensions
self._check_abi()
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 399, in _check_abi
check_compiler_abi_compatibility(compiler)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 187, in check_compiler_abi_compatibility
if not check_compiler_ok_for_platform(compiler):
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 163, in check_compiler_ok_for_platform
which = subprocess.check_output(['which', compiler], stderr=subprocess.STDOUT)
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/dell/anaconda3/envs/SMAP/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['which', 'x86_64-conda_cos6-linux-gnu-c++']' returned non-zero exit status 1.

hello, I am also a zju student, and I'm doing my graduate thesis base yours work. but I can't get the result(MPJPE) in yours paper on MuPoTS with your release model. so can you release the eval code in MuPoTS? Thank you so much!!!

Hi, I want to known how to apply temporal smothing to the inference results in the video

It seems that this training code would take long time to run? And the results can not be reproduced using the code you have released。

_pickle.UnpicklingError: A load persistent id instruction was encountered, but no persistent_load function was specified.

the dataset form Moon：
|-- MuCo
| | |-- data
| | | |-- augmented_set
| | | |-- unaugmented_set

for training ，only need the unaugmented_set or all data?

hello, if I want to change the net input size, how to change "association.cpp" ??
just change the size?
vector heatmapDim = {43, 128, 208}; // 1/4 height, 1/4 width

when i run bash test.sh
from cvpack.utils.logger import get_logger
from model.smap import SMAP
from model.refinenet import RefineNet
from lib.utils.dataloader import get_test_loader
from lib.utils.comm import is_main_process
from exps.stage3_root2.test_util import *
from dataset.custom_dataset import CustomDataset

the following error occurred :
Traceback (most recent call last):
File "test.py", line 14, in
from cvpack.utils.logger import get_logger
ModuleNotFoundError: No module named 'cvpack'
other modules such as model,lib,exps can not be found as well.
how to solve it?

Hi, after successfully installed all the dependence, I tried to test on my own images, but got an error like above, and the detail error information like:
pred_body2_2d = dapalib.connect(...)
RuntimeError: scan failed on 2nd step: cudaErrorInvalidValue:invalid argument

my environment: pytorch1.3 cuda10.1 python3.7

I finally found the detailed configuration for the CMU panoptic. Thanks! #1 (comment)

However, still it is unclear which images you used for training and testing.
The paper says

Following [41], we choose two cameras (16 and 30), 9600 images from four activities (Haggling, Mafia, Ultimatum, Pizza) as our test set, and 160k images from different sequences as our training set.

As far as I know, there are much more image frames than 9600 in the test sequences you listed here #1 (comment).

I hope you can tell the specific frame sampling configuration!

Excuse me ,I have some problem in coordinate conversion.Whether pred_bodys_2d or new_pred_bodys_3d in test.py can not recover the pixel coordinates matching to the origin images.Thanks a lot!(By the way ,your code for 3D HPE works perfect.)

Thanks for your sharing. There are some questions about Panoptic.

1. Which sequence of Panoptic data has been used during training
2. Can you share 160422_mifia2's 3D pose annotation, because the official website is now unavailable

Thank you in advance

when running inferece on custom images, the error are as follows:
Traceback (most recent call last): File "test.py", line 22, in <module> import dapalib ImportError: /home/dell/anaconda3/envs/SMAP/lib/python3.7/site-packages/dapalib-0.0.0-py3.7-linux-x86_64.egg/dapalib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

Have you encountered this question before and could you give me any solution?

Excuse me ,I try to train the model and encounter the problem like this:
Traceback (most recent call last):
File "train.py", line 115, in
main()
File "train.py", line 55, in main
data_loader = get_train_loader(cfg, num_gpu=num_gpu, is_dist=engine.distributed,use_augmentation=True, with_mds=True)
File "/home/mist/SMAP/lib/utils/dataloader.py", line 23, in get_train_loader
dataset = Dataset(cfg, 'train', transform, use_augmentation, with_mds)
File "/home/mist/SMAP/dataset/base_dataset.py", line 43, in init
data = data_this['root'] + data
KeyError: 'root'

As for coco2017, I find create_annot.py to transform the format,and it works.But for MuCo dataset,how to transform its format? My MuCo dateset is downloaded in your link(https://drive.google.com/drive/folders/1yL2ey3aWHJnh8f_nhWP--IyC9krAPsQN).

For pytorch >= 1.1.0, compilation of cpp extension will fail by expected primary-expression before float at hmsIn.data_ptr<float> in L52 in association.cpp with gcc >= 6.0. This is because cpp extension api tensor.data_ptr is available in pytorch1.0, however it's moved to tensor.data in later version.

I found you used different evaluation metrics in 'MuPoTS-3D' and 'Panoptic' dataset. But you didn't release the evaluate code of 'Panoptic' dataset. Is it convenient for you to release the evluate code of it?

您好，可否分享一下数据集的tar包，我这边一直下载不下来，谢谢了

hi,I want to ask that you how to calculate the running time of the model.It's seems that if I calculate all the time includes all net parts and some post-porcess parts,the average time is about 100 ms that is different to the value in paper.But i think it's enough fast in real scene.Thanks a lot!!

when i start training with this project, i find that the loss is too large to converge. after 70k iteration, loss_2d loss about 384,loss_bone about 200,loss_root about 2.is there anything wrong with these loss. thanks.

Thanks for sharing your nice work!

As mentioned in your paper, the size of root depth map is the same as the input image image. However, I find it seems that
(1)the root joint locations are also reduced by a quarter as 2d heatmap in base_dataset.py;
(2) root joint locations are used to regress 3D pose instead of using root depth map.

Maybe I miss some points.

Is the root position in the result file? If so, is it in the world coordinate or in the camera coordinate?
Looking forward to your reply.

Thanks for sharing your great work!
I have a question about your experimental details on CMU Panoptic dataset: which sequences do you train and test on?

hello, could you tell me that how to select the best smap model and to genrate the refineNet training dataset to train the refineNet?? I found that if the whole body can be detected, the results would be worse though the refineNet ..
And that, if I change the backone and remove the crose-to-fine step, the root depth map cannot reach the result about the paper ..
thanks

Thanks for providing the script to convert the coco to your format. Is it possible to provide the same thing for the MuCo and CMUP?

Excuse me, after scaning your code,I think SOLVER.IMG_PER_GPU in config means batchsize.So I set it from 2 to 4,but the costing time doubles.I don't know the reason and could you give some explainations?Thanks a lot!

Hi, thanks for your great work.

I try to run your model on my custom videos. And I notice in the final JSON file, the shape of 2D predictions is (15, 4). I expect the shape is (15, 2) and have no idea why there are 4 values. Could you please clarify the meaning of each coordinate?

Thanks for this great repo. I'm trying to run the code from pretrained SMAP and RefineNet. I followed the instructions same as Run inference on custom images but I'm getting following error (attached error screenshot) in stage3_root2 test.sh

My system environment is Ubuntu 18.04,Python 3.7 and torch 1.4

Is there any need to provide MuCo datasets for inference from pretrained models?

Thank you for sharing your amazing work. I'm doing the research in 3DHPE. I can't get the result(MPJPE) in yours paper on MuPoTS with your release model. so can you release the eval code in MuPoTS? I'm appreciate it for your help and time.

Dear authors,
I was wondering:

1. how do you compute the MPJPE metric for evaluation in CMU Panoptic dataset?
2. Do you use visibility in order to select only visible joints in the image or do you do the calculation or the error with all the joints regardless of their visibility on the image?

Thanks!

I want to see the results on the Human36M Dataset!
Is there a Model trained on Human36M dataset?
If there is one, it will be really great if you provide it.