google / aistplusplus_api Goto Github PK

How to convert the 2d keypoint JSON file detected by openpose to the same 2d keypoint format as the AIST++ dataset.

I'm very interested in this dataset so I did some experiments with SMPL backbone. But the result turns out quite surprising, so I further analyzed the dataset. My approach is to use run_vis.py to re-run unreasonable videos on my model. And here are my results:

• gBR_sFM_c09_d04_mBR4_ch07

with the setting

flags.DEFINE_string(
    'video_name',
    'gBR_sFM_c09_d04_mBR4_ch07',
    'input video name to be visualized.')
flags.DEFINE_enum(
    'mode', 'SMPL', ['2D', '3D', 'SMPL', 'SMPLMesh'],
    'visualize 3D or 2D keypoints, or SMPL joints on image plane.')

• gBR_sBM_c05_d04_mBR0_ch08

with the setting

flags.DEFINE_string(
    'video_name',
    'gBR_sBM_c05_d04_mBR0_ch08',
    'input video name to be visualized.')
flags.DEFINE_enum(
    'mode', 'SMPL', ['2D', '3D', 'SMPL', 'SMPLMesh'],
    'visualize 3D or 2D keypoints, or SMPL joints on image plane.')

• gJB_sBM_c06_d07_mJB3_ch05

with the setting

flags.DEFINE_string(
    'video_name',
    'gJB_sBM_c06_d07_mJB3_ch05',
    'input video name to be visualized.')
flags.DEFINE_enum(
    'mode', 'SMPLMesh', ['2D', '3D', 'SMPL', 'SMPLMesh'],
    'visualize 3D or 2D keypoints, or SMPL joints on image plane.')

I'm wondering if anyone else has similar results? Or did I make a mistake on the code running? Cause I found hundreds of videos like above.

I meet 504 error: Gateway Time-out when I download the videos.
I've tried many ways but failed and need some help.

Hey, thanks for your amazing work for constructing this dataset. I encountered a small problem when I try to extract 3d joints from SMPL parameters. The scale doesn't seem right. It turns our we need to use "https://github.com/liruilong940607/smplx" instead of the original repo for "python setup.py install" after "pip install smplx". I feel it important to indicate that in the main introduction since it's a key step.

Just a minor suggestion. Hope you don't mind.

Hi, @liruilong940607
I have some questions about the input and output of transformer network.

1. As the paper said, during training, the input of the Audio Transformer is music feature Y with shape (240, 35) and the input of Motion Transformer is motion data X with shape (120, 219) , so for all of Q,K,V are same, include the query vector of decoder of these transformer. Is this right?
2. For the cross-modal transformer, assume I get H(X) (240, 512) and H(Y) (120, 512) after audio and motion transformer, how to fuse features from music and motion features? Just concatenate both in sequence dimension resulting in features with shape (240 + 120, 512)?
3. What's the query vector of cross-modal transformer decoder? How to get a motion output with 20 frames?
4. What is the representation of motion prediction? Is it same as the input motion data with 24*3*3 + 3 = 219. But I think it is not good to regress a 3x3 rotation matrix directly? And for the regression of global translation, do you directly regress the absolute translation (x,y,z) in 3D space? Or just the offset relative to the first frame of seed motion?
   Thanks!

Hi,
Thanks a lot for releasing this dataset.
You are providing visualization of keypoints, that's great. However you are not providing a way to visualize SMPL from each camera view point, would it be possible to incorporate this in the run_vis.py file?
For the moment the SMPL poses is independant to camera viewpoints, how to you make it dependant from a specific camera viewpoint?
Thanks a lot for your help.
Fabien

We noticed that your paper stated that the dataset contained 3D joint rot data, but we cannot find it on the web page. So I was wondering if there are any accesses to download the 3D joint rot data. We hope to do some research with this data.
I look forward to hearing from you.

您好，我在使用过程中将keypoints2d 绘制在图像的时候是对的，将keypoint3d 经过投影绘制在图像上，差的很多。
1、 根据video name 从mapping 里面获取 cames的参数，
2、 世界坐标系下的kpt 转换到 2d图像上的kpt
您能帮我分析一下是哪里不对吗？头都大了

I Downloaded the Keypoints 3D only and it says "all annotations are 60 fps", however when i open them they seems to be 30fps for example a motion of duration : 1 minute and 35 seconds is around 2800 frames not 5600 is it right ? should i treat the keypoints3d as 30fps.

Plus: if it is so and i want to extract audio_features with the same fps from audio, is it right to use a sample rate = 48000 and hope_size = 800 ?

Hi ruilong, I'm trying the visualize the test result from the trained MINT model, I try to run the following demo code and the output is a freezing image like below.

Could you tell me is there a code to visualize(render) the generated dance output to a video like the videos provided in theoverview of AIST++? Thank you so much.

I still have concern. I use your code but the results are still squeezed together. Is the smplx version or SMPL_MALE.pkl any different ?

Hi,

Your work is excellent and I am interested in it! Now I'm trying my hands on visualizing the SMPL joints.

But I have no idea about where the file "SMPL_MALE.pkl" is and what function it serves as. Could you please tell me about it?

Sincerely,
Shuhong

Hi, thanks for your great contribution of this job. I wonder what the roations are in smpl_poses. I know the the dim of smpl_poses is [N, (24 * 3)]. Howerver, it don't seem right when I use the smpl order as the ReadME describes.

[
 "root", 
 "left_hip", "left_knee", "left_foot", "left_toe", 
 "right_hip", "right_knee", "right_foot", "right_toe",
 "waist", "spine", "chest", "neck", "head", 
 "left_in_shoulder", "left_shoulder", "left_elbow", "left_wrist",
 "right_in_shoulder", "right_shoulder", "right_elbow", "right_wrist"
 ]

I want to reorder the axis angles in smpl_poses with the follow order:

[
    "root",  # 0
    "lhip",  # 1
    "rhip",  # 2
    "waist",  # 3
    "lknee",  # 4
    "rknee",  # 5
    "spine",  # 6
    "lankle",  # 7
    "rankle",  # 8
    "chest",  # 9
    "ltoe",  # 10
    "rtoe",  # 11
    "neck",  # 12
    "lclavicle",  # 13
    "rclavicle",  # 14
    "head",  # 15
    "lshoulder",  # 16
    "rshoulder",  # 17
    "lelbow",  # 18
    "relbow",  # 19
    "lwrist",  # 20
    "rwrist",  # 21
]

I build a mapping index array to select the poses as follow:

order_map = [0, 5, 1, 9, 6, 2, 10, 7, 3, 11, 8, 4, 12, 18, 14, 13, 19, 15, 20, 16, 21, 17]

smpl_poses = smpl_poses.reshape(
    smpl_poses.shape[0], -1, 3)[:, order_map, :].reshape(smpl_poses.shape[0], -1)

But, I got a weird result:

Do I make a mistake?

In the dataset "aist_plusplus_final", the music data is missed. If I want to start my task that focus on music conditioned motion generation, maybe the AIST++ can't be used.

Hi, thanks for your amazing dataset! I found this dataset does not contain the shape parameters of SMPL model. Also, the learning rate is set to 0 for all stages. Also, shape parameters will affect the scale optimization. So, which shape parameters should I use for visualizing the SMPL mesh?

Hi，thanks for sharing such a great dataset. I have downloaded the dataset but failed to find the ignore_list.txt which is mentioned in AIST++ websites ?

I am trying to project the keypoints viewed from the Camera of (0,0,0) exactly as the image through the world_to_camera method and additional methods. For example:

In above image, assuming that the camera was shot at (0,0,0) in the current 3d space, I would like to match the rotation and translation of the keypoints as shown in the photo above. (The photo above was adjusted manually and was not correct. Also, don't mind scattered points of any color other than the human keypoint below)

However, when the camera translations of the annotations are displayed in 3d space, they appear as follows compared to other keypoints.
You can see that the height does not match with the human keypoints, and it is not surrounded by 360 degrees and is clustered in a specific area.

Looking at it a little further, it looks like this:

I think the coordinate scale of camera translation and human keypoints translation is different or there are other reasons.
If what I didn't aware about something can you explain?

Thank you very much for providing the dataset.

Hi @liruilong940607 ,
Thank you for your wonderful work!

With the help of smpl.forward().joints , 'smpl_poses' and 'smpl_trans' conver to 'keypoints3d' .
But How to conver 'keypoints3d' to 'smpl_poses' and 'smpl_trans' ?

Could you tell me why do you choose 1600 as initial f for camera intrinsics? Does this value influence the result importantly? I would really appreciate your response

I tried to repeat Step 4 to get neutral SMPL annotations. But I got such an exception:

File "/home/anna/code/aistplusplus_api/processing/run_estimate_smpl.py", line 172, in main
    smpl, loss = smpl_regressor.fit(keypoints3d, dtype='openpose25', verbose=True)
  File "/home/anna/code/aistplusplus_api/processing/run_estimate_smpl.py", line 113, in fit
    keypoints3d = keypoints3d[:, mapping_target, :]
IndexError: index 17 is out of bounds for axis 1 with size 17

What is wrong? How can it be fixed?

Hi there!
Hope you are well! :)

I am desperately trying to visualize the SMPL pose parameters from your dataset but it seems that your axis convention does not meet mine.

I have got inverted and inconsistent rotations as in the following picture :

I have tried to swap (x, y z) axis, oppose signs in the x, y or z direction without success.

Do you have any idea of the transformation I need to make to visualize the animation properly ?
Could you please provide the axis convention used for your skeleton please ?

Thank you :D

hi, ruilong,
Thanks for you great work, what's difference between keypoints3d_true and keypoints3d? I use keypoints3d to generate smpl, but it didn't work.

`
smpl_poses, smpl_scaling, smpl_trans = AISTDataset.load_motion(
'motions', 'gBR_sBM_cAll_d04_mBR0_ch01')
smpl = None
smpl = SMPL(model_path='/home/public_data/sjc/smpl/models/', gender='MALE', batch_size=1)
keypoints3d = smpl.forward(
global_orient=torch.from_numpy(smpl_poses[:, 0:1]).float(),
body_pose=torch.from_numpy(smpl_poses[:, 1:]).float(),
transl=torch.from_numpy(smpl_trans/smpl_scaling).float(),
).joints.detach().numpy()[:, 0:24, :]

keypoints3d_true = AISTDataset.load_keypoint3d(
'keypoints3d', 'gBR_sBM_cAll_d04_mBR0_ch01', use_optim=False)

smpl, loss = smpl_regressor.fit(keypoints3d, dtype='coco', verbose=True)
`

Hi,
would you please provide fbx models for smpl body motions?
pkl models are quite good for python, but it will be convenient to use the fbx model in 3D software.

Hi, I have used the model to create the SMPL but is there any way to reconstruct 3d skeletons given in the github page?

I can't find the download button of SMPL model on the download page of https://smpl.is.tue.mpg.de/downloads.

Is the download button of SMPL model has been cancelled？

Thanks!

Hi，

Thanks for your. wonderful datasets.

I recently do some thing about 3D humane pose and I tried to download your dataset. However I always meet the error 'multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f344db94160>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'.

Thus I am not sure whether I have finished the downloading. I find a video file list in downloader.py, so I download it individually. I find the video_list contains 12669 items, but it not equal to the annotations which only contains 1408 SMPL files.

I wonder how to check the dataset? One more thing, how can I fixed the error when I using?

Looking forward to hearing from you,

I'm trying to run the code of visualize the smpl mesh but it failed and here is the error.
Would you please tell me why it doesn't work? Thanks.

file URL https://s3.abci.ai/aistdancedb.ongaaccel.jp/v1.0.0/video/10M/gLO_sBM_c01_d13_mLO1_ch09.mp4
downoad failed，How to get the video file ?

Thanks for your great job.
I have trouble when rendering the SMPL mesh. could you share the renderer code?
Thanks.

Hi,
Is it possible to post a script that aligned the music and motion recordings?
As I can see, in the AIST dataset the audio fragments do not match the length of the corresponding fragments, and when extracting audio directly from a video fragment it is not clear how to properly align the individual features of the audio and movement frames.

Timur Sokhin

Hi,
when i use ffmpeg_video_read to extract the image,i specified a frame rate of 60,but i found the number of frames and the number of corresponding motions are unequal.
while if i don't specify fps,the number are equal.
i use gHO_sBM_c01/02...09_d19_mHO2_ch02 .mp4 and get 481 images,but smpl poses only have 480.so do i need to specify the frame rate?
Thanks！