briang13 / flex Goto Github PK

Hi, I am trying the code base with a custom dataset with a different number of keypoints. But the model for some reason not learning angles and even after 300 epochs the angle prediction is not even near gt angles. Can you please guide me on this issue.
@BrianG13

Hello, since i am using a custom dataset with different number of joints i was wondering whether i can use a different forward kinematics code to compute joints after predicting angles, of course i would first convert those quaternion angles to euler then use them but can this function of forward kinematics be replaced

fake_pose_3d = self.fk_layer.forward(self.parents, skeleton.repeat(_input.shape[2], 1, 1), fake_full_rotations_list[i].contiguous().view(-1, self.config.arch.n_joints, self.rotation_number)).view(_input.shape[0], _input.shape[2], -1) this is the method i am talking about, in model.py

@BrianG13

Hello, can this project collect camera input and output 3D visualization results in real time? @BrianG13

Hi, (i) Has FLEX been tested on real Multi-person 3D Human Pose datasets since the paper only discusses synthetic dataset testing for multi-person setting?
(ii) Also, can the authors comment on the inference speed of the model when 2D keypoints are presented for each views using pretrained backbone like YOLOv7? Which part of the model makes the inference non-real time?

Upon inspecting the data, I notice that the 3D data, i.e. 3D pose and 3D rotation, also have the view element to them. In the evaluation code, the prediction of the network also contains the view dimension for 3D data. Why is this the case? They should be uniform across all views and should not have the view dimension, both intuitively and as stated in the paper. Am I missing something here?

Hi, i read your paper and came across that confidence score was also used, however in the config file confidence is set to false. Has the codebase not provided the use of confidence score, if it has how can we use it

Can the author elaborate on the following lines from the Conclusions and limitations section of the article? "One limitation of our approach is the dependency on 3D joint location ground-truth, and in particular, the requirement that it is given at the axis system of the train cameras." As far as I understand, the dependency of the 3D joint location ground-truth is due to the learning based approach that optimizes the Loss Functions. Not sure if I understand the second limitation.

Thank you for the amazing work! I was running the evaluation script. It seems you are using gt bones standard deviation for both GTbones2skel and bones2skel method. Could you please guide me what parameters could I use for inference. I tried using the mean of the predicted bones, no luck so far. @sigal-raab @BrianG13
skeleton = GTbones2skel(fakes_bones_list[i].clone().detach(), norm_parameters[2], norm_parameters[3], n_joints=self.config.arch.n_joints, start_position_as_totem=starting_pos_totem)

Hi, thank you again for your work
Could you please give me some instructions on how to use get the visualization from model's output?

Hey,

Thanks for the amazing work. I wonder where I can get the rigged skeleton model to visualize the results in Blender or whether we could create it manually. I'm relatively new to the field.

Best,
K

Thank you for your excellent work, i am trying to create a bvh file using bone hierarchy. Can you please guide me where to find this. Similar to this fashion @BrianG13
hierarchy = {'hips': [],
'hips2': ['hips'],
'lefthip': ['hips'],
'leftknee': ['lefthip', 'hips'],
'leftfoot': ['leftknee', 'lefthip', 'hips'],
'righthip': ['hips'],
'rightknee': ['righthip', 'hips'],
'rightfoot': ['rightknee', 'righthip', 'hips'],
'middleback':['hips2','hips'],
'neck': ['middleback','hips2','hips'],
'neck2': ['neck','middleback','hips2','hips'],
'neck3': ['neck', 'middleback', 'hips2','hips'],
'leftshoulder': ['neck','middleback', 'hips2','hips'],
'leftelbow': ['leftshoulder','neck','middleback', 'hips2','hips'],
'leftwrist': ['leftelbow', 'leftshoulder', 'neck','middleback', 'hips2','hips'],
'rightshoulder': ['neck','middleback', 'hips2','hips'],
'rightelbow': ['rightshoulder', 'neck','middleback', 'hips2','hips'],
'rightwrist': ['rightelbow', 'rightshoulder', 'neck','middleback', 'hips2','hips'],
'middlehead': ['neck','middleback', 'hips2','hips'],
'tophead': ['middlehead', 'neck','middleback', 'hips2','hips'],}

Hello, Thanks for excellent work. I am trying your codebase with my custom dataset and was confused regarding the file which contains the data for discriminator. Can that file be reused for any custom dataset and if not, how can this data be created for a custom dataset
@BrianG13

I read your paper with great interest, and thank you very much for sharing your code. I had a few questions after reading your paper.

1. According to your paper, you used Iskakov et al, ICCV 2019 model. Iskakov's proposed model includes an algebraic model and a volumetric model, is it correct that you used the algebraic model as a 2d pose estimator?

2. And did you use ResNet152 backbone and 384x384 image size for 2d pose estimation?

Thank you.

Thank you for the amazing work! I observed the base skeleton that was created in the FK module was different then the T pose that is normally created but I also saw in the else condition that a different base skeleton was there which resembled the T pose but it was not being used as it did not get in the else condition, can you shed some light upon it and if we were to use a custom dataset how are we to incorporate a base skeleton for it. Thanks @BrianG13

` if start_position_as_totem == True:
            skel_in[:, 1, 0] = -unnorm_bones[:, 0, 0]
            skel_in[:, 4, 0] = unnorm_bones[:, 0, 0]
            skel_in[:, 2, 1] = -unnorm_bones[:, 0, 1]
            skel_in[:, 5, 1] = -unnorm_bones[:, 0, 1]
            skel_in[:, 3, 1] = -unnorm_bones[:, 0, 2]
            skel_in[:, 6, 1] = -unnorm_bones[:, 0, 2]
            skel_in[:, 8, 1] = unnorm_bones[:, 0, 3]
            skel_in[:, 9, 1] = unnorm_bones[:, 0, 4]
            skel_in[:, 12, 1] = unnorm_bones[:, 0, 4]
            skel_in[:, 16, 1] = unnorm_bones[:, 0, 4]
            skel_in[:, 10, 1] = unnorm_bones[:, 0, 5]
            skel_in[:, 11, 1] = unnorm_bones[:, 0, 6]
            skel_in[:, 13, 1] = unnorm_bones[:, 0, 7]
            skel_in[:, 14, 1] = unnorm_bones[:, 0, 8]
            skel_in[:, 15, 1] = unnorm_bones[:, 0, 9]
            skel_in[:, 17, 1] = unnorm_bones[:, 0, 7]
            skel_in[:, 18, 1] = unnorm_bones[:, 0, 8]
            skel_in[:, 19, 1] = unnorm_bones[:, 0, 9]
else:
            skel_in[:, 1, 0] = -unnorm_bones[:, 0, 0]
            skel_in[:, 4, 0] = unnorm_bones[:, 0, 0]
            skel_in[:, 2, 1] = -unnorm_bones[:, 0, 1]
            skel_in[:, 5, 1] = -unnorm_bones[:, 0, 1]
            skel_in[:, 3, 1] = -unnorm_bones[:, 0, 2]
            skel_in[:, 6, 1] = -unnorm_bones[:, 0, 2]
            skel_in[:, 8, 1] = unnorm_bones[:, 0, 3]
            skel_in[:, 9, 1] = unnorm_bones[:, 0, 4]
            skel_in[:, 12, 1] = unnorm_bones[:, 0, 4]
            skel_in[:, 16, 1] = unnorm_bones[:, 0, 4]
            skel_in[:, 10, 1] = unnorm_bones[:, 0, 5]
            skel_in[:, 11, 1] = unnorm_bones[:, 0, 6]
            skel_in[:, 13, 0] = unnorm_bones[:, 0, 7]
            skel_in[:, 14, 0] = unnorm_bones[:, 0, 8]
            skel_in[:, 15, 0] = unnorm_bones[:, 0, 9]
            skel_in[:, 17, 0] = -unnorm_bones[:, 0, 7]
            skel_in[:, 18, 0] = -unnorm_bones[:, 0, 8]
            skel_in[:, 19, 0] = -unnorm_bones[:, 0, 9]`

Thank you for the amazing work! I want to run inference using videos from multiple views. Is there a demo script that I can use? @BrianG13 Thanks

Hi @BrianG13 , I am trying to train FLEX on my custom dataset. I am a bit confused about the code below:

offset = 10
train_frames = random.randint(10, 50) * 4 if self.config.trainer.train_frames == 0 else self.config.trainer.train_frames
train_frames = 196
self.sequence_index = np.array(slice_set(offset, train_frames, self.frame_numbers))
self.r_sequence_index = np.array(slice_set(offset, train_frames,  self.r_frame_numbers)) if self.is_train else 0

My dataset contains maximum 200 frames per video. Should I change my offset and train frames? Thanks

Hi，when I do evaluation by using your model but could not get your results. Have you checked the shared model ? @BrianG13

Hi @BrianG13 @sigal-raab you mentioned in the paper that "OpenPose and Iskakov et al. provide confidence values that we add to the network input. CPN does not provide these values, hence we assign identical confidence values for all joints when using it"

Can you please confirm the weights that you have provided is trained using confidence score? If yes, could you please guide me where should I pass the confidence scores for testing in the wild. If no, can you please guide me how can I retrain the model using confidence scores?

Thanks lots

Did you implement the TesseTrack paper? If yes, can you put that in your repo too?

"For comparing with the best available method, we have chosen the current state-of-the-art multi-view algorithm of Iskakov et al. [30] (TesseTrack [60] is marginally better, but it does not provide code). Since their algorithm is not epfree, we imitate parameter estimation by running a controlled perturbation of the camera parameters. We re-train their method with distorted data to simulate an environment where camera distortion parameters are unknown."

Will this method work if we train on 4 cameras but test on 3 or 2 cameras?

As you mentioned in the paper you trained on human3.6m and evaluated on KTH Football 2 dataset, but the details on the number of cameras aren't discussed. I understand human3.6m has 4 synced cameras and KTH has 3. So in order to evaluate FLEX on KTH for checking its generalization, did you train on human3.6m 3 cameras? or trained with 4?

I followed the training instructions as mentioned but encountered the following error
Traceback (most recent call last):
File "C:/Program Files/JetBrains/PyCharm Community Edition 2022.2.2/plugins/python-ce/helpers/pydev/pydevd.py", line 1496, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.2.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:\Users\Admin\flex\train.py", line 222, in
train(config, args.resume)
File "C:\Users\Admin\flex\train.py", line 139, in train
trainer.train()
File "C:\Users\Admin\flex\base\base_trainer.py", line 87, in train
result = self._train_epoch(epoch)
File "C:\Users\Admin\flex\trainer\multi_view_trainer.py", line 36, in _train_epoch
for batch_idx, data in enumerate(self.data_loader):
File "C:\Users\Admin.conda\envs\flex\lib\site-packages\torch\utils\data\dataloader.py", line 435, in next
data = self._next_data()
File "C:\Users\Admin.conda\envs\flex\lib\site-packages\torch\utils\data\dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "C:\Users\Admin.conda\envs\flex\lib\site-packages\torch\utils\data\dataloader.py", line 1111, in _process_data
data.reraise()
File "C:\Users\Admin.conda\envs\flex\lib\site-packages\torch_utils.py", line 428, in reraise
raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 0

Hello, i am looking to parallel train my model using multi gpu, can you please guide me regarding this. Thank you
@BrianG13