y-zheng18 / gimo

Official repo of our ECCV 2022 paper "GIMO: Gaze-Informed Human Motion Prediction in Context"

Hi there 👋

About me: https://y-zheng18.github.io/

gimo's Issues

start and end frame in dataset.csv

Hi, thanks for the great work, I want to know whether you just use the frames between the start and end frame in dataset.csv for training. If so, how do you define the start and end frame and why do not use other frames?

Quick Start Fail

Hi, I'm trying to run the codebase on Colab and quick start using the demo data using this command:
!bash scripts/eval.sh

Then, I met the following issue can you help me to figure out how to resolve it, thank you:

`2023-06-20 08:33:25.741889: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.

To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-20 08:33:27.671593: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
loading scene of 2022-01-23-080759
Traceback (most recent call last):
File "/content/drive/MyDrive/GIMO/eval.py", line 284, in
evaluator = SMPLX_evalutor(config)
File "/content/drive/MyDrive/GIMO/eval.py", line 23, in init
self.vposer, _ = load_vposer(self.config.vposer_path, vp_model='snapshot')
File "/usr/local/lib/python3.10/dist-packages/human_body_prior/tools/model_loader.py", line 56, in load_vposer
ps, trained_model_fname = expid2model(expr_dir)
File "/usr/local/lib/python3.10/dist-packages/human_body_prior/tools/model_loader.py", line 31, in expid2model
best_model_fname = sorted(glob.glob(os.path.join(expr_dir, 'snapshots', '*.pt')), key=os.path.getmtime)[-1]
IndexError: list index out of range`

Obtain the smplx parameters in scene coordinates, confusing points about align_smpl.py and eval_dataset.py

Hi, thanks for your great assets. I am trying to align the smplx parameters with the scene world. I have a few questions after looking into eval_dataset.py and align_smpl.py. I am not an expert in 3D transformation, hope you don't mind my "silly" questions:

I feel some parts in eval_dataset.py and align_smpl.py are not consistent:

1. How could we get the smpl.obj from the smplx.pkl? Should we pass the smplx.pkl to vposer to otain pose parameters, and then use smplx to obtain the shape vertices. I get some related codes in utils/vis_utils.py. But not very sure if there are further transformations.
2. In align_smpl.py, the smpl.obj are loaded, rescaled (become larger), and then transformed to scene coordinates using pose2scene RT matrix. While in eval_dataset, we do nothing to smpl.obj, but to rescale and transform the smpl parameters (specifically the global orientation and translation). Also in eval_dataset, we load the scene_downsampled.ply, instead of the textured mesh. Some other confusing parts includes:

• Is scene_downsampled.ply just downsampled from textured_output.obj? Or there are still rescaling and RT transformation here.
• What is the transform_norm.txt? I notice this is to transform the scenes into some canonical space. But I didn't find explanations about this in the paper or git. Why should we apply transform_norm to smpl parameters in eval_dataset.py ?
• Why in align_smpl.py we apply scale to smpl.obj, while in eval_dataset.py we apply 1/scale to the gloabl orientation and translation.
• In align_smpl.py, we apply the rescale to the whole smpl model. While in eval_dataset.py, we only apply the rescale to the global translation, in which case the scale of the smpl remains unchanged.

I know these are A LOT of questions. I would greatly appreciate it if you could help clarify this. I believe these can also be helpful for other beginner to use this dataset.

About the training details

I'm currently working on a project involving this dataset, and I have a few questions regarding the training details. I would appreciate any insights or guidance you can provide.

1. Optimizer:
   I would like to know what optimizer was used for training and the settings of the optimizer.

2. Loss:
   Could you please clarify the choice of loss function used during training? I saw "lambda_rec" and "lambda_des" in the config.py.

3. Pretrained Scene Encoder:
   I am not sure whether this project uses pretrained scene encoder.

And it would be much appreciated if I could have the training code or more details of the training process.

About dataset

Hi, thanks for your nice work! Is there reasonable text prompt for each motion sequence?

Dataset

Hi. Thank You for your great work. Can I have access to your dataset?

Thank You