rotem-shalev / ham2pose

Official implementation for "Ham2Pose: Animating Sign Language Notation into Pose Sequences" [CVPR 2023]

Ham2Pose

Official implementation for "Ham2Pose: Animating Sign Language Notation into Pose Sequences". The first method for animating HamNoSys into pose sequences.

Getting Started

1. Create a virtual environment

$ conda create --name Ham2Pose python=3.7
$ conda activate Ham2Pose
$ pip3 install requirements.txt

2. Prepare new data: To train the model using data that isn't part of our dataset, download the videos and use OpenPose to extract body, face, and hands keypoints from them (137 in total).

Train

To train a new model using the default configurations run the following command:

$ python train.py

To change specific arguments, pass them as explained in the args.py file, e.g. to change the number of gpus to 4 and the batch size to 32 use:

$ python train.py --num_gpus 4 --batch_size 32

To pass a complete yaml configuration file use:

$ python train.py --config_file configs/config.yaml

Test

To test an existing model with the default configuration use:

$ python test.py

The default configuration will test our supplied pretrained model "Ham2Pose". To train and test a different model, either change the model_name in the configuration, or delete the existing checkpoint from the models directory.

To change other arguments use one of the options mentioned under Train.

Citation

If you find this research useful, please cite the following:

@article{shalev2022ham2pose,
  title={Ham2Pose: Animating Sign Language Notation into Pose Sequences},
  author={Shalev-Arkushin, Rotem and Moryossef, Amit and Fried, Ohad},
  journal={arXiv preprint arXiv:2211.13613},
  year={2022}
}

ham2pose's Issues

Saving Ham2Pose as json files

Hi authors,
Thank you for sharing your remarkable code base. I was trying to get the coco keypoints in json formats (similar to openpose outputs). But I am finding hard to extract it because of the pose_format library module. Can you guide me in getting the json file for every frame ?

Hamnosys text to model input

Hi authors,
thank you for sharing the code base.

How can I convert the the Hamnosys text say (hamextfingerl,hampalml,hamchest,hammoveur,hamplus,hamsymmlr) to the model input which is an utf-8 encoding like
(\xee\x83\xa9\xee\x80\x80\xee\x80\x8d\xee\x80\xa6\xee\x80\xba\xee\x81\xaa\xee\x83\xa0\xee\x83\x90\xee\x81\xa9\xee\x83\xa1\xee\x82\x89\xee\x83\x86)

Dataset arrangements

Hi,

Thank you for your amazing work!

I would like to know more about the dataset layout that you used in the paper. You have mentioned 4 datasets in the paper but I only found 2 in the code.

Also, will there be a guideline on how to organize the dataset for training your method? (e.g. How to download, data format, path)

Many thanks,

How to get the text description from HamNoSys?

Hi,

I would like to know how did you extract the text description in the figure like: "Two flat hands with fingers closed, rotated towards each other, touching, then symmetrically moving diagonally downwards". Thank you very much!

Best

tokenizer issue

Hi! I am getting this issue after the installing the dependencies
RuntimeError: Failed to import transformers.models.auto because of the following error (look up to see its traceback): No module named 'tokenizers.pre_tokenizers'

Is anyone else facing this issue?