icodein / ai-dance-based-on-human-pose-estimation

Human Pose Estimation using Deep Learning.

AI-Dance-based-on-Human-Pose-Estimation

A Human Pose Skeleton represents the orientation of a person in a graphical format. Essentially, it is a set of coordinates that can be connected to describe the pose of the person. Each co-ordinate in the skeleton is known as a part (or a joint, or a keypoint). A valid connection between two parts is known as a pair (or a limb).

Dependencies

• Keras
• Librosa
• Moviepy
• OpenCV
• Pytube3
• Tensorflow

Flow of Human Pose Estimation

• Extraction of pose coordinates from dance videos using openpose human pose estimation.
• Training LSTM network on extracted coordinates using video as input and coordinates as output.
• Display output videos by joining predicted coordinates to generate dancing human stick figures.

How to Execute Code

1. You will first have to download the repository and then extract the contents into a folder.
2. Make sure you have the correct version of Python installed on your machine. This code runs on Python 3.6 above.
3. Now, install the required libraries.
4. Now go to src folder and run extract_data.py to download videos and audios to data folder. You can add youtube videos links to "video_links.txt" file for downloading.
5. Download pretrained weights for pose estimation from here. Download pose_iter_440000.caffemodel and save it in "models" folder.
6. Run demo.py to train LSTM and display predicted dance video.

python main.py --video data/video.mp4 --audio data/audio.wav --background data/bg1.jpg

Results

1. Pose estimation using openpose on image

2. Pose estimation using openpose on video

NOte : For more information about implementation details, please go through AI Dance based on Human Pose Estimation

References

1. OpenPose
2. Moviepy
3. Pytube
4. Dancing AI