gordon-lim / ai-learns-to-tiktok

Training an encoder-decoder model to generate a sequence of poses from spectrograms of music snippets.

AI Learns to TikTok

Training an encoder-decoder (CNN-LSTM) model to generate a dance (or a sequences of poses) from spectrograms of music snippets.

Table of Contents

1. About The Project
   • Built With
2. Overview
3. Prerequisites
4. Usage
5. Logs
6. Contributing
7. License
8. Contact
9. Acknowledgements
10. Useful links
    • YouTube videos

About The Project

I was inspired by the recent TikTok trend. I figured since there are so many users on TikTok, it would be quite easy obtaining large amounts of dance video data. This along with my knowledge on pose estimation from doing Udacity's Intel Edge AI for IoT Developers Nanodegree and image captioning from doing Udacity's Computer Vision Nanodegree, I felt that this project would be accessible to me.

Built With

• PyTorch
• OpenPose

Overview

Prerequisites

OpenPose needs to be installed

Usage

Refer to Jupyter Notebook in the order corresponding to their indexes

Logs

• (041120) AI only 'sees' left channel of music as you can only create spectrogram from one channel
• (091120) 15 second spectrogram might cause data to be lost
• (091120) In image captioning I dealt with around 10 words, but for a 15 second video at 15fps I will have 225 frames. I don't think the LSTM may work well with a 225 word long sentence
• (141120) Not able to use cv2 and openpose within the same script because of a memory allocation error (TCMalloc); was working fine until I decided not to work in a virtual env for scikit learn to work
• (281120) My pose plots are upside down but I don't think it should matter
• (011220) Tried to overfit the model just for a proof of concept but couldn't get my l1loss below 20. Model will repeat the same predictions after a few frames.
• (041220) After revisting my code, I realised that I assumed the input shape to lstm as (batch, seq_len, input_size). I found out from the documentation that the default is (seq_len, batch, input_size). This is huge! I have since 'set batch_first=True' in the lstm params and will be training my model from scratch.

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Gordon - [email protected]

Acknowledgements

• Udacity

Useful Links

• Human Pose Estimation
• Effects of spectrogram pre-processing for audio classification
• Create csv for dataloader
• TCMalloc error
• How to normalize spectrograms

YouTube videos

• Human Pose Estimation on Dancers
• Method to make real person dance from pose data
• Similar project by Carykh
• Was wondering if data was lost in spectrograms
• Building dataloader