cleardry / avatars4all

Live real-time avatars from your webcam in the browser. No dedicated hardware or software installation needed. A pure Google Colab wrapper for live First-order-motion-model, aka Avatarify in the browser. And other Colabs providing an accessible interface for using FOMM, Wav2Lip and Liquid-warping-GAN with your own media and a rich GUI.

avatars4all

Live real-time avatars from your webcam in the browser. No dedicated hardware or software installation needed. A pure Google Colab wrapper for live First-order-motion-model, aka Avatarify in the browser. And other Colabs providing an accessible interface for using FOMM, Wav2Lip and Liquid-warping-GAN with your own media and a rich GUI.

Based on the works:

1. First Order Motion Model for Image Animation, https://aliaksandrsiarohin.github.io/first-order-model-website
2. Avatarify, https://github.com/alievk/avatarify
3. Webcam for Google Colab over Websocket, https://github.com/a2kiti/webCamGoogleColab
4. Wav2Lip, Accurately Lip-sync Videos to Any Speech, http://bhaasha.iiit.ac.in/lipsync
5. Liquid Warping GAN (Impersonator), https://svip-lab.github.io/project/impersonator
6. Liquid Warping GAN (Impersonator++), https://www.impersonator.org/work/impersonator-plus-plus.html
7. pyAudioAnalysis, https://github.com/tyiannak/pyAudioAnalysis
8. pyannote-audio, https://github.com/pyannote/pyannote-audio
9. U^2-Net, https://github.com/NathanUA/U-2-Net
10. MODNet, https://github.com/ZHKKKe/MODNet

In this repository you will find:

1. Colab for live real-time talking head deep-fakes from your webcam. (j.mp/cam2head)
2. Colab for creating talking head deep-fakes (VoxCeleb model) from YouTube or other videos. (j.mp/vid2head)
3. Colab for creating full body deep-fakes (Tai chi and fashion models) from YouTube or other videos. (j.mp/vid2body)
4. Colab for creating full body deep-fakes (impersonator model) from YouTube or other videos. (j.mp/vid2act)
5. Colab for creating full body deep-fakes (impersonator++ model) from YouTube or other videos. (j.mp/vid2warp)
6. Colab for creating lip sync deep-fakes based on audio. (j.mp/wav2lip)
7. Colab Green screen effect for video with optional background video, and sketch, bokeh and more effects. (j.mp/vid2green))

Features:

1. The fastest purely online solution I am aware of for live real-time first-order-motion-model avatars from your webcam.
2. A new auto-calibration mode that works in real-time!
3. A new exaggeration factor to get those damn muppets to open their mouths!
4. Drag and drop local/web images on the GUI to upload new avatars!
5. Options to switch between avatars, including newly generated StyleGAN faces, as inspired by Avatarify, of:

• People: https://thispersondoesnotexist.com
• Man, woman, boy, girl: https://fakeface.rest
• Waifus! https://www.thiswaifudoesnotexist.net
• Forsunas! https://thisfursonadoesnotexist.com
• Muppets! https://eyalgruss.com/thismuppetdoesnotexist (made especially for this with Doron Adler @norod78)

6. Smart auto-pad/crop/resize to the head or body, for images and for offline videos, tuned for best results.
7. Full control of model parameters as well as zoom and buffering options in the GUI.
8. Upload your own images and videos or pull them from the web including from YouTube, etc., and optionally trim videos.
9. Visualization of facial landmarks and their alignment between source and target.
10. Download videos with original audio and framerate, and optimized for compatibility.
11. One click operation with Runtime -> Run all.
12. Optional Wav2Lip post processing following head animation.
13. Combining Wav2Lip with speaker diarization for automatic animated skit creation from audio ("Wav2Skit").

Reference implementations:

1. https://colab.research.google.com/github/AliaksandrSiarohin/first-order-model/blob/master/demo.ipynb
2. https://colab.research.google.com/github/tg-bomze/Face-Image-Motion-Model/blob/master/Face_Image_Motion_Model_(Photo_2_Video)_Eng.ipynb
3. https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb
4. https://colab.research.google.com/github/a2kiti/webCamGoogleColab/blob/master/webCamGoogleColab_websocketVersion.ipynb
5. https://colab.research.google.com/github/thefonseca/colabrtc/blob/master/examples/colabrtc.ipynb
6. https://github.com/l4rz/first-order-model/tree/master/webrtc
7. https://gist.github.com/myagues/aac0c597f8ad0fa7ebe7d017b0c5603b
8. https://colab.research.google.com/drive/1tZpDWXz49W6wDcTprANRGLo2D_EbD5J8
9. https://colab.research.google.com/github/svip-lab/impersonator/blob/master/impersonator.ipynb
10. https://colab.research.google.com/drive/1bwUnj-9NnJA2EMr7eWO4I45UuBtKudg_
11. https://terryky.github.io/tfjs_webgl_app/face_landmark
12. https://eyaler.github.io/tfjs_webgl_app/face_landmark

Workshops, tutorials and talks

• 2020-05 Generative Jam
• 2020-06 Убежище / Suoja / Shelter Festival
• 2020-06 Open Data Science Conference (ODSC)
• 2020-07 47th South Africa National Arts Festival (NAF)
• 2020-07 16th Athens Digital Arts Festival (ADAF)
• 2020-09 ReclaimFutures Conference
• 2020-10 /’fu:bar/ Glitch Art Fest̯ivalͦ
• 2020-10 26th International Symposium on Electronic Art (ISEA)
• 2020-11 Technarte Art & Technology International Conference
• 2020-11 30th Electronic Visualisation & the Arts London (EVA)
• 2020-11 18th Piksel Festival for Electronic Art and Free Technologies
• 2020-12 IDC Herzliya DataTech
• 2021-01 34th Stuttgarter Filmwinter Festival for Expanded Media
• 2021-02 Dorot Haifa Conference for Science Fiction, Fantasy and RPG
• 2021-03 York University Computational Arts, video part 1
• 2021-03 Mozilla Festival 2021 video
• 2021-04 2nd Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN @ IUI)
• 2021-09 12th International Conference on Computational Creativity (ICCC)