pawansharmaaaa / lip_wise_gfpgan

wav2lip that uses GFPGAN for upscaling.

LIP_WISE_GFPGAN: Wise enhancements for wav2lip.

📝 Note:

This Project has moved here

Introduction

LipWise is an enhanced version of the Wav2Lip model, a deep learning model for lip-synced video generation. It operates by processing an input audio clip alongside a reference video featuring a speaking individual. The model then synthesizes a new video where the lip movements of the person in the reference video align seamlessly with the provided audio. LipWise offers several significant improvements over the original Wav2Lip model, including:

• High-resolution output via GFPGAN and RealESRGAN
• Frame removal script
• Enhanced face detection algorithm

Features

• High-resolution output via GFPGAN and RealESRGAN: LipWise integrates GFPGAN and RealESRGAN, two cutting-edge GANs that specialize in upscaling images to higher resolutions while preserving impeccable image quality and intricate details. This allows LipWise to generate high-resolution lip-synced videos, resulting in a markedly clearer and more lifelike output.
• Frame removal script: LipWise includes a script to automatically eliminate frames devoid of detectable facial features. This significantly enhances the overall quality of the generated videos by removing frames in which facial recognition is absent or suboptimal. This ensures that the lip-syncing process concentrates on frames characterized by precise facial data, thereby yielding a smoother and more coherent final video.
• Enhanced face detection algorithm: LipWise boasts an improved face detection algorithm, augmenting the accuracy of face identification and tracking in the reference video. This heightened precision in face detection plays a pivotal role in achieving meticulous lip synchronization, ensuring the model aligns the generated lips accurately with the speaker's movements.

🤗 ACKNOWLEDGEMENTS:

Thanks to the following open-source projects:

• Wav2Lip
• GFPGAN
• libfacedetection

lip_wise_gfpgan's Issues

How can I run this now

Hi, I love this version so much, but it can't work now, because the repo is not available. How can I run it ?

!git clone https://github.com/pawansharmaaaa/Wav2Lip_Mod.git