# A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

**Authors:** K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C V Jawahar  
**arXiv:** [2008.10010](https://arxiv.org/abs/2008.10010)  
**Date:** 2020-08-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2008.10010v1/

## Key points

- Authors K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and C V Jawahar address the challenge of synchronizing lip movements in talking face videos with target speech segments.
- Previous methods struggle to blend lip movements of diverse identities in dynamic videos, leading to out-of-sync issues.
- Researchers propose a solution using insights from a robust lip-sync discriminator to improve synchronization.
- Novel evaluation benchmarks and metrics are introduced for assessing lip synchronization in unconstrained videos.
- Extensive quantitative evaluations show that the Wav2Lip model achieves near-indistinguishable accuracy from real synced videos.
- A demo video on their website showcases the impact of the Wav2Lip model and evaluation benchmarks.
- Interested individuals can access code and models on their GitHub repository and explore an interactive demo through a provided link.

## Summary

In their research paper "A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild," authors K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and C V Jawahar tackle the complex problem of accurately synchronizing lip movements in talking face videos with target speech segments. While previous methods have been successful in generating precise lip movements on specific individuals seen during training, they struggle to seamlessly blend the lip movements of diverse identities in dynamic and unconstrained videos. This often results in significant parts of the video being out-of-sync with the new audio. To overcome this challenge, the researchers identify key factors contributing to this issue and propose a solution by utilizing insights from a robust lip-sync discriminator. They also introduce novel evaluation benchmarks and metrics designed to precisely assess lip synchronization in unconstrained videos. Through extensive quantitative evaluations on these rigorous benchmarks, the team demonstrates that their Wav2Lip model achieves near-indistinguishable accuracy from real synced videos. Additionally, they provide a demo video showcasing the significant impact of their Wav2Lip model and evaluation benchmarks on their website. Interested individuals can access the code and models at their GitHub repository and explore an interactive demo through a provided link. This comprehensive study not only advances the field of speech-to-lip generation but also sets a high standard for evaluating and achieving realistic lip synchronization in challenging real-world scenarios.