StreamVC: Real-Time Low-Latency Voice Conversion

StreamVC is a state-of-the-art streaming voice conversion solution that seamlessly matches the voice timbre of a target speech while preserving the content and prosody of the source speech. Unlike previous approaches, this innovative technology generates the resulting waveform with low latency directly from the input signal, making it ideal for real-time communication scenarios such as calls and video conferencing where voice anonymization may be necessary. Developed based on the architecture and training strategy of the SoundStream neural audio codec, StreamVC boasts lightweight yet high-quality speech synthesis capabilities. A key aspect highlighted in this research is its ability to learn soft speech units causally, effectively preserving pitch stability without compromising source timbre information. To ensure optimal performance, StreamVC utilizes the LibriTTS train-clean-100 subset to derive cluster centroids for HuBERT pseudo-labels, creating a robust training dataset consisting of 555.15 hours of speech from 2311 speakers. The model undergoes extensive training over 1.3 million steps with a batch size of 128. To evaluate its generalization capabilities, an evaluation dataset is created where both source and target speakers are unseen during training. Comprehensive evaluations are conducted on a significant number of pairs of source and target speech utterances selected from various datasets like LibriTTS and VCTK. Baseline models and evaluation metrics are employed to assess the effectiveness and efficiency of StreamVC in comparison to existing methods. The results showcase promising advancements in real-time low-latency voice conversion technology, paving the way for enhanced communication experiences across diverse applications.

StreamVC: Advancing Real-Time Voice Conversion Technology

Voice conversion technology has come a long way in recent years, with various methods and techniques being developed to transform the voice of a speaker into that of another. However, one major challenge faced by existing approaches is maintaining the timbre or quality of the source speech while converting it to match the target speaker's voice. This is where StreamVC comes in – a state-of-the-art streaming voice conversion solution that seamlessly matches the voice timbre of a target speech while preserving the content and prosody of the source speech.

Developed based on the architecture and training strategy of SoundStream neural audio codec, StreamVC boasts lightweight yet high-quality speech synthesis capabilities. Unlike previous approaches, this innovative technology generates the resulting waveform with low latency directly from the input signal, making it ideal for real-time communication scenarios such as calls and video conferencing where voice anonymization may be necessary.

Key Features:

- Low Latency: One of StreamVC's key features is its ability to generate waveforms with low latency directly from input signals. This makes it highly suitable for real-time communication scenarios where quick processing is essential. - Timbre Preservation: Unlike other methods that often compromise on source timbre information during conversion, StreamVC effectively preserves pitch stability without compromising on timbre. - Causal Learning: The model learns soft speech units causally, which helps maintain pitch stability and ensures optimal performance during conversions. - Robust Training Dataset: To ensure high-quality results, StreamVC utilizes 555.15 hours of speech data from 2311 speakers derived from LibriTTS train-clean-100 subset to derive cluster centroids for HuBERT pseudo-labels. - Extensive Training: The model undergoes extensive training over 1.3 million steps with a batch size of 128 to achieve optimal performance.

Evaluation Process:

To evaluate its generalization capabilities, an evaluation dataset is created where both source and target speakers are unseen during training. Comprehensive evaluations are conducted on a significant number of pairs of source and target speech utterances selected from various datasets like LibriTTS and VCTK. Baseline models and evaluation metrics are employed to assess the effectiveness and efficiency of StreamVC in comparison to existing methods.

Results:

The results showcase promising advancements in real-time low-latency voice conversion technology, paving the way for enhanced communication experiences across diverse applications. StreamVC outperforms baseline models in terms of both objective metrics such as Mel Cepstral Distortion (MCD) and subjective evaluations by human listeners.

Conclusion:

StreamVC is a groundbreaking technology that addresses key challenges faced by existing voice conversion methods, such as maintaining timbre quality while converting voices. Its low latency, causal learning approach, and robust training dataset make it highly efficient and effective for real-time communication scenarios where quick processing is crucial. With its promising results, StreamVC has the potential to revolutionize voice conversion technology and enhance communication experiences across various applications.