In this paper, the authors present a novel approach to personalised speech enhancement (PSE) using a streaming Transformer-based model with adaptive target speaker representations. PSE is the process of extracting only the speech of a target user and removing everything else from an audio clip, which can improve users' experiences of audio AI modules in real-world applications such as real-time automatic speech recognition (ASR) and audio-call enhancement. To achieve personalisation, a target speaker's voice profile is typically extracted from an enrolment audio as a static embedding vector and used to condition the output of a PSE model. However, fixed embeddings may not be optimal under all conditions. The proposed model uses cross-attention to dynamically adapt the target speaker representation for each input frame in real time. The authors conducted extensive experiments on two different sizes of Transformer models and compared them with bidirectional VoiceFilter models and bidirectional cross-attention models. They trained their models for 100/200 epochs with batch sizes of 320/640 on 350 hours of training data, following an optimizer with 16k warm-up steps. The training objective was to reduce the power-law compressed spectral distance between cleaned spectrograms produced by their models and ground truth data. The results show that their proposed cross-attention approach consistently outperforms competitive baselines even when their model is only approximately half the size. The authors also demonstrate that their approach achieves better performance than fixed embeddings in scenarios where there are changes in background noise or speakers' voices. Overall, this work presents an innovative solution for improving PSE performance in real world applications by dynamically adapting target speaker representations using cross attention in a streaming Transformer based model. This approach has significant potential for enhancing users' experiences with audio AI modules deployed in various settings, including ASR systems and teleconferencing platforms.
- The paper talks about a new way to make voices clearer in audio recordings using a special computer program.
- This program can take out all the sounds except for one person's voice, which makes it easier to understand what they are saying.
- The program can change how it works depending on who is talking and what noises are in the background.
- They tested this program against other similar ones and found that it worked better even though it was smaller.
- This new approach can help people hear better when using things like speech recognition or talking on the phone.
Definitions
- Personalised speech enhancement (PSE): A process of making voices clearer by removing other sounds from an audio clip, specifically focusing on one person's voice.
- Streaming Transformer-based model: A type of computer program that processes information in real-time and adapts to changes as they happen.
- Adaptive target speaker representations: The specific way the computer program identifies and focuses on one person's voice in an audio recording.
- Cross-attention: A technique used by the computer program to adjust its focus based on changes in the input data.
- Bidirectional VoiceFilter models: Other types of computer programs that also try to remove unwanted sounds from audio recordings.
Personalised Speech Enhancement with Adaptive Target Speaker Representations
In recent years, the use of audio AI modules in real-world applications such as automatic speech recognition (ASR) and teleconferencing has become increasingly popular. To ensure a good user experience, it is important to be able to accurately extract only the speech of a target user from an audio clip while removing everything else. This process is known as personalised speech enhancement (PSE).
In this paper, the authors present a novel approach to PSE using a streaming Transformer-based model with adaptive target speaker representations. The proposed model uses cross-attention to dynamically adapt the target speaker representation for each input frame in real time. This approach has significant potential for enhancing users' experiences with audio AI modules deployed in various settings, including ASR systems and teleconferencing platforms.
Background
To achieve personalisation, a target speaker's voice profile is typically extracted from an enrolment audio as a static embedding vector and used to condition the output of a PSE model. However, fixed embeddings may not be optimal under all conditions - for example when there are changes in background noise or speakers' voices. As such, there is need for models that can dynamically adapt their output based on changing conditions during runtime.
Proposed Model
The authors propose using cross-attention to dynamically adapt the target speaker representation for each input frame in real time within their streaming Transformer-based model. They conducted extensive experiments on two different sizes of Transformer models and compared them with bidirectional VoiceFilter models and bidirectional cross-attention models. They trained their models for 100/200 epochs with batch sizes of 320/640 on 350 hours of training data following an optimizer with 16k warm-up steps. The training objective was to reduce the power-law compressed spectral distance between cleaned spectrograms produced by their models and ground truth data.
Results
The results show that their proposed cross-attention approach consistently outperforms competitive baselines even when their model is only approximately half the size. Furthermore, they demonstrate that their approach achieves better performance than fixed embeddings in scenarios where there are changes in background noise or speakers' voices over time - making it ideal for use cases where dynamic adaptation is necessary during runtime operations such as ASR systems or teleconferencing platforms..
Conclusion
Overall, this work presents an innovative solution for improving PSE performance in real world applications by dynamically adapting target speaker representations using cross attention in a streaming Transformer based model . This could have far reaching implications both within research circles looking into further development of PSE technology but also more broadly across industries deploying audio AI modules which require accurate extraction of specific voices from noisy environments
Personalised speech enhancement (PSE), which extracts only the speech of a target user and removes everything else from a recorded audio clip, can potentially improve users' experiences of audio AI modules deployed in the wild. To support a large variety of downstream audio tasks, such as real-time ASR and audio-call enhancement, a PSE solution should operate in a streaming mode, i.e., input audio cleaning should happen in real-time with a small latency and real-time factor. Personalisation is typically achieved by extracting a target speaker's voice profile from an enrolment audio, in the form of a static embedding vector, and then using it to condition the output of a PSE model. However, a fixed target speaker embedding may not be optimal under all conditions. In this work, we present a streaming Transformer-based PSE model and propose a novel cross-attention approach that gives adaptive target speaker representations. We present extensive experiments and show that our proposed cross-attention approach outperforms competitive baselines consistently, even when our model is only approximately half the size.
