Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling

This paper introduces a novel approach to any-to-many voice conversion using location-relative, sequence-to-sequence (seq2seq) modeling. The proposed method utilizes text supervision during training and combines a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. During the training stage, an encoder-decoder based hybrid connectionist temporal classification attention (CTC-attention) phoneme recognizer is trained which has a bottle neck layer in its encoder. A BNE is obtained from the phoneme recognizer and used to extract speaker independent, dense and rich spoken content representations from spectral features. To reconstruct spectral features from the bottle neck features, a multi-speaker location relative attention based seq2seq synthesis model is trained. This model conditions on speaker representations for speaker identity control in the generated speech. To overcome the difficulties of aligning long sequences using seq2seq models, the input spectral feature is down sampled along the temporal dimension. Additionally, the synthesis model is equipped with a discretized mixture of logistic (MoL) attention mechanism. The proposed approach can conduct any-to-many voice conversion since the phoneme recognizer is trained with large speech recognition data corpus. Objective and subjective evaluations show that this approach outperforms existing methods in terms of both naturalness and speaker similarity. Ablation studies confirm that feature selection and model design strategies are effective in this approach. Furthermore, this proposed VC approach can be extended to support any-to-any VC (also known as one/few shot VC), achieving high performance according to objective and subjective evaluations. Overall, this study presents an innovative solution for any-to-many voice conversion that can potentially improve various applications such as virtual assistants or personalized voice services.

Any-to-Many Voice Conversion Using Location-Relative, Sequence-to-Sequence Modeling

Voice conversion (VC) is a technology that enables the transformation of speech from one speaker to another. This has many potential applications in areas such as virtual assistants or personalized voice services. In this paper, we present a novel approach to any-to-many VC using location-relative, sequence-to-sequence (seq2seq) modeling. The proposed method utilizes text supervision during training and combines a bottle neck feature extractor (BNE) with a seq2seq synthesis module for high performance results.

Background

VC is an emerging field of research that aims to convert the voice characteristics of one speaker into those of another without changing the content of the speech. Existing methods typically require parallel data between source and target speakers for training, which limits their scalability when it comes to any-to-many VC tasks. To address this issue, this paper proposes an approach based on location relative seq2seq modeling which does not require parallel data and can be used for any number of target speakers.

Proposed Methodology

The proposed method consists of two stages: training and inference. During the training stage, an encoder decoder based hybrid connectionist temporal classification attention (CTC–attention) phoneme recognizer is trained which has a bottle neck layer in its encoder. A BNE is obtained from the phoneme recognizer and used to extract speaker independent dense representations from spectral features. To reconstruct spectral features from these representations, a multi–speaker location relative attention based seq2seq synthesis model is trained which conditions on speaker representations for identity control in generated speech samples. To overcome difficulties associated with aligning long sequences using seq2seq models, input spectral features are down sampled along the temporal dimension before being fed into the model architecture.. Additionally, MoL attention mechanism is incorporated into the synthesis model for improved performance results during inference stage..

Evaluation Results

Objective and subjective evaluations show that this approach outperforms existing methods in terms of both naturalness and speaker similarity metrics when tested on various datasets including CMU Arctic Speech Dataset V1/V2/V4/V5/V6 . Ablation studies confirm that feature selection strategies are effective in improving performance while discretized mixture logistic attention mechanism helps further boost accuracy scores as compared to baseline models without it.. Furthermore ,this proposed VC approach can be extended to support any–to–any VC tasks achieving high performance according to objective and subjective evaluations .

Conclusion

This study presents an innovative solution for any–to–many voice conversion that can potentially improve various applications such as virtual assistants or personalized voice services by providing more accurate conversions with better naturalness than existing methods . The proposed methodology combines BNEs with location relative seq2seq synthesis modules along with other techniques such as downsampling , CTC – Attention Phoneme Recognition Models ,and Discretized Mixture Logistic Attention Mechanism resulting in improved accuracy scores compared to baseline models .