Search Results for « VoiceCraft »

We introduce VOICECRAFT, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts 1 .VOICECRAFT employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable generation within an existing sequence.On speech editing tasks, VOICECRAFT produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SotA models including VALL-E and the popular commercial model XTTS v2.Crucially, the models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings.In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named REALEDIT.We encourage readers to listen to the demos at https: //jasonppy.github.io/VoiceCraft_web.

Voicecraft workshop aims to establish a research community focused on the design and evaluation of Voice Assistant (VA) personas for both task-oriented functions (e.g., information search, online shopping) and personal growth applications (e.g., coaching, mindful reflection, tutoring). Through discussion and collaborative efforts, we will seek to propose a set of practices and standards that will help to improve the ecological validity of VA personas. In particular, we will explore topics such as the interaction design of voice-based interfaces, the impact of agent personas on the user experience, and the approaches for designing such VA agents. This workshop will serve as a platform to build a better-equipped community to explore VA personas that provide a better fit to range of everyday interaction scenarios.

This dataset contains the manual validation of an LLM-based speaker-attribution pipeline that assigns every dialogue line in a film's subtitle track to the character who utters it, developed for research on gendered speaking time in cinema. It covers five films in five languages: C'è ancora domani (Italian, 2023), Furiosa: A Mad Max Saga (English, 2024), Das Lehrerzimmer (German, 2023), Le Comte de Monte-Cristo (French, 2024) and La infiltrada (Spanish, 2024), for a total of 7,415 dialogue lines and about 3 hours 54 minutes of speech. Each film comes as one xlsx file with two sheets. The lines sheet has one row per speaking turn with four core columns: line_id (the cue number in the source subtitle file), timestamp, predicted_speaker (the pipeline's output) and true_speaker (the researcher's manual coding, done character by character against the film); where present, a notes column records validation corrections, i.e. turns found to contain words by a second speaker, split pro-rata by word count, with the corrected fragment quoted. The coding notes sheet documents, for each film, the subtitle source, the unit of analysis, the exclusions (non-dialogue cues), the evidence used for attribution, and the centrality coding: named characters follow Lauzen's major-character criterion (more than one scene and instrumental to the narrative), while all other speakers are pooled into minor character (male) / minor character (female). Character relabels (episodic named characters pooled into the minor buckets under the researcher's supervision) are documented in the coding notes of each film. The subtitle text is not redistributed for copyright reasons: rows link back to the publicly available subtitle tracks (OpenSubtitles ids in the coding notes) via line_id and timestamp. A companion Jupyter notebook reproduces all reported figures: per-film confusion matrices weighted by speaking time and a summary figure with duration-weighted accuracy per film (from 88.0% to 96.5%; pooled: 90.3%), gender attribution (pooled: 96.6%) and the Gini bias of named-character speaking-time concentration (pipeline − manual, within ±0.02 for all films and genders). The dataset supports the reuse of subtitle-based speaker attribution in research on character speech distribution, and provides a benchmark for text-only attribution methods in five languages.

We present VoiceCraft-Dub, a novel approach for automated video dubbing that synthesizes high-quality speech from text and facial cues. This task has broad applications in filmmaking, multimedia creation, and assisting voice-impaired individuals. Building on the success of Neural Codec Language Models (NCLMs) for speech synthesis, our method extends their capabilities by incorporating video features, ensuring that synthesized speech is time-synchronized and expressively aligned with facial movements while preserving natural prosody. To inject visual cues, we design adapters to align facial features with the NCLM token space and introduce audio-visual fusion layers to merge audio-visual information within the NCLM framework. Additionally, we curate CelebV-Dub, a new dataset of expressive, real-world videos specifically designed for automated video dubbing. Extensive experiments show that our model achieves high-quality, intelligible, and natural speech synthesis with accurate lip synchronization, outperforming existing methods in human perception and performing favorably in objective evaluations. We also adapt VoiceCraft-Dub for the video-to-speech task, demonstrating its versatility for various applications.

We introduce VoiceCraft-X, an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot Text-to-Speech (TTS) synthesis across 11 languages: English, Mandarin, Korean, Japanese, Spanish, French, German, Dutch, Italian, Portuguese, and Polish. VoiceCraft-X utilizes the Qwen3 large language model for phoneme-free cross-lingual text processing and a novel token reordering mechanism with time-aligned text and speech tokens to handle both tasks as a single sequence generation problem. The model generates high-quality, natural-sounding speech, seamlessly creating new audio or editing existing recordings within one framework. VoiceCraft-X shows robust performance in diverse linguistic settings, even with limited per-language data, underscoring the power of unified autoregressive approaches for advancing complex, real-world multilingual speech applications. Audio samples are available at https://zhishengzheng.com/voicecraft-x/.

We introduce VoiceCraft-X, an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot text-to-speech (TTS) synthesis across 11 languages: English, Mandarin, Korean, Japanese, Spanish, French, German, Dutch, Italian, Portuguese, and Polish. VoiceCraft-X utilizes the Qwen3 large language model for phoneme-free cross-lingual text processing and a novel token reordering mechanism with time-aligned text and speech tokens to handle both tasks as a single sequence generation problem. The model generates high-quality, natural-sounding speech, seamlessly creating new audio or editing existing recordings within one framework. VoiceCraft-X shows robust performance in diverse linguistic settings, even with limited per-language data, underscoring the power of unified autoregressive approaches for advancing complex, real-world multilingual speech applications. Audio samples are available at https://voicecraft-x.github.io/.

This paper introduces PhoAudiobook, a newly curated dataset comprising 941 hours of highquality audio for Vietnamese text-to-speech.Using PhoAudiobook, we conduct experiments on three leading zero-shot TTS models: VALL-E, VoiceCraft, and XTTS-V2.Our findings demonstrate that PhoAudiobook consistently enhances model performance across various metrics.Moreover, VALL-E and VoiceCraft exhibit superior performance in synthesizing short sentences, highlighting their robustness in handling diverse linguistic contexts.We publicly release PhoAudiobook to facilitate further research and development in Vietnamese textto-speech.

We introduce MAVE (Mamba with Cross-Attention for Voice Editing and Synthesis), a novel autoregressive architecture for text-conditioned voice editing and high-fidelity text-to-speech (TTS) synthesis, built on a cross-attentive Mamba backbone. MAVE achieves state-of-the-art performance in speech editing and very competitive results in zero-shot TTS, while not being explicitly trained on the latter task, outperforming leading autoregressive and diffusion models on diverse, real-world audio. By integrating Mamba for efficient audio sequence modeling with cross-attention for precise text-acoustic alignment, MAVE enables context-aware voice editing with exceptional naturalness and speaker consistency. In pairwise human evaluations on a random 40-sample subset of the RealEdit benchmark (400 judgments), 57.2% of listeners rated MAVE - edited speech as perceptually equal to the original, while 24.8% prefered the original and 18.0% MAVE - demonstrating that in the majority of cases edits are indistinguishable from the source. MAVE compares favorably with VoiceCraft and FluentSpeech both on pairwise comparisons and standalone mean opinion score (MOS) evaluations. For zero-shot TTS, MAVE exceeds VoiceCraft in both speaker similarity and naturalness, without requiring multiple inference runs or post-processing. Remarkably, these quality gains come with a significantly lower memory cost and approximately the same latency: MAVE requires ~6x less memory than VoiceCraft during inference on utterances from the RealEdit database (mean duration: 6.21s, A100, FP16, batch size 1). Our results demonstrate that MAVE establishes a new standard for flexible, high-fidelity voice editing and synthesis through the synergistic integration of structured state-space modeling and cross-modal attention.

In this paper, we propose reverse inference optimization (RIO), a simple and effective method designed to enhance the robustness of autoregressive-model-based zero-shot text-to-speech (TTS) systems using reinforcement learning from human feedback (RLHF). To assess the quality of speech produced by the TTS system without human annotations, RIO introduces a novel concept termed as reverse inference based on the Bayesian principle, which suggests that a high-quality generated speech should be able to be used as a prompt for subsequent generation using the same TTS model. By leveraging reverse inference as the standard to select exemplars used in RLHF from the speech samples generated by the TTS system itself, RIO steers the subsequent optimization towards a direction of enhancing the TTS robustness. The RIO framework, comprising sampling, automatic annotating, and learning, obviates the need for a reward model or pairwise preference data, and significantly improves the stability of zero-shot TTS performance by reducing the discrepancies between training and inference conditions. Our experimental results verify that RIO can effectively improve both subjective and objective metrics, including mean opinion scores, word error rates, and speaker similarity. Remarkably, RIO can also diminish the incidence of bad outputs to nearly zero percent, rivalling the robustness when using ground-truth speech as the prompt.