Direct Speech Translation for Automatic Subtitling

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Uses this paper in depth Not relevant

   SBAAM! Eliminating Transcript Dependency in Automatic Subtitling

   Marco Gaido, Sara Papi, Matteo Negri et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cited 3 times · Not summarized yet

   Abstract

   Subtitling plays a crucial role in enhancing the accessibility of audiovisual content and encompasses three primary subtasks: translating spoken dialogue, segmenting translations into concise textual units, and estimating timestamps that govern their on-screen duration. Past attempts to automate this process rely, to varying degrees, on automatic transcripts, employed diversely for the three subtasks. In response to the acknowledged limitations associated with this reliance on transcripts, recent research has shifted towards transcription-free solutions for translation and segmentation, leaving the direct generation of timestamps as uncharted territory. To fill this gap, we introduce the fir…

2. Very close Not relevant

   Research and development of a subtitle management system using artificial intelligence

   A. Striuk, V. Hordiienko · 2025 · CS&SE@SW · cited 8 times · Not summarized yet

   Abstract

   Subtitles play a vital role in making video content accessible to a wider audience, including individuals with hearing impairments and those who do not understand the spoken language. However, the manual creation of subtitles is a time-consuming and labor-intensive process. This paper proposes an AI-powered subtitle management system that automates the generation and management of subtitles for video content. The system leverages state-of-the-art automatic speech recognition (ASR) and machine translation (MT) technologies to generate accurate and synchronized subtitles in multiple languages. The proposed system architecture consists of a speech recognition module, a machine translation modul…

3. Very close Cited by this paper Not relevant

   Automation in the Intralingual Subtitling Process

   K. Vitikainen, Maarit Koponen · 2021 · Journal of audiovisual translation · cited 6 times · Not summarized yet

   Abstract

   The demand for intralingual subtitles for television and video content is increasing. In Finland, major broadcasting companies are required to provide intralingual subtitles for all or a portion of their programming in Finnish and Swedish, excluding certain live events. To meet this need, technology could offer solutions in the form of automatic speech recognition and subtitle generation. Although fully automatic subtitles may not be of sufficient quality to be accepted by the target audience, they can be a useful tool for the subtitler. This article presents research conducted as part of the MeMAD project, where automatically generated subtitles for Finnish were tested in professional workf…

4. Very close Cited by this paper Not relevant

   Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora

   Sara Papi, Alina Karakanta, Matteo Negri et al. · 2022 · AACL · cited 8 times · Not summarized yet

   Abstract

   Speech translation for subtitling (SubST) is the task of automatically translating speech data into well-formed subtitles by inserting subtitle breaks compliant to specific displaying guidelines. Similar to speech translation (ST), model training requires parallel data comprising audio inputs paired with their textual translations. In SubST, however, the text has to be also annotated with subtitle breaks. So far, this requirement has represented a bottleneck for system development, as confirmed by the dearth of publicly available SubST corpora. To fill this gap, we propose a method to convert existing ST corpora into SubST resources without human intervention. We build a segmenter model that…

5. Very close Not relevant

   Leveraging Broadcast Media Subtitle Transcripts for Automatic Speech Recognition and Subtitling

   Jakob Poncelet, Hugo Van hamme · 2025 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet

   Abstract

   The recent advancement of speech recognition technology has been driven by large-scale datasets and attention-based architectures, but many challenges still remain, especially for low-resource languages and dialects. This paper explores the integration of weakly supervised transcripts from TV subtitles into automatic speech recognition (ASR) systems, aiming to improve both verbatim transcriptions and automatically generated subtitles. To this end, verbatim data and subtitles are regarded as different domains or languages, due to their distinct characteristics. We propose and compare several end-to-end architectures that are designed to jointly model both modalities with separate or shared en…

6. Close Not relevant

   Real-Time Subtitle Generation for Live Videos Using AI and Machine Learning

   Sahil A. Anand, S. Priya, S. Shameem · 2025 · 2025 3rd International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Int · Not summarized yet

   Abstract

   Real-time subtitling of live video streams is still a major challenge because of the difficulties involved in processing spontaneous speech, dealing with multiple speakers, and coping with varying audio conditions with low latency. This research compares the performance of an end-to-end automatic subtitling system that integrates state-of-the-art automatic speech recognition (ASR) with sophisticated natural language processing. Processed 500 hours of live broadcast content in news, sports, and entertainment segments with a hybrid architecture that combined bidirectional LSTM networks and transformer models. Results indicate a 94.2%-word accuracy rate with an average latency of 1.2 seconds, w…

7. Close Not relevant

   From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs

   Alessandro M. Lucca, Francesco Corso, F. Pierri · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Abstract

   Subtitles are essential for video accessibility and audience engagement. Modern Automatic Speech Recognition (ASR) systems, built upon Encoder-Decoder neural network architectures and trained on massive amounts of data, have progressively reduced transcription errors on standard benchmark datasets. However, their performance in real-world production environments, particularly for non-English content like long-form Italian videos, remains largely unexplored. This paper presents a case study on developing a professional subtitling system for an Italian media company. To inform our system design, we evaluated four state-of-the-art ASR models (Whisper Large v2, AssemblyAI Universal, Parakeet TDT…

8. Close Not relevant

   Subtitle Translation as Markup Translation

   Colin Cherry, N. Arivazhagan, D. Padfield et al. · 2021 · Interspeech · cited 9 times · Not summarized yet

   Abstract

   Automatic subtitle translation is an important technology to make video content available across language barriers. Subti-tle translation complicates the normal translation problem by adding the challenge of how to format the system output into subtitles. We propose a simple technique that treats subtitle translation as standard sentence translation plus alignment driven markup transfer, which enables us to reliably maintain timing and formatting information from the source subtitles. We also introduce two metrics to measure the quality of subtitle boundaries: a Timed BLEU that penalizes mistimed tokens with respect to a reference subtitle sequence, and a measure of how much Timed BLEU is lo…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   MuST-Cinema: a Speech-to-Subtitles corpus

   Alina Karakanta, Matteo Negri, M. Turchi · 2020 · International Conference on Language Resources and Evaluation · cited 31 times · Not summarized yet

   Abstract

   Growing needs in localising audiovisual content in multiple languages through subtitles call for the development of automatic solutions for human subtitling. Neural Machine Translation (NMT) can contribute to the automatisation of subtitling, facilitating the work of human subtitlers and reducing turn-around times and related costs. NMT requires high-quality, large, task-specific training data. The existing subtitling corpora, however, are missing both alignments to the source language audio and important information about subtitle breaks. This poses a significant limitation for developing efficient automatic approaches for subtitling, since the length and form of a subtitle directly depends…

2. Close A method it uses Not relevant

   Direct Segmentation Models for Streaming Speech Translation

   Javier Iranzo-Sánchez, Adrià Giménez Pastor, J. Silvestre-Cerdà et al. · 2020 · Conference on Empirical Methods in Natural Language Processing · cited 22 times · Not summarized yet

   Abstract

   The cascade approach to Speech Translation (ST) is based on a pipeline that concatenates an Automatic Speech Recognition (ASR) system followed by a Machine Translation (MT) system. These systems are usually connected by a segmenter that splits the ASR output into hopefully, semantically self-contained chunks to be fed into the MT system. This is specially challenging in the case of streaming ST, where latency requirements must also be taken into account. This work proposes novel segmentation models for streaming ST that incorporate not only textual, but also acoustic information to decide when the ASR output is split into a chunk. An extensive and throughly experimental setup is carried out …

3. Close A method it uses Not relevant

   Machine Translation for Subtitling: A Large-Scale Evaluation

   T. Etchegoyhen, L. Bywood, Mark Fishel et al. · 2014 · International Conference on Language Resources and Evaluation · cited 41 times · Not summarized yet

   Abstract

   This article describes a large-scale evaluation of the use of Statistical Machine Translation for professional subtitling. The work was carried out within the FP7 EU-funded project SUMAT and involved two rounds of evaluation: a quality evaluation and a measure of productivity gain/loss. We present the SMT systems built for the project and the corpora they were trained on, which combine professionally created and crowd-sourced data. Evaluation goals, methodology and results are presented for the eleven translation pairs that were evaluated by professional subtitlers. Overall, a majority of the machine translated subtitles received good quality ratings. The results were also positive in terms …

4. Close Not relevant

   Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?

   Alina Karakanta, Matteo Negri, M. Turchi · 2020 · International Workshop on Spoken Language Translation · cited 36 times · Not summarized yet

   Abstract

   Subtitling is becoming increasingly important for disseminating information, given the enormous amounts of audiovisual content becoming available daily. Although Neural Machine Translation (NMT) can speed up the process of translating audiovisual content, large manual effort is still required for transcribing the source language, and for spotting and segmenting the text into proper subtitles. Creating proper subtitles in terms of timing and segmentation highly depends on information present in the audio (utterance duration, natural pauses). In this work, we explore two methods for applying Speech Translation (ST) to subtitling, a) a direct end-to-end and b) a classical cascade approach. We d…

5. Close* Not relevant

   Machine Translation of TV Subtitles for Large Scale Production

   M. Volk, R. Sennrich, Christian Hardmeier et al. · 2010 · JEC · cited 41 times · Not summarized yet

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

   Sara Papi, Marco Gaido, Matteo Negri · 2023 · International Workshop on Spoken Language Translation · cited 9 times · Not summarized yet

   Abstract

   This paper describes the FBK’s participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one …

2. Close Not relevant

   EVE: Towards End-to-End Video Subtitle Extraction with Vision-Language Models

   Hai-Yang Yu, Meng-Yang Zhao, Jinghui Lu et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet

   Abstract

   Video subtitles play a crucial role in short videos and movies, as they not only help models better understand video content but also support applications such as video translation and content retrieval. Existing video subtitle extraction methods typically rely on multi-stage frameworks, where errors accumulate across stages and temporal dependencies are underutilized due to frame-wise processing. Moreover, although some Large Vision-Language Models (LVLMs) possess strong OCR capabilities, predicting accurate timestamps for subtitle texts remains challenging. To this end, we propose an End-to-end Video subtitle Extraction framework based on LVLMs, named EVE, which can output subtitles and th…

3. Close* Not relevant

   Leveraging broadcast media subtitle transcripts for automatic speech recognition and subtitling

   Jakob Poncelet, H. Van Hamme · 2026 · Journal on Audio, Speech, and Music Processing · cited 1 time · Not summarized yet
4. Close Not relevant

   Automatic Subtitling and Subtitle Compression: FBK at the IWSLT 2024 Subtitling track

   Marco Gaido, Sara Papi, Mauro Cettolo et al. · 2024 · International Workshop on Spoken Language Translation · cited 4 times · Not summarized yet

   Abstract

   The paper describes the FBK submissions to the Subtitling track of the 2024 IWSLT Evaluation Campaign, which covers both the Automatic Subtitling and the Subtitle Compression task for two language pairs: English to German (en-de) and English to Spanish (en-es). For the Automatic Subtitling task, we submitted two systems: i) a direct model, trained in constrained conditions, that produces the SRT files from the audio without intermediate outputs (e.g., transcripts), and ii) a cascade solution that integrates only free-to-use components, either taken off-the-shelf or developed in-house. Results show that, on both language pairs, our direct model outperforms both cascade and direct systems trai…

5. Close Not relevant

   Automatic Sound Effect Classification and Subtitle Generation System Using Pre-Trained Audio Neural Networks With Transfer Learning

   Hyeyoon Jung, Kyeongsu Byun, Joonho Kwon · 2026 · IEEE Access · Not summarized yet

   Abstract

   Recent advances in automatic subtitle generation have primarily focused on speech recognition, whereas subtitle generation for nonverbal sound effects has received comparatively little attention. This limitation hinders comprehensive delivery of audiovisual information. Sound effects—such as gunshots, screams, and crying—are crucial for providing narrative context and conveying emotional nuances in video content. Nevertheless, current subtitle systems frequently omit these elements, relying heavily on manual annotation and underscoring the need for automated approaches. In this paper, we propose a system for the automatic detection, classification, and subtitling of a broad range of nonverba…