The paper "Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development" introduces a human-in-the-loop (HITL) data annotation pipeline that combines the advantages of both humans and machines to generate high-quality, large-scale speech datasets. The pipeline uses machine pre-labeling and fully manual auditing to annotate datasets quickly, accurately, and cost-effectively. Quality control mechanisms such as blind testing, behavior monitoring, and data validation have been adopted in the annotation pipeline to mitigate potential bias introduced by machine-generated labels. The authors conducted A/B testing and pilot studies that demonstrated the HITL pipeline can improve annotation speed and capacity by at least 80%, while maintaining comparable or higher quality than manual double pass annotation. They are leveraging this scalable pipeline to create ultra-high volume off-the-shelf (UHV-OTS) speech corpora for multiple languages with the capability to expand to 10,000+ hours per language annually. Customized datasets can be produced from the UHV-OTS corpora using dynamic packaging. The UHV-OTS is a long term Appen project that aims to support commercial and academic research data needs in speech processing. To support academic and open source community research under the CC BY SA license, Appen will donate a number of free speech datasets from the UHV OTS each year. Additionally, the authors are releasing the code of the data preprocessing and pre tagging pipeline under Apache 2.0 license to allow reproduction of their results. Overall, this paper presents an innovative approach for generating high quality speech datasets at scale using a combination of human expertise and machine learning techniques. The HITL pipeline has significant implications for advancing research in speech processing by providing researchers with access to large volumes of annotated data that can be used for training machine learning models.
The paper talks about a way to make speech datasets using both humans and machines. This helps make the datasets quickly, accurately, and cheaply. They use different methods to check that the data is good quality. They tested this method and found it works well and can make lots of data for different languages. People can use this data for research and Appen will give some away for free. The authors are sharing their code so others can try it too.
Definitions
- Human-in-the-loop (HITL): A process where humans work together with machines to complete a task.
- Data annotation: Adding labels or information to data.
- Machine pre-labeling: Using machines to add labels or information to data before humans check it.
- Quality control mechanisms: Methods used to ensure that the data is accurate and reliable.
- Ultra-high volume off-the-shelf (UHV-OTS): A large amount of pre-made data that people can use without having to create it themselves.
Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development
The advancement of speech processing research has been limited by the lack of access to high quality, large-scale datasets. To address this challenge, Appen has developed a human-in-the-loop (HITL) data annotation pipeline that combines the advantages of both humans and machines to generate high-quality, large-scale speech datasets. This paper presents an overview of the HITL pipeline and its implications for advancing research in speech processing.
Overview of HITL Pipeline
The HITL pipeline uses machine pre-labeling and fully manual auditing to annotate datasets quickly, accurately, and cost effectively. Quality control mechanisms such as blind testing, behavior monitoring, and data validation have been adopted in the annotation pipeline to mitigate potential bias introduced by machine generated labels. The authors conducted A/B testing and pilot studies that demonstrated the HITL pipeline can improve annotation speed and capacity by at least 80%, while maintaining comparable or higher quality than manual double pass annotation.
UHV OTS Corpora
The authors are leveraging this scalable pipeline to create ultra-high volume off-the shelf (UHV OTS) speech corpora for multiple languages with the capability to expand to 10,000+ hours per language annually. Customized datasets can be produced from UHV OTS corpora using dynamic packaging. The UHV OTS is a long term Appen project that aims to support commercial and academic research data needs in speech processing.
Supporting Academic Research
To support academic and open source community research under Creative Commons Attribution Share Alike license (CC BY SA), Appen will donate a number of free speech datasets from UHV OTS each year. Additionally, the authors are releasing code related to their work under Apache 2 license which allows reproduction of their results on Github repository: https://github/appenltd/scalable_data_annotation_pipeline .
Implications for Speech Processing Research
Overall, this paper presents an innovative approach for generating high quality speech datasets at scale using a combination of human expertise and machine learning techniques. The HITL pipeline has significant implications for advancing research in speech processing by providing researchers with access to large volumes of annotated data that can be used for training machine learning models more efficiently than ever before possible without sacrificing accuracy or quality standards set forth by industry professionals..
This paper introduces a human-in-the-loop (HITL) data annotation pipeline to generate high-quality, large-scale speech datasets. The pipeline combines human and machine advantages to more quickly, accurately, and cost-effectively annotate datasets with machine pre-labeling and fully manual auditing. Quality control mechanisms such as blind testing, behavior monitoring, and data validation have been adopted in the annotation pipeline to mitigate potential bias introduced by machine-generated labels. Our A/B testing and pilot results demonstrated the HITL pipeline can improve annotation speed and capacity by at least 80% and quality is comparable to or higher than manual double pass annotation. We are leveraging this scalable pipeline to create and continuously grow ultra-high volume off-the-shelf (UHV-OTS) speech corpora for multiple languages, with the capability to expand to 10,000+ hours per language annually. Customized datasets can be produced from the UHV-OTS corpora using dynamic packaging. UHV-OTS is a long-term Appen project to support commercial and academic research data needs in speech processing. Appen will donate a number of free speech datasets from the UHV-OTS each year to support academic and open source community research under the CC-BY-SA license. We are also releasing the code of the data pre-processing and pre-tagging pipeline under the Apache 2.0 license to allow reproduction of the results reported in the paper.
