In recent years, there has been a significant surge in the use of self-supervised learning (SSL) across various domains such as language, vision, speech, and audio. One of the main challenges faced in implementing semantic-rich acoustic tokenizers for general audio pre-training lies in the continuous nature of audio data and the unavailability of phoneme sequences like speech. To address this challenge head-on, a novel iterative audio pre-training framework called BEATs has been proposed. This framework aims to learn Bidirectional Encoder representations from Audio Transformers by optimizing an acoustic tokenizer and an audio SSL model through iterative processes. has shown to be more effective in encouraging SSL models to abstract high-level audio semantics while discarding redundant details akin to human perception. The authors behind this groundbreaking work include Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu,
Daniel Tompkins,
Zhuo Chen,
and Furu Wei. In the initial iteration of BEATs, random projection is employed as the acoustic tokenizer to train an audio SSL model using a mask and label prediction approach. Subsequently, an acoustic tokenizer is trained for the next iteration by distilling semantic knowledge from either the pre-trained or fine-tuned audio SSL model. This iterative process continues with the expectation of mutual enhancement between the acoustic tokenizer and the audio SSL model. Experimental results have demonstrated that BEATs' acoustic tokenizers are capable of generating discrete labels with rich audio semantics. Furthermore,has been achieved by , surpassing previous approaches that utilized larger amounts of training data and model parameters significantly. Notably, has set a new benchmark with a state-of-the-art mean average precision (mAP) of 50.6% on AudioSet-2M without requiring any external data sources. Additionally, it has achieved an accuracy rate of 98.1% on ESC-50 classification tasks. For those interested in exploring further or utilizing these advancements, both code implementations and pre-trained models are readily accessible at https://aka.ms/beats.
Summary- Self-supervised learning (SSL) is being used more in different areas like language, vision, speech, and audio.
- A new method called BEATs helps improve how computers understand sound by training them with special tools.
- BEATs helps computers focus on important sound details and ignore less important ones, similar to how people listen.
- The creators of BEATs are Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei.
- BEATs has done very well in tests for understanding sound and classifying it correctly.
Definitions
- Self-supervised learning (SSL): A way for computers to learn from data without human labeling or supervision.
- Semantic: Related to meaning or understanding.
- Acoustic: Relating to sound or the sense of hearing.
- Tokenizers: Tools that break down data into smaller parts for analysis or processing.
- Iterative: Involving a process that repeats steps over and over to improve results.
Introduction
In recent years, there has been a significant surge in the use of self-supervised learning (SSL) across various domains such as language, vision, speech, and audio. One of the main challenges faced in implementing semantic-rich acoustic tokenizers for general audio pre-training lies in the continuous nature of audio data and the unavailability of phoneme sequences like speech.
To address this challenge head-on, a novel iterative audio pre-training framework called BEATs has been proposed by a team of researchers including Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu,
Daniel Tompkins,
Zhuo Chen,
and Furu Wei. This groundbreaking work aims to learn Bidirectional Encoder representations from Audio Transformers by optimizing an acoustic tokenizer and an audio SSL model through iterative processes.
The BEATs Framework
The BEATs framework is designed to encourage SSL models to abstract high-level audio semantics while discarding redundant details akin to human perception. It achieves this through an iterative process that involves training both an acoustic tokenizer and an audio SSL model simultaneously.
In the initial iteration of BEATs, random projection is employed as the acoustic tokenizer to train an audio SSL model using a mask and label prediction approach. Subsequently, an acoustic tokenizer is trained for the next iteration by distilling semantic knowledge from either the pre-trained or fine-tuned audio SSL model. This iterative process continues with the expectation of mutual enhancement between the acoustic tokenizer and the audio SSL model.
Results
Experimental results have demonstrated that BEATs' acoustic tokenizers are capable of generating discrete labels with rich audio semantics. Furthermore, has been achieved by , surpassing previous approaches that utilized larger amounts of training data and model parameters significantly.
Notably, has set a new benchmark with a state-of-the-art mean average precision (mAP) of 50.6% on AudioSet-2M without requiring any external data sources. Additionally, it has achieved an accuracy rate of 98.1% on ESC-50 classification tasks.
Availability
For those interested in exploring further or utilizing these advancements, both code implementations and pre-trained models are readily accessible at https://aka.ms/beats.
Conclusion
In conclusion, the BEATs framework proposed by Sanyuan Chen and their team has shown to be a highly effective approach for audio pre-training through its iterative process of optimizing acoustic tokenizers and audio SSL models. Its results have surpassed previous approaches and set new benchmarks in the field of self-supervised learning for audio data. With its availability for use by others, we can expect to see even more advancements in this area as researchers continue to build upon this groundbreaking work.
The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.
