# LLaMA-Omni: Seamless Speech Interaction with Large Language Models

**Authors:** Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, Yang Feng  
**arXiv:** [2409.06666](https://arxiv.org/abs/2409.06666)  
**Date:** 2024-09-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.06666v1/

## Key points

- Authors introduce LLaMA-Omni, a model architecture for speech interaction with large language models (LLMs)
- LLaMA-Omni aims to improve user experience by enabling real-time speech interaction with LLMs
- The model integrates components such as a pretrained speech encoder, a speech adaptor, an LLM core, and a streaming speech decoder
- LLaMA-Omni eliminates the need for speech transcription and enables simultaneous generation of text and speech responses directly from spoken instructions with low latency
- Authors curated InstructS2S-200K dataset to align the model with practical speech interaction scenarios
- Experimental results show that LLaMA-Omni outperforms previous models in response quality and achieves low response latency of 226ms
- Training the model requires less than three days on four GPUs, highlighting its efficiency and potential for rapid development
- Introduction of LLaMA-Omni represents a significant advancement in enhancing user experiences through seamless and efficient speech interactions with large language models

## Summary

In their paper titled "LLaMA-Omni: Seamless Speech Interaction with Large Language Models," authors Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng introduce a novel model architecture designed to revolutionize speech interaction with large language models (LLMs). The existing landscape of models like GPT-4o has shown the potential for real-time interaction with LLMs through speech, offering a significant improvement in user experience compared to traditional text-based interactions. However, there remains a gap in exploring how to effectively build speech interaction models based on open-source LLMs. To bridge this gap, the authors propose LLaMA-Omni as an innovative solution tailored for low-latency and high-quality speech interaction with LLMs. is a cutting-edge model that integrates various components including a pretrained speech encoder, a speech adaptor, an LLM core, and a streaming speech decoder. This model eliminates the need for speech transcription by enabling simultaneous generation of text and speech responses directly from spoken instructions with remarkably low latency. Building upon the latest advancements in the field such as the  model, the authors align their model with practical speech interaction scenarios by curating a dataset named InstructS2S-200K. This dataset comprises 200K pairs of speech instructions and corresponding speech responses to facilitate training and evaluation processes. Experimental results showcase that  outperforms previous  in terms of response quality both in content and style while achieving an impressive response latency as low as 226ms. Furthermore, training this model requires less than three days on just four GPUs, underscoring its efficiency and potential for rapid development of future . Overall, the introduction of  represents a significant advancement in enhancing user experiences through seamless and efficient speech interactions with large language models. The authors' work paves the way for further exploration and development in this domain while setting new standards for real-time communication interfaces powered by cutting-edge technology.