# Speed Always Wins: A Survey on Efficient Architectures for Large Language Models

**Authors:** Weigao Sun, Jiaxi Hu, Yucheng Zhou, Jusen Du, Disen Lan, Kexin Wang, Tong Zhu, Xiaoye Qu, Yu Zhang, Xiaoyu Mo, Daizong Liu, Yuxuan Liang, Wenliang Chen, Guoqi Li, Yu Cheng  
**arXiv:** [2508.09834](https://arxiv.org/abs/2508.09834)  
**Date:** 2025-08-13  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2508.09834v1/

## Key points

- Large Language Models (LLMs) have revolutionized language understanding, generation, reasoning, and multimodal models.
- The foundation of modern LLMs lies in Transformer models with excellent scaling properties but substantial computational requirements.
- Researchers are exploring innovative architectural designs and optimization strategies to address challenges and unlock the full potential of LLMs.
- Efficient architectures for LLMs include Linear Sequence Modeling, Sparse Sequence Modeling, Efficient Full Attention, Sparse Mixture of Experts, Hybrid Architectures, and Diffusion LLMs.
- These architectural principles can be adapted beyond language tasks to other modalities like vision, audio, and multi-modality applications.
- By combining different components and leveraging techniques like sparse modeling and mixture-of-experts approaches, researchers aim to develop scalable and resource-aware foundation models for AI systems.

## Summary

Large Language Models (LLMs) have revolutionized language understanding, generation, reasoning, and multimodal models. The foundation of modern LLMs lies in Transformer models, which offer excellent scaling properties but come with substantial computational requirements that hinder large-scale training and practical deployment. To address these challenges and unlock the full potential of LLMs, researchers have been exploring innovative architectural designs and optimization strategies. This survey delves into various efficient architectures for LLMs, categorizing them into distinct methods to provide a comprehensive overview. These methods include Linear Sequence Modeling, which aims to reduce the quadratic complexity of self-attention to linear complexity by reformulating the attention mechanism; Sparse Sequence Modeling, which selectively focuses on a subset of interactions to reduce computational and memory requirements; Efficient Full Attention, which enhances standard softmax attention's efficiency while retaining theoretical quadratic complexity; Sparse Mixture of Experts, introducing conditional computation for increased model capacity without proportional computational cost; Hybrid Architectures combining linear sequence modeling with traditional full attention layers; and Diffusion LLMs exploring non-autoregressive diffusion models for language generation. Furthermore, the survey highlights the adaptability of these architectural principles beyond language tasks to other modalities such as vision, audio, and multi-modality applications. By strategically combining different components and leveraging innovative techniques like sparse modeling and mixture-of-experts approaches, researchers aim to develop scalable and resource-aware foundation models that are efficient yet versatile in various AI systems. The core architecture behind recent breakthroughs in LLMs is the Transformer model with its self-attention mechanism enabling effective capture of long-range dependencies. However, the quadratic complexity of this mechanism poses challenges in terms of computational efficiency for tasks involving long-context inputs. <kd> Keywords: Large Language Models (LLMs), Transformer models,
Efficient Architectures,
Scalability,
Resource-aware </kd>

Through systematic examination and categorization of efficient architectures for LLMs, this survey presents a blueprint for modern advancements in AI systems towards more efficient and high-quality text synthesis while considering implications for future research directions.