# Continuous Autoregressive Language Models

**Authors:** Chenze Shao, Darren Li, Fandong Meng, Jie Zhou  
**arXiv:** [2510.27688](https://arxiv.org/abs/2510.27688)  
**Date:** 2025-10-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2510.27688v1/

## Key points

- Continuous Autoregressive Language Models (CALM) introduced as a novel paradigm shift in language modeling
- CALM utilizes a high-fidelity autoencoder to compress tokens into continuous vectors, reducing generative steps and improving performance-compute trade-off
- Comprehensive likelihood-free framework supports robust training, evaluation, and controllable sampling in the continuous domain
- CALM achieves comparable performance to strong discrete baselines at lower computational cost through experiments
- Next-vector prediction enhances performance in CALM
- Authors provide access to their code repository on GitHub and invite readers to explore their project further through their dedicated website

## Summary

The authors introduce Continuous Autoregressive Language Models (CALM) as a novel paradigm shift in language modeling. By utilizing a high-fidelity autoencoder to compress tokens into continuous vectors, CALM reduces the number of generative steps required and improves performance-compute trade-off. The comprehensive likelihood-free framework developed by the authors supports robust training, evaluation, and controllable sampling in the continuous domain. Through experiments, they demonstrate that CALM achieves comparable performance to strong discrete baselines at a lower computational cost. and  are enhanced through next-vector prediction in CALM. The authors provide access to their code repository on GitHub and invite readers to explore their project further through their dedicated website.