# Training LLMs over Neurally Compressed Text

**Authors:** Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant  
**arXiv:** [2404.03626](https://arxiv.org/abs/2404.03626)  
**Date:** 2024-04-04  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2404.03626v1/

## Key points

- The paper explores training large language models (LLMs) using highly compressed text.
- Authors investigate benefits of training LLMs on neurally compressed text, including improved efficiency in training and serving processes and enhanced handling of long text spans.
- One major challenge is that strong compression can hinder effective learning from the data.
- Authors introduce Equal-Info Windows, a novel compression technique that segments text into blocks with uniform bit-length compression.
- Results show successful learning over neurally compressed text and improvements in performance compared to traditional subword tokenizers as scale increases.
- Comparative evaluations against byte-level baselines reveal that Equal-Info Windows outperforms them significantly on perplexity and inference speed benchmarks.
- While it may have slightly worse perplexity compared to subword tokenizers for models trained with equivalent parameters, it offers the advantage of shorter sequence lengths, leading to fewer autoregressive generation steps and decreased latency during model inference.
- The paper includes an extensive analysis of factors contributing to learnability and provides practical recommendations for enhancing the performance of high-compression tokenizers.

## Summary

The paper "Training LLMs over Neurally Compressed Text" explores the concept of training large language models (LLMs) using highly compressed text. The authors - Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein and Noah Constant - investigate the potential benefits of directly training LLMs on neurally compressed text. This includes improved efficiency in both training and serving processes as well as enhanced handling of long text spans. However, one major challenge in this approach is that strong compression can make it difficult for models to effectively learn from the data. To address this issue, the authors introduce a novel compression technique called Equal-Info Windows. This method segments text into blocks with uniform bit-length compression and has shown successful learning over neurally compressed text. The results demonstrate improvements in performance as scale increases compared to traditional subword tokenizers. Comparative evaluations against byte-level baselines also reveal that Equal-Info Windows outperforms them significantly on perplexity and inference speed benchmarks. While it may deliver slightly worse perplexity compared to subword tokenizers for models trained with equivalent parameters, it offers the advantage of shorter sequence lengths. This reduction translates to fewer autoregressive generation steps and decreased latency during model inference. The paper also includes an extensive analysis of factors contributing to learnability and provides practical recommendations for enhancing the performance of high-compression tokenizers. Overall, the study sheds light on the potential benefits and challenges associated with training LLMs over neurally compressed text and presents a promising solution in Equal-Info Windows for improving model efficiency and effectiveness in processing highly compressed textual data.