# Repeat After Me: Transformers are Better than State Space Models at Copying

**Authors:** Samy Jelassi, David Brandfonbrener, Sham M. Kakade, Eran Malach  
**arXiv:** [2402.01032](https://arxiv.org/abs/2402.01032)  
**Date:** 2024-02-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.01032v1/

## Key points

- GSSMs (generalized state space models) compared to transformer models in copying tasks from input context
- Theoretical analysis shows that a two-layer transformer can copy strings of exponential length, while GSSMs are limited by fixed-size latent state
- Empirical experiments confirm that transformers outperform GSSMs in efficiency and generalization on synthetic tasks requiring copying from context
- Pretrained large language models also show significant superiority of transformer models over state space models in copying and retrieving information from context
- Transformers demonstrate superior performance and efficiency compared to GSSMs in tasks involving copying from input context

## Summary

This paper explores the performance of "generalized state space models" (GSSMs) compared to transformer models in tasks involving copying from input context. While GSSMs are appealing for their inference-time efficiency, the authors demonstrate that they have limitations compared to transformers. The authors begin by conducting a theoretical analysis on the task of string copying and prove that a two-layer transformer can copy strings of exponential length. This is in contrast to GSSMs which are fundamentally limited by their fixed-size latent state. This theoretical result highlights the advantage of transformers in handling tasks that require copying from context. To validate their findings empirically, experiments are performed on synthetic tasks requiring copying from context. The results show that transformers outperform GSSMs in terms of efficiency and generalization on these tasks, further supporting their superiority for such tasks. Additionally, pretrained large language models are evaluated and it is found that transformer models significantly outperform state space models at both copying and retrieving information from context. These findings suggest a fundamental gap between transformers and GSSMs in practical applications. Overall, this study provides insights into the limitations of GSSMs compared to transformer models when it comes to tasks requiring copying from input context. The results highlight the superior performance and efficiency of transformers in such scenarios, emphasizing their dominance as the preferred architecture for sequence modeling.