# M+: Extending MemoryLLM with Scalable Long-Term Memory

**Authors:** Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, Zexue He  
**arXiv:** [2502.00592](https://arxiv.org/abs/2502.00592)  
**Date:** 2025-02-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.00592v1/

## Key points

- Introduction of M+, a memory-augmented model for enhancing long-term information retention in large language models (LLMs)
- Three stages in the training process: Continual Training of MemoryLLM, Long-Context Modeling with Long Documents, and Training with long-term memory
- Utilization of backbone model Llama-3.1-8B with memory tokens in each layer
- Training on short documents from the fineweb-edu dataset followed by longer documents to improve long-context modeling abilities
- Integration of long-term memory to enhance M+ further
- Experimental results showing M+ outperforming MemoryLLM and other baselines by extending knowledge retention capabilities significantly
- Evaluation on various benchmarks for long-context understanding and knowledge retention tasks, demonstrating superior performance
- Future work aimed at reducing CPU-GPU communication overhead for more efficient generation with M+
- Impact on education, research, and industry as well as concerns about AI safety, reliability, fairness, bias propagation, and ethical considerations

## Summary

In this work, we introduce M+, a memory-augmented model that enhances long-term information retention in large language models (LLMs). The training process consists of three stages: Continual Training of MemoryLLM, Long-Context Modeling with Long Documents, and Training with long-term memory. We start with the backbone model Llama-3.1-8B equipped with memory tokens in each layer and train it on short documents from the fineweb-edu dataset. Subsequently, we train on longer documents ranging from 4k to 64k tokens to improve long-context modeling abilities. Finally, we introduce long-term memory to enhance M+ further. Experimental results show that M+ outperforms MemoryLLM and other strong baselines by significantly extending knowledge retention capabilities from under 20k to over 160k tokens while maintaining similar GPU memory overhead. In addition, M+ is evaluated on various benchmarks for long-context understanding and knowledge retention tasks, demonstrating superior performance. In conclusion, M+ offers enhanced long-term retention abilities for LLMs by integrating a long-term memory mechanism with a co-trained retriever. Future work aims to reduce CPU-GPU communication overhead for more efficient generation with M+. The impact of this work extends to areas such as education, research, and industry but also raises concerns about AI safety, reliability, and fairness if not carefully managed. Further analysis is needed to understand the implications of increased memory capacity in LLMs on bias propagation and other ethical considerations.