# Titans: Learning to Memorize at Test Time

**Authors:** Ali Behrouz, Peilin Zhong, Vahab Mirrokni  
**arXiv:** [2501.00663](https://arxiv.org/abs/2501.00663)  
**Date:** 2024-12-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.00663v1/

## Key points

- Novel approach to designing a long-term neural memory module
- Prioritization of events that violate expectations for better memorization
- Introduction of a decaying mechanism to manage limited memory capacity
- Optimization of meta neural network with mini-batch gradient descent, momentum, and weight decay
- Titans architecture with three hyper-heads: Core, Long-term Memory, and Persistent Memory
- Three variants of Titans incorporating memory as context, layer, or gated branch
- Experimental evaluations demonstrating superiority over modern recurrent models and hybrid variants
- Outperformance of Transformers in various tasks and context window sizes
- Training procedures involving LLama 2 tokenizer, AdamW optimizer, cosine annealing schedule, batch size, and weight decay settings
- Superior performance of the neural memory module in Titans compared to other models including Transformer++

## Summary

This paper presents a novel approach to designing a long-term neural memory module that can effectively learn to memorize at test time and incorporate it into an architecture. Inspired by human long-term memory systems, the neural memory module prioritizes events that violate expectations, making them more memorable. A decaying mechanism is introduced to better manage limited memory capacity by considering the proportion of memory size and data surprise. This mechanism optimizes a meta neural network with mini-batch gradient descent, momentum, and weight decay. The Titans architecture consists of three hyper-heads: Core for short-term memory processing using attention with a limited window size, Long-term Memory for storing long past information, and Persistent Memory for encoding task-independent knowledge. Three variants of Titans incorporate memory as context, layer, or gated branch. Experimental evaluations on various tasks demonstrate the superiority of Titans over modern recurrent models and hybrid variants. They also outperform Transformers with the same context window size and show competitive performance with Transformers using the entire context while scaling effectively to larger than 2M context window sizes. The training procedures involve using LLama 2 tokenizer with a vocabulary size of 32K and training length of 4K tokens. AdamW optimizer with a learning rate of 4𝑒-4 and cosine annealing schedule are employed with a batch size of 0.5M tokens and weight decay of 0.1. In language modeling tasks focusing on perplexity and accuracy measures, the neural memory module in Titans outperforms other models including Transformer++, showcasing the importance of weight decay and momentum in achieving superior performance.