# Learning to Reason and Memorize with Self-Notes

**Authors:** Jack Lanchantin, Shubham Toshniwal, Jason Weston, Arthur Szlam, Sainbayar Sukhbaatar  
**arXiv:** [2305.00833](https://arxiv.org/abs/2305.00833)  
**Date:** 2023-05-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.00833v1/

## Key points

- Large language models struggle with limited context memory and multi-step reasoning in state-tracking tasks
- Self-Notes is a proposed method that allows the model to explicitly think and recall information on the fly as it reads the context, extending its memory and enabling multi-step reasoning
- Unlike recent scratchpad approaches, Self-Notes allow the model to deviate from the input context at any time
- The authors demonstrate through experiments on multiple tasks that their method can successfully generalize to longer and more complicated instances from their training setup by taking Self-Notes at inference time
- Rationales have been explored for interpretability and intermediate computations, with Scratchpad being closest to Self-Notes as an online variant
- Chain-of-thought reasoning using rationales has also been shown to be beneficial for zero- and few-shot in-context learning with large language models
- Unlike Scratchpad or chain-of-thought reasoning, Self-Notes are done while reading the entire input context in real-time
- Overall, Self Notes provides a promising solution for improving large language models' performance in state tracking tasks and multi-step reasoning by allowing them to take explicit notes while reading input contexts.

## Summary

Large language models have been shown to struggle with limited context memory and multi-step reasoning, particularly in state-tracking tasks. To address this issue, the authors propose a simple method called Self-Notes that allows the model to explicitly think and recall information on the fly as it reads the context, thus extending its memory and enabling multi-step reasoning. Unlike recent scratchpad approaches, Self-Notes allow the model to deviate from the input context at any time. The authors demonstrate through experiments on multiple tasks that their method can successfully generalize to longer and more complicated instances from their training setup by taking Self-Notes at inference time. The use of rationales has been explored for interpretability and intermediate computations, with Scratchpad being closest to Self-Notes as an online variant. Chain-of-thought reasoning using rationales has also been shown to be beneficial for zero- and few-shot in-context learning with large language models. However, unlike Scratchpad or chain-of-thought reasoning, Self-Notes are done while reading the entire input context in real time. Overall, the proposed Self Notes method provides a promising solution for improving large language models' performance in state tracking tasks and multi step reasoning by allowing them to take explicit notes while reading input contexts.