# GPT is becoming a Turing machine: Here are some ways to program it

**Authors:** Ana Jojic, Zhen Wang, Nebojsa Jojic  
**arXiv:** [2303.14310](https://arxiv.org/abs/2303.14310)  
**Date:** 2023-03-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.14310v1/

## Key points

- GPT-3 models can perform iterative behaviors necessary for executing programs involving loops
- This goes beyond just writing or recalling programs and includes popular algorithms found in computer science curricula or software developer interviews
- The authors achieve this by triggering the execution and description of iterations through Regimenting Self-Attention (IRSA) in one or a combination of three ways:
- using strong repetitive structure in an example of an execution path for a target program with one particular input
- prompting with fragments of execution paths
- explicitly forbidding self-attention to parts of the generated text
- IRSA leads to larger accuracy gains than replacing the model with the more powerful GPT-4 on dynamic program execution
- IRSA has promising applications in education as prompts and responses resemble student assignments in data structures and algorithms classes
- Prompts that may not even cover one full task example can trigger algorithmic behavior, allowing for solving problems previously thought hard for language models such as logical puzzles.
- Prompt design plays an even more critical role in language model performance than previously recognized.

## Summary

In their paper titled "GPT is becoming a Turing machine: Here are some ways to program it," Ana Jojic, Zhen Wang, and Nebojsa Jojic demonstrate that the GPT-3 family of models can be prompted to perform iterative behaviors necessary for executing programs involving loops. This goes beyond just writing or recalling programs and includes popular algorithms found in computer science curricula or software developer interviews. The authors achieve this by triggering the execution and description of iterations through Regimenting Self-Attention (IRSA) in one or a combination of three ways: 1) using strong repetitive structure in an example of an execution path for a target program with one particular input; 2) prompting with fragments of execution paths; and 3) explicitly forbidding self-attention to parts of the generated text. The authors find that IRSA leads to larger accuracy gains than replacing the model with the more powerful GPT-4 on dynamic program execution. They also note that IRSA has promising applications in education as prompts and responses resemble student assignments in data structures and algorithms classes. Additionally, they show that prompts that may not even cover one full task example can trigger algorithmic behavior, allowing for solving problems previously thought hard for language models such as logical puzzles. Thus, prompt design plays an even more critical role in language model performance than previously recognized. Overall, this study highlights how appropriate prompting can enable GPT-3 models to perform iterative behaviors necessary for executing programs involving loops. It also underscores the importance of prompt design in language model performance evaluation and has implications for education where these prompts resemble student assignments.