# Linearity of Relation Decoding in Transformer Language Models

**Authors:** Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, David Bau  
**arXiv:** [2308.09124](https://arxiv.org/abs/2308.09124)  
**Date:** 2023-08-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.09124v1/

## Key points

- The paper explores encoding of knowledge in transformer language models (LMs) through relations.
- A significant portion of knowledge in LMs can be expressed in terms of various relations such as synonyms between words and attributes of entities.
- Certain types of relations within LMs can be approximated by a single linear transformation applied to the subject representation.
- Linear relation representations can be derived for factual, commonsense, and linguistic relationships by constructing a first-order approximation to the LM from a single prompt.
- LM predictions accurately capture relational knowledge that is not linearly encoded in their representations, suggesting a nuanced approach to understanding how transformer LMs encode and utilize relational knowledge.
- The findings reveal a simple yet heterogeneously deployed strategy for representing knowledge within transformer LMs, shedding light on the interpretability and complexity of relational knowledge encoding in these models.

## Summary

The paper "Linearity of Relation Decoding in Transformer Language Models" by Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov and David Bau explores the encoding of knowledge in transformer language models (LMs) through relations. The authors demonstrate that a significant portion of this knowledge can be expressed in terms of various relations such as synonyms between words and attributes of entities. They propose that for certain types of relations, the computation within LMs can be approximated by a single linear transformation applied to the subject representation. By constructing a first-order approximation to the LM from a single prompt, linear relation representations can be derived for factual, commonsense and linguistic relationships. However, the study also highlights instances where LM predictions accurately capture relational knowledge that is not linearly encoded in their representations. This suggests a nuanced approach to understanding how transformer LMs encode and utilize relational knowledge. The findings reveal a simple yet heterogeneously deployed strategy for representing knowledge within transformer LMs. This research sheds light on the interpretability and complexity of relational knowledge encoding in these models and contributes valuable insights into their ability to process and understand various types of relationships within language data.