# Tree Transformer: Integrating Tree Structures into Self-Attention

**Authors:** Yau-Shian Wang, Hung-Yi Lee, Yun-Nung Chen  
**arXiv:** [1909.06639](https://arxiv.org/abs/1909.06639)  
**Date:** 2019-09-14  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.06639v1/

## Key points

- Authors Yau-Shian Wang, Hung-Yi Lee, and Yun-Nung Chen address limitations of existing Transformer models in capturing hierarchical structures in NLP tasks.
- The model introduces an additional constraint to attention heads of the bidirectional Transformer encoder to encourage following tree structures for capturing complex linguistic relationships.
- Key innovation is the "Constituent Attention" module that induces tree structures from raw texts by implementing self-attention between adjacent words.
- Tree Transformer demonstrates improved performance in various NLP tasks, including better language modeling and more interpretable attention scores.
- Experiments show that the model outperforms traditional Transformers in inducing tree structures and enhancing overall task performance.
- Represents a significant advancement by integrating explicit tree structures into self-attention mechanisms, improving interpretability and capturing complex linguistic dependencies effectively.

## Summary

In their paper titled "Tree Transformer: Integrating Tree Structures into Self-Attention," authors Yau-Shian Wang, Hung-Yi Lee, and Yun-Nung Chen address the limitations of existing Transformer models in capturing hierarchical structures in natural language processing tasks. The  model introduces an additional constraint to the attention heads of the bidirectional Transformer encoder to overcome this limitation. This constraint encourages the attention heads to follow , which are essential for capturing complex linguistic relationships. The key innovation of the Tree Transformer lies in its "Constituent Attention" module, which automatically induces tree structures from raw texts by implementing self-attention between adjacent words. By incorporating  into the attention mechanism, the Tree Transformer demonstrates improved performance in various NLP tasks, including better language modeling and more interpretable attention scores. The experiments conducted by Wang et al. show that their model outperforms traditional Transformers in inducing tree structures and enhancing overall task performance. Overall, the Tree Transformer represents a significant advancement in  by integrating explicit tree structures into self-attention mechanisms. This innovative approach not only improves model interpretability but also enhances its ability to capture complex linguistic dependencies effectively.