# Pure Transformers are Powerful Graph Learners

**Authors:** Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong  
**arXiv:** [2207.02505](https://arxiv.org/abs/2207.02505)  
**Date:** 2022-07-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.02505v2/

## Key points

- Standard Transformers can be effective in graph learning without any graph-specific modifications
- Tokenized Graph Transformer (TokenGT) approach treats all nodes and edges as independent tokens and augment them with token embeddings before feeding them to a Transformer
- With an appropriate choice of token embeddings, TokenGT is theoretically at least as expressive as an invariant graph network composed of equivariant linear layers
- TokenGT outperforms message-passing Graph Neural Networks (GNN) baselines and achieves competitive results compared to other Transformer variants when trained on a large-scale graph dataset (PCQM4Mv2)
- The implementation of TokenGT is available on GitHub
- TokenGT works well with large-scale data and achieve promising results in graph learning both in theory and practice
- The authors explore the capability of TokenGT on the PCQM4Mv2 quantum chemistry regression dataset, which has 3.7 million molecular graphs, using both node and type identifiers in their model
- They also apply kernel attention that approximates the attention computation to linear cost
- This research can contribute to advancing machine learning technology for various applications supported by organizations like the Korea government (MSIT)

## Summary

In their paper titled "Pure Transformers are Powerful Graph Learners," Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong demonstrate that standard Transformers can be effective in graph learning without any graph-specific modifications. The authors propose a Tokenized Graph Transformer (TokenGT) approach where they treat all nodes and edges as independent tokens and augment them with token embeddings before feeding them to a Transformer. With an appropriate choice of token embeddings, the authors prove that this approach is theoretically at least as expressive as an invariant graph network composed of equivariant linear layers. The authors compare their TokenGT approach with message-passing Graph Neural Networks (GNN) and Transformer variants with sophisticated graph-specific inductive bias. They train their model on a large-scale graph dataset (PCQM4Mv2) and show that TokenGT outperforms GNN baselines and achieves competitive results compared to other Transformer variants. The implementation of TokenGT is available on GitHub. The authors also explore the capability of TokenGT on the PCQM4Mv2 quantum chemistry regression dataset, which has 3.7 million molecular graphs. They use both node and type identifiers in their model and experiment with ORF and Laplacian eigenvector as node identifiers. They also apply kernel attention that approximates the attention computation to linear cost. Overall, the authors demonstrate that minimal graph-specific inductive bias models like TokenGT work well with large-scale data and achieve promising results in graph learning both in theory and practice. The details of their experiments are provided in Appendix A.3.3., while the conclusion highlights how this research can contribute to advancing machine learning technology for various applications supported by organizations like the Korea government (MSIT).