# Masked Attention is All You Need for Graphs

**Authors:** David Buterez, Jon Paul Janet, Dino Oglic, Pietro Lio  
**arXiv:** [2402.10793](https://arxiv.org/abs/2402.10793)  
**Date:** 2024-02-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.10793v1/

## Key points

- Graph Neural Networks (GNNs) widely utilized for learning on graphs due to flexibility, speed, and performance
- Designing powerful GNNs requires extensive research and carefully chosen message passing operators
- MAG proposed as a simple alternative approach using attention mechanisms exclusively for graph representation
- MAG demonstrates state-of-the-art performance on long-range tasks, surpassing strong baselines and complex methods across various tasks
- MAG exhibits improved transfer learning capabilities compared to traditional GNNs with efficient time and memory scaling
- Sub-linear memory scaling in relation to the number of nodes or edges enables efficient learning on dense graphs
- Specific task evaluations like citation networks show MAG as top-performing method by a significant margin
- Readout functions for GNNs and PMA module for MAG are essential for diverse range of problems from different domains

## Summary

In the realm of learning on graphs,  have been widely utilized due to their flexibility, speed, and satisfactory performance. However, designing powerful and general-purpose GNNs often requires extensive research efforts and the use of carefully chosen message passing operators. In light of this challenge, a remarkably simple alternative approach for learning on graphs has been proposed - . MAG exclusively relies on attention mechanisms to represent graphs as node or edge sets, with connectivity enforced through masking the attention weight matrix to create custom attention patterns for each graph. Despite its simplicity, MAG has demonstrated state-of-the-art performance on long-range tasks, surpassing strong message passing baselines and more complex attention-based methods across over 55 node and graph-level tasks. Additionally, MAG exhibits significantly improved transfer learning capabilities compared to traditional GNNs while showcasing comparable or better time and memory scaling. Notably, MAG boasts sub-linear memory scaling in relation to the number of nodes or edges present in a graph, enabling efficient learning on dense graphs and ensuring future-proofing of the approach. Delving deeper into specific task evaluations, such as citation networks have been explored using representative datasets like PPI,CITESEER,and CORA. emerges as the top-performing method by a significant margin in these scenarios. On the other hand, encompass a diverse range of problems from various domains that necessitate readout functions for GNNs and a PMA module for MAG. Despite variations in readouts' effectiveness noted in prior studies,. Overall,  presents a promising avenue for learning on graphs with its simplicity yet remarkable performance across different task categories.