AI summary · arXiv: 1907.04652v1 · cs.LG

Graph Representation Learning via Hard and Channel-Wise Attention Networks

Hongyang Gao, Shuiwang Ji
Original paper · DOI 10.1145/3292500.3330897 Submitted to arXiv on 05 Jul. 2019 Published in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (25 Jul. 2019) 9 pages, KDD19 License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

  • Attention operators are essential in graph representation learning across various domains such as computer vision, natural language processing, and network embedding learning.
  • Traditional graph attention operators (GAOs) have high computational demands, limiting their applicability to large graphs.
  • The hard graph attention operator (hGAO) selectively attends to crucial nodes, enhancing performance while reducing computational costs by focusing solely on significant elements.
  • The channel-wise graph attention operator (cGAO) conducts attention operations along channels, resulting in substantial reductions in computational resource requirements.
  • Models incorporating hGAO and cGAO consistently outperform existing approaches in experimental findings.
  • hGAO exhibits significantly superior performance compared to traditional GAOs across node and graph embedding tasks.
  • cGAO achieves remarkable savings in computational resources, making it well-suited for handling large-scale graphs effectively.

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Graph representation learning attention operators hard attention mechanism channel-wise graph attention operator computational efficiency

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
Attention Operators in Graph Representation Learning: Enhancing Performance and Efficiency In the realm of graph representation learning, attention operators have emerged as a powerful tool across various domains such as computer vision, natural language processing, and network embedding learning. These operators facilitate the aggregation of information from neighboring nodes by assigning learnable weights. However, traditional graph attention operators (GAOs) have been hindered by their high computational demands, limiting their applicability to large graphs. To address these limitations, a groundbreaking study introduces two novel graph attention operators: the hard graph attention operator (hGAO) and the channel-wise graph attention operator (cGAO). The hGAO leverages a hard attention mechanism that selectively attends to crucial nodes, thereby enhancing performance while reducing computational costs by focusing solely on significant elements. On the other hand, the cGAO conducts attention operations along channels, eliminating reliance on adjacency matrices and resulting in substantial reductions in computational resource requirements. Experimental findings demonstrate that deep models incorporating these innovative operators consistently outperform existing approaches. Notably, the hGAO exhibits significantly superior performance compared to traditional GAOs across node and graph embedding tasks. Furthermore, efficiency analyses reveal that the cGAO achieves remarkable savings in computational resources, making it well-suited for handling large-scale graphs effectively. In conclusion, this research contributes valuable insights into advancing graph representation learning through enhanced attention mechanisms. By introducing hGAO and cGAO as efficient alternatives to conventional GAOs,this study paves the way for more effective utilization of attention operators in complex graph-based applications.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts 59 papers already cite it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 31 papers around this one, the name that comes back most: Rex Ying (2 papers)

2 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter The latest letter of its field, "Artificial intelligence this week" of 16 November 2026: Artificial intelligence this week: What impressive scores leave out · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close
    Masked Attention is All You Need for Graphs
    David Buterez, Jon Paul Janet, Dino Oglic et al. · 2024 · Machine Learning · cited 45 times · AI summary available
    Abstract

    Graph neural networks (GNNs) and variations of the message passing algorithm are the predominant means for learning on graphs, largely due to their flexibility, speed, and satisfactory performance. The design of powerful and general purpose GNNs, however, requires significant research efforts and often relies on handcrafted, carefully-chosen message passing operators. Motivated by this, we propose a remarkably simple alternative for learning on graphs that relies exclusively on attention. Graphs are represented as node or edge sets and their connectivity is enforced by masking the attention weight matrix, effectively creating custom attention patterns for each graph. Despite its simplicity, …

  2. Close
    Hypergraph Convolution and Hypergraph Attention
    Song Bai, Feihu Zhang, P. Torr · 2019 · Pattern Recognition · cited 976 times · Not summarized yet
    Abstract

    Recently, graph neural networks have attracted great attention and achieved prominent performance in various research fields. Most of those algorithms have assumed pairwise relationships of objects of interest. However, in many real applications, the relationships between objects are in higher-order, beyond a pairwise formulation. To efficiently learn deep embeddings on the high-order graph-structured data, we introduce two end-to-end trainable operators to the family of graph neural networks, i.e., hypergraph convolution and hypergraph attention. Whilst hypergraph convolution defines the basic formulation of performing convolution on a hypergraph, hypergraph attention further enhances the c…

  3. Close
    Sparse Graph Attention Networks
    Yang Ye, Shihao Ji · 2019 · IEEE Transactions on Knowledge and Data Engineering · cited 220 times · Not summarized yet
    Abstract

    Graph Neural Networks (GNNs) have proved to be an effective representation learning framework for graph-structured data, and have achieved state-of-the-art performance on many practical predictive tasks, such as node classification, link prediction and graph classification. Among the variants of GNNs, Graph Attention Networks (GATs) learn to assign dense attention coefficients over all neighbors of a node for feature aggregation, and improve the performance of many graph learning tasks. However, real-world graphs are often very large and noisy, and GATs are prone to overfitting if not regularized properly. Even worse, the local aggregation mechanism of GATs may fail on disassortative graphs,…

  4. Close
    NO-GAT: Neighbor Overlay-Induced Graph Attention Network
    Tiqiao Wei, Ye Yuan · 2024 · IEEE International Conference on Systems, Man and Cybernetics · cited 1 time · Not summarized yet
    Abstract

    Graph neural networks (GNNs) have garnered significant attention due to their ability to represent graph data. Among various GNN variants, graph attention network (GAT) stands out since it is able to dynamically learn the importance of different nodes. However, present GATs heavily rely on the smoothed node features to obtain the attention coefficients rather than graph structural information, which fails to provide crucial contextual cues for node representations. To address this issue, this study proposes a neighbor overlay-induced graph attention network (NO-GAT) with the following two-fold ideas: a) learning favorable structural information, i.e., overlaid neighbors, outside the node fea…

  5. Close
    Ego Attention Network: Local Awareness Boosts Graph Attention for Graph Representation Learning
    Jia-Xing Li, Hui Ding, Ke Zhang et al. · 2026 · IEEE Transactions on Signal and Information Processing over Networks · cited 2 times · Not summarized yet
    Abstract

    Graph neural networks are now widely studied due to the success of graph representation learning ability, and graph attention networks (GATs) is one of the most popular state-of-the-art. By adapting attention mechanisms to the graph domain, GATs derive their expressive power through a learnable process of aggregating neighboring node features based on weights. Nonetheless, the inherent attention mechanism within GATs predominantly hinges upon individual nodes without taking into account their topological structure, which may lead to feature overfitting and sensitivity to structural noise. To tackle these constraints, we propose a novel graph attention network with local awareness: Ego Attent…

  6. Close
    HONGAT: Graph Attention Networks in the Presence of High-Order Neighbors
    Heng-Kai Zhang, Yi-Ge Zhang, Zhi Zhou et al. · 2024 · AAAI Conference on Artificial Intelligence · cited 17 times · Not summarized yet
    Abstract

    Graph Attention Networks (GATs) that compute node representation by its lower-order neighbors, are state-of-the-art architecture for representation learning with graphs. In practice, however, the high-order neighbors that turn out to be useful, remain largely unemployed in GATs. Efforts on this issue remain to be limited. This paper proposes a simple and effective high-order neighbor GAT (HONGAT) model to both effectively exploit informative high-order neighbors and address over-smoothing at the decision boundary of nodes. Two tightly coupled novel technologies, namely common neighbor similarity and new masking matrix, are introduced. Specifically, high-order neighbors are fully explored by …

  7. Close Cited by this paper
    Graph Attention Networks
    Petar Velickovic, Guillem Cucurull, Arantxa Casanova et al. · 2017 · International Conference on Learning Representations · cited 29,670 times · Not summarized yet
    Abstract

    We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applic…

  8. Close
    LDA-GCN: Localized Differential Attention for Graph Convolutional Network
    Gui-Cai Yang, Min Lu, Hua Ji et al. · 2025 · International Conference on Computer Supported Cooperative Work in Design · Not summarized yet
    Abstract

    Graph Convolutional Networks (GCNs) have demonstrated significant efficacy in graph representation learning. Existing approaches often rely on stacking multiple GCN layers to incorporate high-order neighborhood information. How-ever, deeper GCN architectures tend to learn similar node representations by more focusing on the global topology of the graph, thereby diminishing the influence of local structures and leading to over-smoothing. To address this, we propose a Localized Differential Attention Graph Convolutional Network (LDA -GCN), designed to enhance the preservation of local structural information. LDA-GCN introduces a localized differential attention mechanism that diversifies node …

Prior work it builds on

  1. Close A method it uses
    Learning Convolutional Neural Networks for Graphs
    Mathias Niepert, Mohamed Ahmed, Konstantin Kutzkov · 2016 · International Conference on Machine Learning · cited 2,314 times · Not summarized yet
    Abstract

    Numerous important problems can be framed as learning from graph data. We propose a framework for learning convolutional neural networks for arbitrary graphs. These graphs may be undirected, directed, and with both discrete and continuous node and edge attributes. Analogous to image-based convolutional networks that operate on locally connected regions of the input, we present a general approach to extracting locally connected regions from graphs. Using established benchmark data sets, we demonstrate that the learned feature representations are competitive with state of the art graph kernels and that their computation is highly efficient.

  2. Related A method it uses
    Hierarchical Graph Representation Learning with Differentiable Pooling
    Rex Ying, Jia-Xuan You, Christopher Morris et al. · 2018 · Neural Information Processing Systems · cited 2,641 times · Not summarized yet
    Abstract

    Recently, graph neural networks (GNNs) have revolutionized the field of graph representation learning through effectively learned node embeddings, and achieved state-of-the-art results in tasks such as node classification and link prediction. However, current GNN methods are inherently flat and do not learn hierarchical representations of graphs---a limitation that is especially problematic for the task of graph classification, where the goal is to predict the label associated with an entire graph. Here we propose DiffPool, a differentiable graph pooling module that can generate hierarchical representations of graphs and can be combined with various graph neural network architectures in an e…

  3. Related Used in depth by this paper A method it uses
    Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
    Ke Xu, Jimmy Ba, R. Kiros et al. · 2015 · International Conference on Machine Learning · cited 11,166 times · Not summarized yet
    Abstract

    Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr9k, Flickr30k and MS COCO.

  4. Related A method it uses
    Large-Scale Learnable Graph Convolutional Networks
    Hongyang Gao, Zhengyang Wang, Shuiwang Ji · 2018 · Knowledge Discovery and Data Mining · cited 671 times · Not summarized yet
    Abstract

    Convolutional neural networks (CNNs) have achieved great success on grid-like data such as images, but face tremendous challenges in learning from more generic data such as graphs. In CNNs, the trainable local filters enable the automatic extraction of high-level features. The computation with filters requires a fixed number of ordered units in the receptive fields. However, the number of neighboring units is neither fixed nor are they ordered in generic graphs, thereby hindering the applications of convolutional operations. Here, we address these challenges by proposing the learnable graph convolutional layer (LGCL). LGCL automatically selects a fixed number of neighboring nodes for each fe…

  5. Related Used in depth by this paper
    Learning Visual Question Answering by Bootstrapping Hard Attention
    Mateusz Malinowski, Carl Doersch, A. Santoro et al. · 2018 · European Conference on Computer Vision · cited 106 times · Not summarized yet
    Abstract

    Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while ent…

Papers that cite it

  1. Close Uses this paper in depth
    Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learning
    Tiantian He, Haicang Zhou, Y. Ong et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 11 times · Not summarized yet
    Abstract

    Graph attention networks (GATs) are powerful tools for analyzing graph data from various real-world scenarios. To learn representations for downstream tasks, GATs generally attend to all neighbors of the central node when aggregating the features. In this paper, we show that a large portion of the neighbors are irrelevant to the central nodes in many real-world graphs, and can be excluded from neighbor aggregation. Taking the cue, we present Selective Attention (SA) and a series of novel attention mechanisms for graph neural networks (GNNs). SA leverages diverse forms of learnable node-node dissimilarity to acquire the scope of attention for each node, from which irrelevant neighbors are exc…

  2. Close
    Attention-based graph neural networks: a survey
    Cheng-Cheng Sun, Chen-Hao Li, Xiang Lin et al. · 2023 · Artificial Intelligence Review · cited 143 times · Not summarized yet
    Abstract

    Graph neural networks (GNNs) aim to learn well-trained representations in a lower-dimension space for downstream tasks while preserving the topological structures. In recent years, attention mechanism, which is brilliant in the fields of natural language processing and computer vision, is introduced to GNNs to adaptively select the discriminative features and automatically filter the noisy information. To the best of our knowledge, due to the fast-paced advances in this domain, a systematic overview of attention-based GNNs is still missing. To fill this gap, this paper aims to provide a comprehensive survey on recent advances in attention-based GNNs. Firstly, we propose a novel two-level tax…

  3. Close
    How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
    Dongkwan Kim, Alice H. Oh · 2022 · International Conference on Learning Representations · cited 325 times · Not summarized yet
    Abstract

    Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two gra…

  4. Close
    Kronecker Attention Networks
    Hongyang Gao, Zheng-Yang Wang, Shuiwang Ji · 2020 · Knowledge Discovery and Data Mining · cited 40 times · Not summarized yet
    Abstract

    Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by assuming the data follow matrix-variate normal distributions. Based on this new view, we develop Kronecker attention operators (KAOs) that operate on high-order tensor data directly. More importantly, the proposed KAOs lead to d…

  5. Close
    Causal-Based Supervision of Attention in Graph Neural Network: A Better and Simpler Choice towards Powerful Attention
    Hong-Jun Wang, Ji-Yuan Chen, Lun Du et al. · 2023 · International Joint Conference on Artificial Intelligence · cited 13 times · Not summarized yet
    Abstract

    Recent years have witnessed the great potential of attention mechanism in graph representation learning. However, while variants of attention-based GNNs are setting new benchmarks for numerous real-world datasets, recent works have pointed out that their induced attentions are less robust and generalizable against noisy graphs due to lack of direct supervision. In this paper, we present a new framework which utilizes the tool of causality to provide a powerful supervision signal for the learning process of attention functions. Specifically, we estimate the direct causal effect of attention to the final prediction, and then maximize such effect to guide attention attending to more meaningful …

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 3 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.