AI summary · arXiv: 1907.04652v1 · cs.LG
Graph Representation Learning via Hard and Channel-Wise Attention Networks
The essentials
- Attention operators are essential in graph representation learning across various domains such as computer vision, natural language processing, and network embedding learning.
- Traditional graph attention operators (GAOs) have high computational demands, limiting their applicability to large graphs.
- The hard graph attention operator (hGAO) selectively attends to crucial nodes, enhancing performance while reducing computational costs by focusing solely on significant elements.
- The channel-wise graph attention operator (cGAO) conducts attention operations along channels, resulting in substantial reductions in computational resource requirements.
- Models incorporating hGAO and cGAO consistently outperform existing approaches in experimental findings.
- hGAO exhibits significantly superior performance compared to traditional GAOs across node and graph embedding tasks.
- cGAO achieves remarkable savings in computational resources, making it well-suited for handling large-scale graphs effectively.
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Rex Ying (2 papers)
2 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Masked Attention is All You Need for GraphsDavid Buterez, Jon Paul Janet, Dino Oglic et al. · 2024 · Machine Learning · cited 45 times · AI summary available
Abstract
Graph neural networks (GNNs) and variations of the message passing algorithm are the predominant means for learning on graphs, largely due to their flexibility, speed, and satisfactory performance. The design of powerful and general purpose GNNs, however, requires significant research efforts and often relies on handcrafted, carefully-chosen message passing operators. Motivated by this, we propose a remarkably simple alternative for learning on graphs that relies exclusively on attention. Graphs are represented as node or edge sets and their connectivity is enforced by masking the attention weight matrix, effectively creating custom attention patterns for each graph. Despite its simplicity, …
-
Hypergraph Convolution and Hypergraph AttentionSong Bai, Feihu Zhang, P. Torr · 2019 · Pattern Recognition · cited 976 times · Not summarized yet
Abstract
Recently, graph neural networks have attracted great attention and achieved prominent performance in various research fields. Most of those algorithms have assumed pairwise relationships of objects of interest. However, in many real applications, the relationships between objects are in higher-order, beyond a pairwise formulation. To efficiently learn deep embeddings on the high-order graph-structured data, we introduce two end-to-end trainable operators to the family of graph neural networks, i.e., hypergraph convolution and hypergraph attention. Whilst hypergraph convolution defines the basic formulation of performing convolution on a hypergraph, hypergraph attention further enhances the c…
-
Sparse Graph Attention NetworksYang Ye, Shihao Ji · 2019 · IEEE Transactions on Knowledge and Data Engineering · cited 220 times · Not summarized yet
Abstract
Graph Neural Networks (GNNs) have proved to be an effective representation learning framework for graph-structured data, and have achieved state-of-the-art performance on many practical predictive tasks, such as node classification, link prediction and graph classification. Among the variants of GNNs, Graph Attention Networks (GATs) learn to assign dense attention coefficients over all neighbors of a node for feature aggregation, and improve the performance of many graph learning tasks. However, real-world graphs are often very large and noisy, and GATs are prone to overfitting if not regularized properly. Even worse, the local aggregation mechanism of GATs may fail on disassortative graphs,…
-
NO-GAT: Neighbor Overlay-Induced Graph Attention NetworkTiqiao Wei, Ye Yuan · 2024 · IEEE International Conference on Systems, Man and Cybernetics · cited 1 time · Not summarized yet
Abstract
Graph neural networks (GNNs) have garnered significant attention due to their ability to represent graph data. Among various GNN variants, graph attention network (GAT) stands out since it is able to dynamically learn the importance of different nodes. However, present GATs heavily rely on the smoothed node features to obtain the attention coefficients rather than graph structural information, which fails to provide crucial contextual cues for node representations. To address this issue, this study proposes a neighbor overlay-induced graph attention network (NO-GAT) with the following two-fold ideas: a) learning favorable structural information, i.e., overlaid neighbors, outside the node fea…
-
Ego Attention Network: Local Awareness Boosts Graph Attention for Graph Representation LearningJia-Xing Li, Hui Ding, Ke Zhang et al. · 2026 · IEEE Transactions on Signal and Information Processing over Networks · cited 2 times · Not summarized yet
Abstract
Graph neural networks are now widely studied due to the success of graph representation learning ability, and graph attention networks (GATs) is one of the most popular state-of-the-art. By adapting attention mechanisms to the graph domain, GATs derive their expressive power through a learnable process of aggregating neighboring node features based on weights. Nonetheless, the inherent attention mechanism within GATs predominantly hinges upon individual nodes without taking into account their topological structure, which may lead to feature overfitting and sensitivity to structural noise. To tackle these constraints, we propose a novel graph attention network with local awareness: Ego Attent…
-
HONGAT: Graph Attention Networks in the Presence of High-Order NeighborsHeng-Kai Zhang, Yi-Ge Zhang, Zhi Zhou et al. · 2024 · AAAI Conference on Artificial Intelligence · cited 17 times · Not summarized yet
Abstract
Graph Attention Networks (GATs) that compute node representation by its lower-order neighbors, are state-of-the-art architecture for representation learning with graphs. In practice, however, the high-order neighbors that turn out to be useful, remain largely unemployed in GATs. Efforts on this issue remain to be limited. This paper proposes a simple and effective high-order neighbor GAT (HONGAT) model to both effectively exploit informative high-order neighbors and address over-smoothing at the decision boundary of nodes. Two tightly coupled novel technologies, namely common neighbor similarity and new masking matrix, are introduced. Specifically, high-order neighbors are fully explored by …
-
Graph Attention NetworksPetar Velickovic, Guillem Cucurull, Arantxa Casanova et al. · 2017 · International Conference on Learning Representations · cited 29,670 times · Not summarized yet
Abstract
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applic…
-
LDA-GCN: Localized Differential Attention for Graph Convolutional NetworkGui-Cai Yang, Min Lu, Hua Ji et al. · 2025 · International Conference on Computer Supported Cooperative Work in Design · Not summarized yet
Abstract
Graph Convolutional Networks (GCNs) have demonstrated significant efficacy in graph representation learning. Existing approaches often rely on stacking multiple GCN layers to incorporate high-order neighborhood information. How-ever, deeper GCN architectures tend to learn similar node representations by more focusing on the global topology of the graph, thereby diminishing the influence of local structures and leading to over-smoothing. To address this, we propose a Localized Differential Attention Graph Convolutional Network (LDA -GCN), designed to enhance the preservation of local structural information. LDA-GCN introduces a localized differential attention mechanism that diversifies node …
Prior work it builds on
-
Learning Convolutional Neural Networks for GraphsMathias Niepert, Mohamed Ahmed, Konstantin Kutzkov · 2016 · International Conference on Machine Learning · cited 2,314 times · Not summarized yet
Abstract
Numerous important problems can be framed as learning from graph data. We propose a framework for learning convolutional neural networks for arbitrary graphs. These graphs may be undirected, directed, and with both discrete and continuous node and edge attributes. Analogous to image-based convolutional networks that operate on locally connected regions of the input, we present a general approach to extracting locally connected regions from graphs. Using established benchmark data sets, we demonstrate that the learned feature representations are competitive with state of the art graph kernels and that their computation is highly efficient.
-
Hierarchical Graph Representation Learning with Differentiable PoolingRex Ying, Jia-Xuan You, Christopher Morris et al. · 2018 · Neural Information Processing Systems · cited 2,641 times · Not summarized yet
Abstract
Recently, graph neural networks (GNNs) have revolutionized the field of graph representation learning through effectively learned node embeddings, and achieved state-of-the-art results in tasks such as node classification and link prediction. However, current GNN methods are inherently flat and do not learn hierarchical representations of graphs---a limitation that is especially problematic for the task of graph classification, where the goal is to predict the label associated with an entire graph. Here we propose DiffPool, a differentiable graph pooling module that can generate hierarchical representations of graphs and can be combined with various graph neural network architectures in an e…
-
Show, Attend and Tell: Neural Image Caption Generation with Visual AttentionKe Xu, Jimmy Ba, R. Kiros et al. · 2015 · International Conference on Machine Learning · cited 11,166 times · Not summarized yet
Abstract
Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr9k, Flickr30k and MS COCO.
-
Large-Scale Learnable Graph Convolutional NetworksHongyang Gao, Zhengyang Wang, Shuiwang Ji · 2018 · Knowledge Discovery and Data Mining · cited 671 times · Not summarized yet
Abstract
Convolutional neural networks (CNNs) have achieved great success on grid-like data such as images, but face tremendous challenges in learning from more generic data such as graphs. In CNNs, the trainable local filters enable the automatic extraction of high-level features. The computation with filters requires a fixed number of ordered units in the receptive fields. However, the number of neighboring units is neither fixed nor are they ordered in generic graphs, thereby hindering the applications of convolutional operations. Here, we address these challenges by proposing the learnable graph convolutional layer (LGCL). LGCL automatically selects a fixed number of neighboring nodes for each fe…
-
Learning Visual Question Answering by Bootstrapping Hard AttentionMateusz Malinowski, Carl Doersch, A. Santoro et al. · 2018 · European Conference on Computer Vision · cited 106 times · Not summarized yet
Abstract
Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while ent…
Papers that cite it
-
Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised LearningTiantian He, Haicang Zhou, Y. Ong et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 11 times · Not summarized yet
Abstract
Graph attention networks (GATs) are powerful tools for analyzing graph data from various real-world scenarios. To learn representations for downstream tasks, GATs generally attend to all neighbors of the central node when aggregating the features. In this paper, we show that a large portion of the neighbors are irrelevant to the central nodes in many real-world graphs, and can be excluded from neighbor aggregation. Taking the cue, we present Selective Attention (SA) and a series of novel attention mechanisms for graph neural networks (GNNs). SA leverages diverse forms of learnable node-node dissimilarity to acquire the scope of attention for each node, from which irrelevant neighbors are exc…
-
Attention-based graph neural networks: a surveyCheng-Cheng Sun, Chen-Hao Li, Xiang Lin et al. · 2023 · Artificial Intelligence Review · cited 143 times · Not summarized yet
Abstract
Graph neural networks (GNNs) aim to learn well-trained representations in a lower-dimension space for downstream tasks while preserving the topological structures. In recent years, attention mechanism, which is brilliant in the fields of natural language processing and computer vision, is introduced to GNNs to adaptively select the discriminative features and automatically filter the noisy information. To the best of our knowledge, due to the fast-paced advances in this domain, a systematic overview of attention-based GNNs is still missing. To fill this gap, this paper aims to provide a comprehensive survey on recent advances in attention-based GNNs. Firstly, we propose a novel two-level tax…
-
How to Find Your Friendly Neighborhood: Graph Attention Design with Self-SupervisionDongkwan Kim, Alice H. Oh · 2022 · International Conference on Learning Representations · cited 325 times · Not summarized yet
Abstract
Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two gra…
-
Kronecker Attention NetworksHongyang Gao, Zheng-Yang Wang, Shuiwang Ji · 2020 · Knowledge Discovery and Data Mining · cited 40 times · Not summarized yet
Abstract
Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by assuming the data follow matrix-variate normal distributions. Based on this new view, we develop Kronecker attention operators (KAOs) that operate on high-order tensor data directly. More importantly, the proposed KAOs lead to d…
-
Causal-Based Supervision of Attention in Graph Neural Network: A Better and Simpler Choice towards Powerful AttentionHong-Jun Wang, Ji-Yuan Chen, Lun Du et al. · 2023 · International Joint Conference on Artificial Intelligence · cited 13 times · Not summarized yet
Abstract
Recent years have witnessed the great potential of attention mechanism in graph representation learning. However, while variants of attention-based GNNs are setting new benchmarks for numerous real-world datasets, recent works have pointed out that their induced attentions are less robust and generalizable against noisy graphs due to lack of direct supervision. In this paper, we present a new framework which utilizes the tool of causality to provide a powerful supervision signal for the learning process of attention functions. Specifically, we estimate the direct causal effect of attention to the final prediction, and then maximize such effect to guide attention attending to more meaningful …
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 3 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.