# Graph-based Knowledge Distillation: A survey and experimental evaluation

**Authors:** Jing Liu, Tongya Zheng, Guanzheng Zhang, Qinfen Hao  
**arXiv:** [2302.14643](https://arxiv.org/abs/2302.14643)  
**Date:** 2023-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.14643v1/

## Key points

- Graph-based Knowledge Distillation methods and their applications
- Graph Neural Networks (GNNs) require data labels and complex models
- Knowledge Distillation enhances GNNs by transferring soft-label supervision from a large teacher model to a small student model
- Three types of Graph-based Knowledge Distillation methods: DKD, GKD, SKD
- Types are further divided based on the position of knowledge distillation (output layer, middle layer, constructed graph)
- Analysis and comparison of various algorithm ideas with experimental results
- Applications in computer vision (CV), natural language processing (NLP), recommendation systems (RS), and other fields
- Need for theoretical analysis on interpretability of knowledge distillation
- Suggestion to explore new efficient graph distillation methods by combining with other technologies like adversarial learning, neural architecture search, reinforcement learning
- Summary of Graph-based Knowledge Distillation method and future directions for expansion into other uses and applications.

## Summary

This paper provides a comprehensive overview of Graph-based Knowledge Distillation methods and their applications. Graph Neural Networks (GNNs) have gained attention for their performance in various graph applications but require sufficient data labels and complex network models. To address these challenges, Knowledge Distillation (KD) has been introduced to enhance existing GNNs by transferring the soft-label supervision from a large teacher model to a small student model. The paper categorizes and summarizes three types of Graph-based Knowledge Distillation methods: Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type is further divided based on the position of knowledge distillation such as the output layer, middle layer, and constructed graph. The paper analyzes and compares various algorithm ideas discussing their advantages and disadvantages supported by experimental results. Additionally, it lists the applications of graph-based knowledge distillation in computer vision (CV), natural language processing (NLP), recommendation systems (RS), and other fields. The paper also highlights the need for theoretical analysis on the interpretability of knowledge distillation and suggests exploring new efficient graph distillation methods by combining knowledge distillation with other technologies like adversarial learning, neural architecture search, reinforcement learning etc. Finally, the paper concludes by summarizing the method of Graph-based Knowledge Distillation and discussing future directions for its expansion into other uses and applications.