# Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification

**Authors:** Quanshi Zhang, Xu Cheng, Yilan Chen, Zhefan Rao  
**arXiv:** [2208.08741](https://arxiv.org/abs/2208.08741)  
**Date:** 2022-08-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2208.08741v1/

## Key points

- Knowledge distillation often leads to superior performance in DNNs compared to traditional learning from scratch
- The paper introduces a new perspective on knowledge distillation by quantifying the knowledge points encoded in intermediate layers of a DNN
- Three hypotheses are proposed regarding knowledge distillation:
- DNNs trained using knowledge distillation encode more knowledge points compared to those trained from scratch
- Knowledge distillation enables DNNs to learn different knowledge points simultaneously, while DNNs trained from scratch tend to encode various knowledge points sequentially
- DNNs trained with knowledge distillation are often optimized more stably than those trained from scratch
- Three types of metrics with annotations of foreground objects are designed to verify these hypotheses, including measures such as quantity and quality of knowledge points, learning speed of different knowledge points, and stability of optimization directions
- Experiments involving various classification tasks provide evidence supporting the proposed hypotheses
- The paper contributes a new understanding of why knowledge distillation enhances performance in DNNs for classification tasks by quantifying encoded knowledge points and analyzing their characteristics using specific metrics

## Summary

This paper titled "Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification" explores the concept of knowledge distillation and its impact on deep neural networks (DNNs) for classification tasks. The authors highlight that compared to traditional learning from scratch, knowledge distillation often leads to superior performance in DNNs. The paper introduces a new perspective to explain the success of knowledge distillation by quantifying the knowledge points encoded in intermediate layers of a DNN. This quantification is based on information theory, where the signal processing in a DNN is viewed as layer-wise information discarding. The authors define a knowledge point as an input unit whose information is discarded less than other input units. Based on this framework, the authors propose three hypotheses regarding knowledge distillation. Firstly, they suggest that DNNs trained using knowledge distillation encode more knowledge points compared to those trained from scratch. Secondly, they argue that knowledge distillation enables DNNs to learn different knowledge points simultaneously, while DNNs trained from scratch tend to encode various knowledge points sequentially. Lastly, they posit that DNNs trained with knowledge distillation are often optimized more stably than those trained from scratch. To verify these hypotheses, the authors design three types of metrics with annotations of foreground objects. These metrics analyze feature representations of the DNN and include measures such as the quantity and quality of knowledge points, the learning speed of different knowledge points, and the stability of optimization directions. The experiments conducted by the authors involve diagnosing various DNNs for different classification tasks including image classification, 3D point cloud classification, binary sentiment classification, and question answering. The results obtained through these experiments provide evidence supporting their proposed hypotheses. In conclusion, this paper contributes a new understanding of why knowledge distillation can enhance performance in DNNs for classification tasks. By quantifying the encoded knowledge points and analyzing their characteristics using specific metrics, the authors shed light on the advantages of knowledge distillation over traditional learning from scratch.