# BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search

**Authors:** Yunjiang Jiang, Yue Shang, Ziyang Liu, Hongwei Shen, Yun Xiao, Wei Xiong, Sulong Xu, Weipeng Yan, Di Jin  
**arXiv:** [2010.10442](https://arxiv.org/abs/2010.10442)  
**Date:** 2020-10-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.10442v1/

## Key points

- Importance of relevance in user experience and business profit for e-commerce search platforms
- Proposal of a data-driven framework for search relevance prediction using BERT distillation
- Distillation process resulting in a student model with over 97% test accuracy compared to teacher models
- Significant reduction in serving costs with lower latency than BERT-Base and TinyBERT
- Introduction of techniques like temperature rescaling and teacher model stacking to enhance accuracy without increasing complexity
- Evaluation on in-house e-commerce search relevance data and public dataset on sentiment analysis from GLUE benchmark
- Embedding analysis and case study demonstrating the strength of the resulting model
- Public availability of data processing and model training source code to reduce energy consumption and promote accessibility for small organizations.

## Summary

The paper titled "BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search" addresses the importance of relevance in user experience and business profit for e-commerce search platforms. The authors propose a data-driven framework for search relevance prediction by distilling knowledge from BERT and related multi-layer Transformer teacher models into simple feed-forward networks. This distillation process results in a student model that achieves over 97% test accuracy compared to the teacher models, while significantly reducing serving costs (with latency 150x lower than BERT-Base and 15x lower than TinyBERT). The authors also introduce techniques such as temperature rescaling and teacher model stacking to further enhance model accuracy without increasing complexity. The experimental results presented in the paper include evaluations on both in-house e-commerce search relevance data and a public dataset on sentiment analysis from the GLUE benchmark. The latter leverages another large-scale public dataset, disregarding potentially noisy labels. The authors perform embedding analysis and present a case study on the in-house data to demonstrate the strength of their resulting model. In an effort to reduce energy consumption of state-of-the-art Transformer models and level the playing field for small organizations lacking access to cutting-edge machine learning hardware, the authors make their data processing and model training source code publicly available. Overall, this paper provides insights into improving search relevance prediction using BERT distillation with massive unlabeled data, showcasing its effectiveness through extensive experiments and analyses.