# ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding

**Authors:** Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang  
**arXiv:** [2210.06155](https://arxiv.org/abs/2210.06155)  
**Date:** 2022-10-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.06155v1/

## Key points

- Significant advancement in pre-training techniques for visually-rich document understanding
- Introduction of ERNIE-Layout by a team of researchers
- ERNIE-Layout enhances layout knowledge and generates better representations by combining text, layout, and image features
- Key innovation of ERNIE-Layout: rearranging input sequences, introducing reading order prediction task, integrating spatial-aware disentangled attention, and replaced regions prediction task during pre-training
- Experimental results show that ERNIE-Layout outperforms existing methods on various downstream tasks
- Research paper detailing ERNIE-Layout accepted at EMNLP 2022 (Findings) authored by a team of experts in the field
- Code and models associated with ERNIE-Layout are publicly available for further exploration and implementation

## Summary

In recent years, there has been a significant advancement in pre-training techniques for visually-rich document understanding. However, many existing methods lack the systematic mining and utilization of layout-centered knowledge, resulting in sub-optimal performance. To address this issue, a team of researchers including Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang,
Shikun Feng,Yu Sun,Hao Tian,Hua Wu,and Haifeng Wang have introduced ERNIE-Layout. This novel document pre-training solution enhances layout knowledge throughout the workflow to generate better representations by combining text features with layout and image features. The key innovation of ERNIE-Layout lies in its approach to rearranging input sequences during the serialization stage and introducing a reading order prediction task as part of the pre-training process. By learning the proper reading order of documents through this correlative task,the model gains a deeper understanding of document structure. Additionally,to enhance layout awareness further,the researchers integrate spatial-aware disentangled attention into the multi-modal transformer architecture.They also incorporate a replaced regions prediction task during pre-training to improve model performance. Experimental results demonstrate that ERNIE-Layout outperforms existing methods on various downstream tasks such as key information extraction,
document image classification,and document question answering datasets.The research paper detailing ERNIE-Layout has been accepted at EMNLP 2022 (Findings)and is authored by a team of experts in the field.The code and models associated with ERNIE-Layout are publicly available for further exploration and implementation.