# MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding

**Authors:** Junlong Li, Yiheng Xu, Lei Cui, Furu Wei  
**arXiv:** [2110.08518](https://arxiv.org/abs/2110.08518)  
**Date:** 2021-10-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.08518v1/

## Key points

- Significant progress in Visually-Rich Document Understanding (VrDU) through multimodal pre-training techniques
- Introduction of the MarkupLM model for document understanding tasks involving markup languages like HTML/XML-based documents
- Outperformance of existing baseline models by MarkupLM on various VrDU tasks
- Importance of leveraging markup structures for document-level pre-training in VrDU tasks, especially for markup-language-based documents
- Potential expansion of MarkupLM to digital-born PDFs and Office documents, exploring synergies with LayoutLM under multi-view settings

## Summary

, , , , 

Significant Progress in Visually-Rich Document Understanding: The Role of Multimodal Pre-Training and the MarkupLM Model

In recent years, there has been significant progress in the field of Visually-Rich Document Understanding (VrDU), particularly through the use of multimodal pre-training techniques that incorporate text, layout, and image information. While these approaches have proven successful for fixed-layout documents like scanned images, there remains a challenge in dynamically rendering layout information for interactive visualization in digital documents. To address this gap, the MarkupLM model has been proposed for document understanding tasks involving markup languages such as HTML/XML-based documents. By jointly pre-training text and markup information, MarkupLM outperforms existing baseline models on various VrDU tasks. This is particularly important as visually-rich documents can be categorized into two types: fixed-layout and markup-language-based. Fixed-layout documents already have pre-rendered layout and style information, making them suitable for existing pre-training methods. However, markup-language-based documents require dynamic rendering of layout and style information based on the device used. This difference highlights the need to leverage markup structures for document-level pre-training in VrDU tasks. MarkupLM addresses this need by integrating text and markup language pre-training within a single framework using Transformer architecture. New pre-training objectives tailored for understanding markup language enhance model performance on datasets like WebSRC and SWDE. In future research, there is potential to expand MarkupLM to digital-born PDFs and Office documents while exploring synergies between LayoutLM and MarkupLM under multi-view settings. To fine-tune MarkupLM for downstream tasks such as reading comprehension and information extraction, common practices from pre-trained language models are utilized. This involves leveraging the last hidden states of tokens for binary or linear classification layers. Overall, the development of MarkupLM represents a promising advancement in enhancing document understanding capabilities through effective integration of text and markup language pre-training methodologies.