# DocFormer: End-to-End Transformer for Document Understanding

**Authors:** Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, R. Manmatha  
**arXiv:** [2106.11539](https://arxiv.org/abs/2106.11539)  
**Date:** 2021-06-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.11539v1/

## Key points

- DocFormer has significantly advanced Visual Document Understanding (VDU) by introducing innovative pre-training methods and multi-modal interaction.
- Models like LayoutLM, LayoutLMv2, and BROS have also contributed to advancements in document understanding through pre-training on large datasets and utilizing graph-based classifiers.
- Recent works like Layout-T5 and TILT have evolved towards multi-modal transformer encoder-decoder architectures based on T5 for addressing cross-modality feature correlation challenges.
- DocFormer's unique features include incorporating text, vision, and spatial features, leveraging a multi-modal self-attention layer, shared spatial embeddings, and novel pre-training tasks without relying on bulky pre-trained object-detection networks.
- DocFormer demonstrates state-of-the-art performance on diverse datasets compared to larger models due to its emphasis on multi-modal interaction and innovative architectural design.

## Summary

In the realm of Visual Document Understanding (VDU), the introduction of DocFormer has significantly advanced the field. VDU poses a challenging task of comprehending documents in various formats and layouts, such as forms and receipts. DocFormer stands out for its innovative approach to pre-training in an unsupervised manner using carefully designed tasks that promote multi-modal interaction. By incorporating text, vision, and spatial features and leveraging a unique multi-modal self-attention layer, DocFormer excels in correlating text to visual tokens and vice versa. The landscape of document understanding has seen notable advancements with models like LayoutLM and LayoutLMv2, which emphasize pre-training on large datasets followed by fine-tuning on specific tasks related to document processing. Additionally, BROS utilizes a BERT-based encoder with a graph-based classifier to predict entity relations within documents. The evolution towards multi-modal transformer encoder-decoder architectures based on T5 has also been observed in recent works like Layout-T5 and TILT. lies in its ability to address the challenge of cross-modality feature correlation, where mapping text descriptions to visual content proves intricate. By introducing shared spatial embeddings and novel pre-training tasks like learning-to-reconstruct and multi-modal masked language modeling,  as a pioneering approach in VDU without relying on bulky pre-trained object-detection networks for visual feature extraction. In summary, is evident through its state-of-the-art performance on diverse datasets compared to larger models. Its emphasis on multi-modal interaction and innovative architectural design positions it at the forefront of Visual Document Understanding research, paving the way for further advancements in this domain.