# DocLLM: A layout-aware generative language model for multimodal document understanding

**Authors:** Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, Xiaomo Liu  
**arXiv:** [2401.00908](https://arxiv.org/abs/2401.00908)  
**Date:** 2023-12-31  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.00908v1/

## Key points

- DocLLM is a lightweight extension to traditional large language models (LLMs) designed for understanding visual documents
- Visual documents contain rich semantics at the intersection of textual and spatial modalities
- DocLLM focuses on bounding box information to capture the cross-alignment between text and spatial modalities
- It decomposes the attention mechanism in classical transformers into disentangled matrices
- DocLLM utilizes a pre-training objective that learns to infill text segments, addressing irregular layouts and heterogeneous content in visual documents
- Experimental results show that DocLLM outperforms state-of-the-art LLMs on 14 out of 16 datasets across all tasks
- It demonstrates good generalization to previously unseen datasets
- Previous research has primarily focused on LLMs without support for understanding visual documents
- Existing vision-language frameworks may not be suitable for handling visually rich documents
- DocLLM presents a promising solution for understanding visual documents by incorporating textual semantics and spatial layout information

## Summary

, , , , 

The authors introduce DocLLM, a lightweight extension to traditional large language models (LLMs) specifically designed for understanding visual documents. Visual documents, such as forms, invoices, receipts, reports, contracts, and other similar records, often contain rich semantics at the intersection of textual and spatial modalities. The complex layouts of these documents play a crucial role in comprehending them effectively. Unlike existing multimodal LLMs that use expensive image encoders to incorporate spatial layout information, DocLLM focuses exclusively on bounding box information to capture the cross-alignment between text and spatial modalities. This is achieved by decomposing the attention mechanism in classical transformers into a set of disentangled matrices. Furthermore, DocLLM utilizes a pre-training objective that learns to infill text segments, allowing it to effectively address irregular layouts and heterogeneous content commonly found in visual documents. To evaluate its performance, the authors fine-tune the pre-trained model using a large-scale instruction dataset covering four core document intelligence tasks. Experimental results show that DocLLM outperforms state-of-the-art LLMs on 14 out of 16 datasets across all tasks and demonstrates good generalization to previously unseen datasets. In terms of related work, previous research has primarily focused on large language models (LLMs) but lacked support for understanding visual documents. Existing vision-language frameworks have utilized complex vision backbone architectures to encode image information but may not be suitable for handling visually rich documents. Overall,<kgd>DocLLM</kgd> presents a promising solution for understanding visual documents by incorporating both textual semantics and spatial layout information. Its lightweight design and effective modeling approach make it well-suited for various form understanding tasks in real-world applications.