# LMDX: Language Model-based Document Information Extraction and Localization

**Authors:** Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Jiaqi Mu, Hao Zhang, Nan Hua  
**arXiv:** [2309.10952](https://arxiv.org/abs/2309.10952)  
**Date:** 2023-09-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.10952v1/

## Key points

- Large Language Models (LLMs) have advanced Natural Language Processing (NLP) and improved performance on various tasks.
- LLMs have limited application to semi-structured document information extraction.
- Challenges in adopting LLMs for this task include the absence of layout encoding and lack of a grounding mechanism.
- The authors propose a methodology called LMDX to address these challenges.
- LMDX enables the adaptation of arbitrary LLMs for document information extraction, supporting extraction of singular, repeated, and hierarchical entities with or without training data.
- LMDX provides grounding guarantees and localizes extracted entities within the document.
- LMDX is specifically applied to the PaLM 2-S LLM and evaluated on VRDU and CORD benchmarks, setting a new state-of-the-art in document information extraction.
- Document information extraction from semi-structured documents involves complexities such as complex layouts, spatial alignment, tabular arrangement of entities, printed or handwritten content, scanning artifacts, and precise entity localization.
- Current approaches involve two stages: text recognition/serialization using OCR services followed by parsing to extract relevant entity values from recognized text.
- Existing approaches have limitations in handling hierarchical entities or serialization errors.
- Some approaches leverage image modality in addition to text and layout information for alignment between modalities.
- Other approaches treat extraction as a sequence generation problem with an auto-regressive decoder on top of a text-layout-image encoder.

## Summary

, , , , 

The use of Large Language Models (LLMs) has greatly advanced Natural Language Processing (NLP) and improved performance on various tasks. However, their application to semi-structured document information extraction has been limited. This task involves extracting key entities from visually rich documents (VRDs) based on a predefined schema. The main challenges in adopting LLMs for this task are the absence of layout encoding within LLMs and the lack of a grounding mechanism to ensure accurate extraction. To address these challenges, the authors propose a methodology called Language Model-based Document Information Extraction and Localization (LMDX). LMDX enables the adaptation of arbitrary LLMs for document information extraction by supporting extraction of singular, repeated, and hierarchical entities with or without training data. It also provides grounding guarantees and localizes the extracted entities within the document. The authors specifically apply LMDX to the PaLM 2-S LLM and evaluate its performance on VRDU and CORD benchmarks. The results demonstrate that LMDX sets a new state-of-the-art in document information extraction and enables the creation of high-quality, data-efficient parsers. The introduction provides additional context on the challenges faced in document information extraction from semi-structured documents. It highlights the complexities involved, such as understanding complex layouts, spatial alignment, tabular arrangement of entities, printed or handwritten content, scanning artifacts, and the need for precise entity localization. Additionally, it emphasizes that most parsers are built with limited annotation resources due to the vast number of document types. Current approaches to document information extraction typically involve two stages: text recognition/serialization using Optical Character Recognition (OCR) services followed by parsing to extract relevant entity values from recognized text. Efforts have been made to fuse text and layout information during parsing using techniques like encoding relative 2D distances of text blocks or encoding relative token positions with graph neural networks. However, these approaches have limitations in handling hierarchical entities or serialization errors. Some approaches also leverage the image modality in addition to text and layout information. This involves using separate image encoders or jointly modeling page images and tokens to learn alignment between modalities. Other approaches treat extraction as a sequence generation problem, adding an auto-regressive decoder on top of a text-layout-image encoder. In summary, the introduction provides an overview of the challenges in document information extraction from semi-structured documents and highlights existing approaches that have been explored. The proposed LMDX methodology aims to address these challenges by adapting LLMs for high-quality extraction, grounding guarantees, and entity localization within the document.