# Unifying Vision, Text, and Layout for Universal Document Processing

**Authors:** Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal  
**arXiv:** [2212.02623](https://arxiv.org/abs/2212.02623)  
**Date:** 2022-12-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.02623v3/

## Key points

- Universal Document Processing (UDOP) model is a groundbreaking advancement in Document AI
- UDOP unifies text, image, and layout modalities into one cohesive framework
- Core of UDOP is the Vision-Text-Layout Transformer for pretraining and multi-domain downstream tasks
- UDOP is pretrained on large-scale unlabeled document corpora using self-supervised objectives and diverse labeled data
- Unique capability of UDOP to generate document images from text and layout modalities through masked image reconstruction
- Achieved state-of-the-art performance on 8 Document AI tasks and holds the top position on the leaderboard
- Stands out as a pioneering solution for strong cross-modal interactions between text and visual modalities in Document AI
- Takes a holistic approach by unifying vision, text, and layout through its transformative architecture
- Continues to push boundaries in Document AI research with ablation studies on pre-training objectives and model architecture variations

## Summary

In the realm of Document AI, the Universal Document Processing (UDOP) model has emerged as a groundbreaking advancement. UDOP is designed to tackle the complex challenges inherent in document processing by unifying text, image, and layout modalities into one cohesive framework. By leveraging the spatial correlation between textual content and document images, UDOP is able to create a unified representation that seamlessly integrates these different modalities. At its core lies the novel Vision-Text-Layout Transformer which revolutionizes pretraining and multi-domain downstream tasks through a prompt-based sequence generation approach. This innovative model is pretrained on large-scale unlabeled document corpora using self-supervised objectives and diverse labeled data. Additionally, UDOP has the unique capability to generate document images from text and layout modalities through masked image reconstruction, setting a new standard for neural document editing and content customization. The significance of UDOP's capabilities extends across various Document AI tasks including document understanding and QA across diverse data domains such as finance reports, academic papers, and websites. Notably,  has achieved state-of-the-art performance on 8 Document AI tasks and currently holds the top position on the leaderboard of the . Furthermore, in response to the distinct challenges posed by Document AI – where strong cross-modal interactions between text and visual modalities are prevalent –  stands out as a pioneering solution. Unlike traditional vision-language frameworks that treat documents as text-only inputs or use separate encoders for text and image modalities with shallow positional embeddings for layout information, takes a holistic approach by unifying vision, text, and layout through its transformative architecture. Through ablation studies on pre-training objectives and model architecture variations like  – which utilizes separate text and vision encoders with position bias for layout information representation –  continues to push boundaries in Document AI research. With its unparalleled ability to efficiently learn diverse vision, text, and layout tasks across different domains while maximizing the correlation between modalities within documents,  represents a significant leap forward in advancing the field of Document AI.