# WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?

**Authors:** An-Lan Wang, Jingqun Tang, Liao Lei, Hao Feng, Qi Liu, Xiang Fei, Jinghui Lu, Han Wang, Weiwei Liu, Hao Liu, Yuliang Liu, Xiang Bai, Can Huang  
**arXiv:** [2505.11015](https://arxiv.org/abs/2505.11015)  
**Date:** 2025-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.11015v2/

## Key points

- Recent advancements in Multimodal Large Language Models (MLLMs) have expanded capabilities to include high-resolution document images, marking a significant evolution in their applicability.
- Existing benchmarks like DocVQA and ChartQA primarily consist of scanned or digital documents, lacking complexity found in real-world scenarios such as variable views, illumination, and physical distortions.
- WildDoc is introduced as the first benchmark designed for assessing document understanding in natural environments, with over 12,000 curated document images reflecting diverse real-world scenarios categorized into Environment, Illumination, View, Distortion, and Effect factors.
- Utilizing document sources from established benchmarks like DocVQA and ChartQA offers advantages such as covering various common document types and facilitating direct comparisons between scanned/digital documents and real-world captured documents.
- A consistency metric is introduced to evaluate model robustness across different real-world conditions by capturing each document four times under distinct scenarios.
- The data collection process involves utilizing documents from previous benchmarks and printing them using high-resolution printers before carefully trimming them for image capture to ensure coverage of a wide range of document types and scenarios encountered in everyday life.
- WildDoc provides a comprehensive evaluation platform for assessing model performance in real-world document understanding tasks.

## Summary

Recent advancements in Multimodal Large Language Models (MLLMs) have expanded their capabilities to encompass high-resolution document images. This marks a significant evolution in their scope of applicability. However, existing benchmarks like DocVQA and ChartQA primarily consist of scanned or digital documents. They fail to capture the complexities posed by real-world scenarios such as variable views, illumination, and physical distortions. This limitation raises questions about the effectiveness of current models under real-world conditions. To address this gap, WildDoc is introduced as the first benchmark specifically designed for assessing document understanding in natural environments. With over 12,000 meticulously curated document images reflecting diverse real-world scenarios categorized into Environment, Illumination, View, Distortion, and Effect factors, WildDoc aims to simulate the complexities encountered in everyday document processing. Utilizing document sources from established benchmarks like DocVQA and ChartQA offers advantages such as covering various common document types and facilitating direct comparisons between scanned/digital documents and real-world captured documents. Additionally, a consistency metric is introduced to evaluate model robustness across different real-world conditions by capturing each document four times under distinct scenarios. The data collection process involves utilizing documents from previous benchmarks and printing them using high-resolution printers before carefully trimming them for image capture. This meticulous approach ensures that the benchmark covers a wide range of document types and scenarios encountered in everyday life. Overall, WildDoc provides a comprehensive evaluation platform for assessing model performance in real-world document understanding tasks. By highlighting performance discrepancies between traditional benchmarks and WildDoc's real-world dataset, this benchmark sheds light on the unique challenges posed by natural environment document processing.