# Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

**Authors:** Roman Colman, Minh Vu, Manish Bhattarai, Martin Ma, Hari Viswanathan, Daniel O'Malley, Javier E. Santos  
**arXiv:** [2412.02886](https://arxiv.org/abs/2412.02886)  
**Date:** 2024-12-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.02886v1/

## Key points

- Traditional OCR methods struggle with noisy backgrounds, varied fonts, and handwritten content
- Additional processing steps using large language models (LLMs) are needed to structure extracted information
- Two-step approach can introduce errors and inefficiencies due to context loss and limited flexibility
- Vision Language Models (VLMs) offer a promising solution for efficient and accurate information extraction from scanned documents
- PatchFinder algorithm is introduced for extracting information from scanned documents by leveraging VLMs
- PatchFinder uses Patch Confidence score to determine optimal patch size for partitioning input document into patches
- PatchFinder demonstrates significant performance improvements over existing methods by integrating visual and text information effectively
- Experimental results show PatchFinder's accuracy rate of 94% on a dataset of 190 noisy scanned documents, surpassing existing models by 18.5 percentage points
- Preliminary results on datasets from Colorado and Pennsylvania showcase PatchFinder's superior performance in handling complex layouts and noisy backgrounds

## Summary

In the realm of information extraction from scanned documents, traditional OCR methods have long been relied upon to convert images of text into machine-readable formats. However, these methods often struggle with noisy backgrounds, varied fonts, and handwritten content. This necessitates additional processing steps using large language models (LLMs) to structure the extracted information. While this two-step approach can be effective, it can also introduce errors and inefficiencies due to context loss and limited flexibility in handling diverse layouts and noise levels. The emergence of Vision Language Models (VLMs) offers a promising solution to enhance the efficiency and accuracy of information extraction tasks from scanned documents. VLMs combine both visual and language components internally, enabling a more streamlined end-to-end workflow compared to traditional OCR methods. Leveraging the strengths of VLMs, this study introduces PatchFinder - a novel algorithm designed for extracting information from scanned documents. PatchFinder operates by first assessing the VLM's confidence in its predictions using a newly proposed confidence-based score called Patch Confidence. Based on this assessment, PatchFinder determines an optimal patch size for partitioning the input document into overlapping patches. It then generates confidence-based predictions for extracting target information from each patch. By effectively integrating visual and text information, PatchFinder demonstrates significant performance improvements over existing methods. Experimental results showcase PatchFinder's effectiveness in leveraging Phi-3v - a 4.2 billion parameter vision language model - achieving an impressive accuracy rate of 94% on a dataset comprising 190 noisy scanned documents. This performance surpasses existing models like ChatGPT-4o by 18.5 percentage points, highlighting the potential of VLMs to revolutionize document analysis tasks for complex and noisy documents. Furthermore, this study establishes a baseline performance using state-of-the-art open-source models on datasets from Colorado and Pennsylvania. The preliminary results demonstrate PatchFinder's superior performance compared to other methods on these datasets, showcasing its efficacy in handling complex layouts and noisy backgrounds. Overall, this research underscores the transformative impact of VLMs in enhancing information extraction processes from scanned documents and sets a new benchmark for accuracy and efficiency in document analysis tasks.