# Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation

**Authors:** Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, Lianwen Jin  
**arXiv:** [2310.16809](https://arxiv.org/abs/2310.16809)  
**Date:** 2023-10-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.16809v1/

## Key points

- GPT-4V(ision) is a Large Multimodal Model (LMM) evaluated for Optical Character Recognition (OCR)
- Performs well in recognizing and understanding Latin contents
- Struggles with multilingual scenarios and complex tasks
- Evaluation based on small-scale test sample due to computational limits of GPT-4V, affecting generalizability
- Assessment primarily focuses on mainstream OCR tasks, not comprehensive coverage of all OCR-related tasks
- Only zero-shot capacity of GPT-4V in OCR was evaluated, without exploring few-shot scenarios or further training/fine-tuning possibilities
- Future research should explore few-shot scenarios using technologies like in-context learning for potential benefits
- In-depth analysis of strengths and weaknesses of GPT-4V provided
- Highlights high inference costs and challenges associated with continuous updating as barriers to real-world deployment
- Existing general LMMs can contribute significantly to OCR development by enhancing semantic understanding, fine-tuning for downstream tasks, and facilitating data construction
- Provides first quantitative evaluation of GPT 4V's performance in OCR tasks
- Offers valuable insights and strategies for researchers and practitioners working on OCR tasks using large multimodal models

## Summary

This paper presents a comprehensive evaluation of the Optical Character Recognition (OCR) capabilities of GPT-4V(ision), a Large Multimodal Model (LMM). The study aims to assess the model's performance across various OCR tasks, including scene text recognition, handwritten text recognition, handwritten mathematical expression recognition, table structure recognition, and information extraction from visually-rich documents. The evaluation reveals that GPT-4V performs well in recognizing and understanding Latin contents but struggles with multilingual scenarios and complex tasks. However, there are several limitations to consider. Firstly, the evaluation is based on a small-scale test sample due to the computational limits of GPT-4V. This limitation may affect the generalizability of the results. Additionally, the assessment primarily focuses on mainstream OCR tasks and does not cover other OCR-related tasks comprehensively. Moreover, only the zero-shot capacity of GPT-4V in OCR was evaluated without exploring few-shot scenarios or further training/fine-tuning possibilities for specific tasks. Future research should explore few-shot scenarios using technologies like in-context learning to unlock potential benefits. Despite these limitations, this study provides an in-depth analysis of GPT-4V's strengths and weaknesses. It highlights its strong ability to recognize Latin content accurately while acknowledging its struggles with multilingual and complex scenarios. Furthermore, it emphasizes the high inference costs and challenges associated with continuous updating as significant barriers to real world deployment. Nevertheless, GPT 4V and other existing general LMMs can still contribute significantly to the development of OCR by enhancing semantic understanding, fine tuning for downstream tasks, and facilitating auto/semi auto data construction. In conclusion, this paper offers a first of its kind quantitative evaluation of GPT 4V's performance in OCR tasks. While acknowledging its limitations, it provides valuable insights and strategies for researchers and practitioners working on OCR tasks using large multimodal models. The authors plan to continuously update the evaluation results and hope that this study will serve as a critical reference for future OCR research.