# Vision language models are blind

**Authors:** Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen  
**arXiv:** [2407.06581](https://arxiv.org/abs/2407.06581)  
**Date:** 2024-07-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.06581v1/

## Key points

- Authors highlight limitations of large language models with vision capabilities (VLMs) like GPT-4o and Gemini 1.5 Pro
- VLMs excel in image-text applications but struggle with basic visual tasks
- Study found state-of-the-art VLMs performed poorly on visual tasks
- Vision capabilities of models likened to a person with myopia or an intelligent person making educated guesses while blind
- Authors propose new benchmark called BlindTest for low-level visual tasks without prior knowledge or complex reasoning

## Summary

In their paper titled "Vision language models are blind," authors Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen highlight the limitations of large language models with vision capabilities (VLMs) such as GPT-4o and Gemini 1.5 Pro. These models excel in many image-text applications and vision-understanding benchmarks but struggle with basic visual tasks that are trivial for humans. The authors conducted a study to assess the performance of four state-of-the-art VLMs on these visual tasks and found them to be shockingly poor. They likened the vision capabilities of these models to that of a person with myopia who sees fine details as blurry or even to an intelligent person making educated guesses while being blind. The code for their study is available at https://vlmsareblind.github.io/. Additionally, the authors discuss existing benchmarks used to evaluate VLMs' vision understanding abilities. These include assessing performance on college-level topics, charts, documents, or videos. While these benchmarks have shown rapid progress in VLMs' performance, they often rely on real-world data that require extensive prior knowledge and can suffer from data leakage issues where VLMs can provide accurate answers without even needing the input image. To address these limitations, the authors propose a new benchmark called BlindTest that focuses on low-level visual tasks that are extremely easy for humans and do not require prior knowledge or complex reasoning. This benchmark aims to provide a more accurate assessment of VLMs' vision capabilities by testing their ability to perceive images like humans do without relying heavily on language processing. Overall, this study sheds light on the challenges faced by current VLMs in basic visual tasks and proposes a new benchmark to better evaluate their vision understanding abilities.