# GeoChain: Multimodal Chain-of-Thought for Geographic Reasoning

**Authors:** Sahiti Yerramilli, Nilay Pande, Rynaa Grover, Jayant Sravan Tamarapalli  
**arXiv:** [2506.00785](https://arxiv.org/abs/2506.00785)  
**Date:** 2025-06-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2506.00785v1/

## Key points

- Introduction of GeoChain, a benchmark for evaluating geographic reasoning in multimodal large language models (MLLMs)
- Utilization of 1.46 million Mapillary street-level images to create over 30 million Q&A pairs with a 21-step chain-of-thought question sequence
- Evaluation framework includes four categories: visual, spatial, cultural, and precise geolocation
- Benchmarking experiments conducted on MLLMs like GPT-4.1 variants, Claude 3.7, and Gemini 2.5 variants reveal challenges in visual grounding, reasoning patterns, and accurate localization
- Limitations include potential biases from pre-training exposure to similar scenes, skewness in geographical distribution impacting generalizability, and dependence on accuracy of semantic segmentation model for locatability score precision

## Summary

, , , , 


In this paper, the authors introduce GeoChain, a groundbreaking benchmark designed to evaluate step-by-step geographic reasoning in multimodal large language models (MLLMs). Leveraging a vast dataset of 1.46 million Mapillary street-level images, GeoChain pairs each image with a complex 21-step chain-of-thought (CoT) question sequence, resulting in over 30 million Q&A pairs. These sequences are meticulously crafted to guide models through various levels of reasoning, from coarse attributes to precise localization, across four distinct categories - visual, spatial, cultural, and precise geolocation. Each question sequence is annotated by difficulty to provide a comprehensive evaluation framework. Furthermore, the images in GeoChain are enriched with semantic segmentation featuring 150 classes and a visual locatability score. The authors conducted benchmarking experiments on contemporary MLLMs such as GPT-4.1 variants, Claude 3.7, and Gemini 2.5 variants using a diverse subset of 2,088 images. The results revealed consistent challenges faced by these models, including weaknesses in visual grounding, erratic reasoning patterns, and difficulties in achieving accurate localization as the complexity of reasoning tasks increased. While GeoChain offers a robust diagnostic methodology critical for advancing complex geographic reasoning within MLLMs, the authors also acknowledge several limitations. One key limitation is that the benchmark is built upon the Mapillary Street-Level Sequences training split which may lead to potential biases due to pre-training exposure to similar visual scenes. Additionally, the geographical distribution of images may exhibit some skewness impacting generalizability across all urban contexts. Moreover, the precision of the locatability score is dependent on the accuracy of an upstream semantic segmentation model which could introduce noise into difficulty stratification. In conclusion, GeoChain represents a significant step forward in evaluating geographic reasoning capabilities in MLLMs but also highlights important considerations for future research and development in this field.