# A Comparative Study in Surgical AI: Datasets, Foundation Models, and Barriers to Med-AGI

**Authors:** Kirill Skobelev, Eric Fithian, Yegor Baranovski, Jack Cook, Sandeep Angara, Shauna Otto, Zhuang-Fang Yi, John Zhu, Daniel A. Donoho, X. Y. Han, Neeraj Mainkar, Margaux Masson-Forsythe  
**arXiv:** [2603.27341](https://arxiv.org/abs/2603.27341)  
**Date:** 2026-03-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2603.27341v1/

## Key points

- Surge in development of AI models for biomedical tasks
- AI models lag behind human experts in surgical image analysis benchmarks
- Challenges in integrating tasks such as multimodal data integration, human interaction, and physical effects in surgery
- Traditional approach of scaling architecture size and training data for AI models
- Uncertainties about the extent to which modern AI can aid in surgical practice
- Limitations of current Vision Language Models in surgical tool detection despite extensive training and model size increase
- Scaling experiments show no significant improvements in performance metrics with increased model size and training time
- Challenges persist across diverse model architectures regardless of additional compute resources
- Data availability and labeling are not the only limiting factors for AI models in surgery applications
- Discussion on main contributors to constraints and proposed potential solutions
- Claims suggesting continued scaling alone could lead to Artificial General Intelligence (AGI)
- Challenges faced by large multimodal foundation models when tested in realistic clinical settings
- Uncertainty on whether vision-language models can lead to Medical AGI in surgery
- Performance limitations on basic perceptual tasks under realistic distribution shifts due to gaps in domain-specific data coverage
- Clinically relevant data availability believed to be more critical than model scale for advancing Surgical AI capabilities

## Summary

In recent years, there has been a surge in the development of Artificial Intelligence (AI) models that have shown promising results in various biomedical tasks. However, when it comes to surgical image analysis benchmarks, AI models have lagged behind human experts. The integration of different tasks such as multimodal data integration, human interaction, and physical effects in surgery makes AI models particularly attractive as collaborative tools if their performance can be improved. The traditional approach of scaling architecture size and training data has been considered an attractive option due to the vast amount of surgical video data generated annually. However, preparing surgical data for AI training requires high levels of expertise and expensive computational resources. This raises uncertainties about the extent to which modern AI can aid in surgical practice. In a case study focusing on surgical tool detection using state-of-the-art AI methods available in 2026, it was found that even with multi-billion parameter models and extensive training, current Vision Language Models fall short in this seemingly simple task. Scaling experiments also indicated that increasing model size and training time did not lead to significant improvements in performance metrics. The experiments suggest that current AI models may face significant obstacles in surgical applications, with some challenges persisting across diverse model architectures regardless of additional compute resources. This raises questions about whether data availability and labeling are the only limiting factors. The paper discusses the main contributors to these constraints and proposes potential solutions. The scaling hypothesis has become prevalent in AI research, with claims suggesting that continued scaling alone could lead to Artificial General Intelligence (AGI). In medicine, large multimodal foundation models have shown promise across medical specialties but have faced challenges when tested in realistic clinical settings. In surgery specifically, vision-language models have been applied to various tasks but whether these models can lead to Medical AGI remains uncertain. Despite advances in foundation models, practical experience suggests that performance on basic perceptual tasks remains limited under realistic distribution shifts due to gaps in domain-specific data coverage. The availability of clinically relevant data is believed to be more critical than model scale for advancing Surgical AI capabilities.