# Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

**Authors:** Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Chenhao Ma, Kevin C. C. Chang, Fei Huang, Reynold Cheng, Yongbin Li  
**arXiv:** [2305.03111](https://arxiv.org/abs/2305.03111)  
**Date:** 2023-05-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.03111v1/

## Key points

- Growing interest in text-to-SQL parsing
- Advancements made by models like Codex and ChatGPT
- Existing benchmarks like Spider and WikiSQL focus on database schema with limited contents
- Introduction of the Bird benchmark catering to large-scale databases and text-to-SQL tasks
- Key features of Bird emphasizing database values, dealing with dirty contents, incorporating external knowledge, and optimizing SQL efficiency
- Importance of considering database values in generating accurate text-to-SQL queries for large databases
- Execution accuracy gap between state-of-the-art models and human performance
- Efficiency analysis provided by Bird for generating efficient text-to-SQL queries
- Belief that Bird benchmark will advance practical applications of text-to-SQL research

## Summary

, , , , 

In recent years, there has been a growing interest in text-to-SQL parsing, which involves converting natural language instructions into executable SQL queries. Notably, advancements in this field have been made by models such as Codex and ChatGPT, showcasing impressive results. However, existing benchmarks like Spider and WikiSQL primarily focus on database schema with limited database contents, creating a gap between academic research and real-world applications. To address this disparity, a new benchmark called Bird has been introduced. The Bird benchmark is designed to cater to large-scale databases and text-to-SQL tasks, comprising 12,751 pairs of data and 95 databases totaling 33.4 GB across 37 professional domains. One of the key features of Bird is its emphasis on database values, presenting unique challenges such as dealing with dirty database contents, incorporating external knowledge from NL questions into SQL queries, and optimizing SQL efficiency for massive databases. In order to effectively tackle these challenges, text-to-SQL models need to excel not only at semantic parsing but also demonstrate proficiency in understanding database values. Experimental results from Bird underscore the importance of considering database values in generating accurate text-to-SQL queries for large databases. Despite the progress made by state-of-the-art models like ChatGPT, which achieves an execution accuracy of 40.08%, there is still a significant gap compared to human performance at 92.96%. Additionally, Bird offers an efficiency analysis that provides valuable insights for generating efficient text-to-SQL queries beneficial for industries. The creators of Bird believe that this benchmark will play a crucial role in advancing the practical applications of text-to-SQL research. For those interested in exploring further details or accessing the leaderboard and source code for Bird, visit https://bird-bench.github.io/.