# SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation

**Authors:** Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, Jie Tang  
**arXiv:** [2406.14991](https://arxiv.org/abs/2406.14991)  
**Date:** 2024-06-21  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2406.14991v1/

## Key points

- SpreadsheetBench is a benchmark designed to test large language models (LLMs) in manipulating real-world spreadsheet scenarios
- Constructed from 912 real questions sourced from online Excel forums, reflecting complex user needs and challenges
- Benchmark includes diverse tabular data structures such as multiple tables, non-standard relational tables, and non-textual elements
- Proposed evaluation metric involves creating multiple spreadsheet files as test cases for each instruction to ensure robust solutions
- Significant performance gap between state-of-the-art models and human users revealed through comprehensive evaluation under single-round and multi-round inference settings
- Comparison with previous benchmarks like SheetCopilotBench highlights complexity and realism of instructions in SpreadsheetBench
- Spreadsheets in SpreadsheetBench feature flexible data organization with non-standard relational tables, textual information, and non-textual elements like colors
- Evaluation includes different types of models showing varying performance scores ranging from 0.05% to 23.65%
- Importance of enhancing coding abilities within LLMs for spreadsheet manipulation tasks emphasized
- Potential benefits of multi-round prompting to improve response accuracy highlighted

## Summary

In this study, the researchers introduce SpreadsheetBench, a challenging benchmark specifically designed to test the capabilities of large language models (LLMs) in manipulating real-world spreadsheet scenarios. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is constructed from 912 real questions sourced from online Excel forums. This reflects the complex needs and challenges faced by users. The associated spreadsheets in the benchmark contain diverse tabular data structures such as multiple tables, non-standard relational tables, and various non-textual elements. The researchers propose a more reliable evaluation metric inspired by online judge platforms. This involves creating multiple spreadsheet files as test cases for each instruction to ensure robust solutions capable of handling different types of spreadsheets. Through a comprehensive evaluation of various LLMs under both single-round and multi-round inference settings, the study reveals a significant performance gap between state-of-the-art models and human users. This underscores the difficulty of the benchmark. Furthermore, comparisons are made between previous benchmarks like SheetCopilotBench and SpreadsheetBench. These highlight the complexity and realism of instructions in SpreadsheetBench. The spreadsheets in SpreadsheetBench feature flexible data organization with non-standard relational tables and cells containing textual information and non-textual elements like colors. Additionally, manipulation categories involve spreadsheets with tables extending beyond 100 columns and 20,000 rows. The study evaluates different types of models including TableQA models, open-source LLMs for general tasks and coding tasks, advanced close-source models, and spreadsheet-specific LLMs. Results show varying performance scores ranging from 0.05% to 23.65% according to the proposed OJ-style evaluation metric. Some methods even score as low as 0%, emphasizing the difficulty of the benchmark. Overall, SpreadsheetBench stands out for its real-world instructions, diverse spreadsheet formats, and comprehensive testing strategy. The results suggest the importance of enhancing coding abilities within LLMs for spreadsheet manipulation tasks and highlight the potential benefits of multi-round prompting to improve response accuracy.