# InstructExcel: A Benchmark for Natural Language Instruction in Excel

**Authors:** Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri  
**arXiv:** [2310.14495](https://arxiv.org/abs/2310.14495)  
**Date:** 2023-10-23  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.14495v1/

## Key points

- The paper explores the capabilities of Large Language Models (LLMs) in solving complex Natural Language Processing (NLP) tasks within the domain of spreadsheets.
- Focuses on whether LLMs can generate code using Excel OfficeScripts to address specific Excel operations based on natural language user instructions.
- Introduces a large-scale benchmark called InstructExcel comprising over 10,000 samples encompassing 170+ Excel operations extracted from 2,000 publicly available Excel spreadsheets.
- Experimental results show that InstructExcel poses a significant challenge for state-of-the-art models like GPT-4.
- Findings suggest that utilizing GPT-4 and providing more contextual examples can enhance model performance on this benchmark.
- Employing dynamic prompting strategies can improve model performance on challenging benchmarks like InstructExcel.

## Summary

The paper "InstructExcel: A Benchmark for Natural Language Instruction in Excel" by Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme and Elnaz Nouri explores the capabilities of Large Language Models (LLMs) in solving complex Natural Language Processing (NLP) tasks within the domain of spreadsheets. The study focuses on whether LLMs can generate code using Excel OfficeScripts (a TypeScript API for executing tasks in Excel) to address specific Excel operations based on natural language user instructions. To facilitate this investigation, the authors introduce a novel large-scale benchmark called InstructExcel. This benchmark comprises over 10,000 samples encompassing 170+ Excel operations extracted from 2,000 publicly available Excel spreadsheets. The dataset was created by leveraging the 'Automate' feature in Excel to automatically generate OfficeScripts from users' actions. Experimental results across various zero-shot and few-shot settings demonstrate that InstructExcel poses a significant challenge for state-of-the-art models like GPT-4. The findings suggest that utilizing GPT-4 instead of GPT-3.5 and providing more contextual examples can enhance model performance on this benchmark. Additionally, employing dynamic prompting strategies can also improve model performance on challenging benchmarks like InstructExcel. Overall,the study highlights the potential of LLMs in generating code for Excel tasks based on natural language instructions and emphasizes the importance of tailored approaches to improve model performance on challenging benchmarks like InstructExcel.