# Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

**Authors:** Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, Dongmei Zhang  
**arXiv:** [2305.13062](https://arxiv.org/abs/2305.13062)  
**Date:** 2023-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.13062v4/

## Key points

- Large language models (LLMs) are increasingly used as few-shot reasoners in Natural Language (NL)-related tasks
- Lack of comprehensive studies on how well LLMs comprehend structured data, particularly tables
- Benchmark designed to evaluate structural understanding capabilities (SUC) of LLMs through seven tasks with unique challenges
- Performance of LLMs varied based on input choices such as table input format, content order, role prompting, and partition marks
- Proposal of "self-augmentation" for effective structural prompting by utilizing internal knowledge of LLMs for critical value/range identification
- Importance of in-context learning for understanding structural information within tables highlighted by significant drop in system performance in zero-shot setting compared to one-shot setting when using HTML format for all tasks
- Placing external information ahead of tables resulted in improved performance across all tasks by providing better generalization and context regarding the structural information
- Investigation into most effective input designs for enabling LLMs to understand tables due to chaotic landscape of varied input designs used in previous work
- Strategic input design choices coupled with self-augmented prompting methods can significantly enhance LLM performance on tabular tasks

## Summary

, , , , 

In the realm of Natural Language (NL)-related tasks, Large language models (LLMs) are increasingly being utilized as few-shot reasoners. However, there is still much to be understood about how well LLMs comprehend structured data, particularly tables. While tables can be inputted into LLMs through serialization, there is a lack of comprehensive studies examining whether LLMs truly grasp such data. To address this gap, a benchmark was designed to evaluate the structural understanding capabilities (SUC) of LLMs. This benchmark consists of seven tasks with unique challenges such as cell lookup, row retrieval, and size detection. Through evaluations on GPT-3.5 and GPT-4, it was found that the performance of LLMs varied based on several input choices including table input format, content order, role prompting, and partition marks. Insights gained from these evaluations led to the proposal of "self-augmentation" for effective structural prompting by utilizing internal knowledge of LLMs for critical value/range identification. When combined with carefully chosen input choices, these structural prompting methods showed promising improvements in LLM performance across various tabular tasks. Further analysis revealed that system performance significantly dropped in a zero-shot setting compared to a one-shot setting when using HTML format for all tasks. This highlights the importance of in-context learning for understanding structural information within tables. Additionally, placing external information ahead of tables resulted in improved performance across all tasks by providing better generalization and context regarding the structural information. The chaotic landscape of varied input designs used in previous work necessitated an investigation into the most effective input designs for enabling LLMs to understand tables. The proposed SUC benchmark aimed to compare various input designs and assess LLMs' structural understanding capabilities through specific tasks focusing on each capability. By conducting experiments with different prompt variants and offering pragmatic guidance on leveraging LLMs for structured data comprehension, this study contributes valuable insights for future research in this field. Overall, the findings suggest that strategic input design choices coupled with self-augmented prompting methods can enhance LLM performance on tabular tasks significantly. This work not only sheds light on improving LLM comprehension of structured data but also provides a foundation for further exploration and development in this area.