# Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

**Authors:** Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, Christopher Ré  
**arXiv:** [2304.09433](https://arxiv.org/abs/2304.09433)  
**Date:** 2023-04-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.09433v2/

## Key points

- Long-standing goal: Develop automated systems for processing semi-structured documents without human effort or domain-specific customization
- Current systems rely on simplifying assumptions and domain-specific training
- Researchers explore the use of large language models (LLMs) for generality in this task
- Prototype system called EVAPORATE powered by LLMs
- Two strategies: direct extraction using LLMs or code synthesis
- Tradeoff between cost and quality: code synthesis is cheaper but less accurate than direct extraction with LLMs
- Introduction of EVAPORATE-CODE+ improves quality by generating multiple candidate functions and ensembling their extractions using weak supervision
- EVAPORATE-CODE+ outperforms state-of-the-art systems and reduces LLM processing load by 110 times on average
- Focus on converting heterogeneous data sources into structured tables for analytical queries (webpages, PDFs, text documents)
- Example application: structuring attributes related to medical devices in FDA 510(k) reviews for premarket notification submissions
- LLM-based systems effectively generate structured views of heterogeneous data

## Summary

A long-standing goal in the data management community is to develop automated systems that can process semi-structured documents and output queryable tables without the need for human effort or domain-specific customization. However, due to the wide variety of potential documents, current systems often make simplifying assumptions and rely on domain-specific training. In this study, the researchers explore whether large language models (LLMs) can help achieve generality in this task. The researchers propose a prototype system called EVAPORATE, which is powered by LLMs. They investigate two different strategies for implementing this system: directly extracting values from documents using LLMs or synthesizing code that performs the extraction. Through evaluations, they find that there is a tradeoff between cost and quality with these approaches. While code synthesis is cheaper, it is less accurate compared to directly processing each document with LLMs. To improve the quality while maintaining low cost, the researchers introduce an extended code synthesis implementation called EVAPORATE-CODE+. This approach achieves better quality than direct extraction by generating multiple candidate functions and ensembling their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms state-of-the-art systems but also reduces the number of tokens the LLM needs to process by 110 times on average across various evaluation settings. The study focuses on developing systems that can convert heterogeneous data sources into structured tables for analytical queries. These data sources include webpages, PDFs, and text documents. The proposed systems aim to identify the schema of these documents and perform extraction to populate tables automatically. As an example application, medical researchers often analyze data from electronic health records (EHR), clinical trials, PubMed knowledge sources, and FDA reports. The researchers consider FDA 510(k) reviews for premarket notification submissions as a motivating setting. Their objective is to automatically structure attributes related to medical devices in these reviews and output a table format. Overall, the study demonstrates that LLM-based systems can effectively generate structured views of heterogeneous data. The proposed EVAPORATE-CODE+ system achieves high-quality results while minimizing the processing load on the LLM.