# PDFTriage: Question Answering over Long, Structured Documents

**Authors:** Jon Saad-Falcon, Joe Barrow, Alexa Siu, Ani Nenkova, Ryan A. Rossi, Franck Dernoncourt  
**arXiv:** [2309.08872](https://arxiv.org/abs/2309.08872)  
**Date:** 2023-09-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.08872v1/

## Key points

- Challenges faced by Large Language Models (LLMs) in document question answering when the document exceeds small context length
- Existing approaches focus on retrieving relevant context from plain text documents, but struggle with structured documents like PDFs, web pages, and presentations
- Introduction of PDFTriage, an approach leveraging both structure and content for context retrieval
- Comparison of PDFTriage with retrieval baselines such as Page Retrieval and Chunk Retrieval in experiments
- Utilization of PDF structure and GPT-3.5's interactive functions by PDFTriage for more accurate answer extraction
- Outperformance of PDFTriage in multi-page tasks like structure questions and table reasoning based on user preferences
- Human evaluation studies showing that PDFTriage provides high-quality answers compared to retrieval baselines
- Evaluation of attributes such as question difficulty, clarity, information needed for answering, and overall quality of answers by each system
- Highlighting the effectiveness of PDFTriage in handling structured documents for QA tasks where existing models fall short

## Summary

In this study, we address the challenges faced by Large Language Models (LLMs) in document question answering (QA) when the document exceeds the small context length of an LLM. Existing approaches focus on retrieving relevant context from plain text documents, but structured documents such as PDFs, web pages, and presentations pose a unique challenge due to their rich formatting and organization. To bridge this gap, we introduce PDFTriage, an approach that leverages both structure and content for context retrieval. Our dataset comprises 908 questions across 82 documents with an average of 4,257 tokens per document. In our experiments, we compare PDFTriage with retrieval baselines such as Page Retrieval and Chunk Retrieval. PDFTriage utilizes the structure of PDFs and GPT-3.5's interactive functions to extract answers more accurately than traditional methods. User preferences indicate that PDFTriage outperforms other approaches in multi-page tasks like structure questions and table reasoning. Human evaluation studies conducted on Upwork with experienced annotators show that PDFTriage excels in providing high-quality answers compared to retrieval baselines. The study evaluates attributes such as question difficulty, clarity, information needed for answering, and overall quality of answers generated by each system. Overall, our research highlights the effectiveness of PDFTriage in handling structured documents for QA tasks where existing models fall short. We provide detailed descriptions of our methodology and results along with a benchmark dataset for further research in this area.