# Better Call GPT, Comparing Large Language Models Against Lawyers

**Authors:** Lauren Martin, Nick Whitehouse, Stephanie Yiu, Lizzie Catterson, Rivindu Perera  
**arXiv:** [2401.16212](https://arxiv.org/abs/2401.16212)  
**Date:** 2024-01-24  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.16212v1/

## Key points

- Comparison between Large Language Models (LLMs) and traditional legal contract reviewers
- Objective: Determine if LLMs can outperform humans in accuracy, speed, and cost efficiency during contract review
- Analysis structured around providing context for each contract scenario and using a standardized contract review playbook
- Senior Lawyers evaluated contracts for adherence to predefined standards and identified specific influential sections
- Ground truth data collected to create benchmarks for accuracy, speed, and cost efficiency
- Duration of contract reviews recorded by Senior Lawyers, Junior Lawyers, Legal Process Outsourcers (LPOs), and LLMs compared
- Hourly rates for lawyers determined based on industry benchmark reports and market data held by Onit Inc.
- Costs for LLMs obtained through commercial pricing provided by service suppliers
- Prominent LLM models from OpenAI, Google Anthropic, Amazon Meta considered for analysis
- Advanced LLMs matched or exceeded human accuracy in determining legal issues
- LLMs completed reviews in seconds compared to hours required by humans
- LLMs operated at a fraction of the cost compared to traditional methods
- Findings indicate a seismic shift in the legal industry with potential for enhanced accessibility and efficiency of legal services.

## Summary

This paper presents a groundbreaking comparison between Large Language Models (LLMs) and traditional legal contract reviewers. The objective was to determine whether LLMs can outperform humans in terms of accuracy, speed, and cost efficiency during contract review. The analysis was structured around a two-factor approach: providing context for each contract scenario and using a standardized contract review playbook. This aimed to simulate real-world situations and ensure practical relevance. To establish ground truth data, Senior Lawyers evaluated each contract for adherence to predefined standards. They also identified specific sections that influenced their judgments. In cases where standards were not met due to missing information, Senior Lawyers explicitly recorded this at the end of their assessment. The collected data was aggregated to create benchmarks for accuracy, speed, and cost efficiency. Senior Lawyers also recorded the duration of each contract review to compare it with the time taken by Junior Lawyers, Legal Process Outsourcers (LPOs), and LLMs. Hourly rates for lawyers were determined based on industry benchmark reports and market data held by Onit Inc., while costs for LLMs were obtained through commercial pricing provided by service suppliers. In selecting models for analysis, prominent entities in the LLM space such as OpenAI, Google Anthropic, Amazon Meta were considered. Preliminary evaluations were conducted on models developed by these organizations to assess their applicability within the legal domain. These tests focused on analyzing reasoning capabilities and the ability to determine legal issues and their location within contracts. The results of the study showed that advanced LLMs matched or exceeded human accuracy in determining legal issues. In terms of speed, LLMs completed reviews in seconds compared to hours required by humans. Additionally, LLMs operated at a fraction of the cost, offering a significant reduction in expenses over traditional methods. These findings indicate a seismic shift in the legal industry, with LLMs poised to disrupt and enhance accessibility and efficiency of legal services. The research asserts that we are entering an era of LLM dominance in contract review, challenging the status quo and calling for a reimagined future of legal workflows.