# LawBench: Benchmarking Legal Knowledge of Large Language Models

**Authors:** Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, Jidong Ge  
**arXiv:** [2309.16289](https://arxiv.org/abs/2309.16289)  
**Date:** 2023-09-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.16289v1/

## Key points

- Large language models (LLMs) have shown strong capabilities in various aspects
- Performance of LLMs in the legal domain is uncertain and safety-critical
- Authors propose a comprehensive evaluation benchmark called LawBench
- LawBench assesses LLMs' legal capabilities at three cognitive levels: knowledge memorization, understanding, and applying
- Benchmark consists of 20 tasks covering classification, regression, extraction, and generation
- Evaluations conducted on 51 LLMs including multilingual, Chinese-oriented, and legal-specific models
- GPT-4 performs best among LLMs in the legal domain by a significant margin
- Fine-tuning LLMs on legally specific text brings some improvements but more work is needed for usable and reliable LLMs in legal tasks
- LawBench provides an in-depth understanding of LLM capabilities and aims to accelerate their development in the legal domain
- All data, model predictions, and evaluation code are publicly available on GitHub
- Research contributes valuable insights into evaluating and advancing LLM performance in law.

## Summary

Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, their performance in the highly specialized and safety-critical legal domain remains uncertain. To address this gap, the authors propose a comprehensive evaluation benchmark called LawBench. LawBench aims to assess LLMs' legal capabilities at three cognitive levels: legal knowledge memorization, legal knowledge understanding, and legal knowledge applying. The benchmark consists of 20 diverse tasks covering single-label classification, multi-label classification, regression, extraction and generation. Extensive evaluations were conducted on 51 LLMs on LawBench including multilingual LLMs, Chinese-oriented LLMs and legal-specific LLMs. The results reveal that GPT-4 is the best performing LLM in the legal domain by a significant margin. Although fine-tuning LLMs on legally specific text brings some improvements there is still a long way to go in obtaining usable and reliable LLMs for legal tasks. The LawBench evaluation benchmark provides an in-depth understanding of the domain specified capabilities of LLMs and aims to accelerate their development in the legal domain. All data, model predictions and evaluation code are publicly available on GitHub. This research contributes valuable insights into evaluating and advancing LLM's performance in the complex field of law.