# LEGAL-BERT: The Muppets straight out of Law School

**Authors:** Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, Ion Androutsopoulos  
**arXiv:** [2010.02559](https://arxiv.org/abs/2010.02559)  
**Date:** 2020-10-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.02559v1/

## Key points

- Authors discuss the application of BERT models in the legal domain
- Limited exploration of BERT adaptation guidelines in specialized domains like law
- Three main strategies for applying BERT models to legal tasks: using original BERT, adapting with additional pre-training, and pre-training from scratch on domain-specific corpora
- Importance of considering specific requirements and characteristics of specialized domains during fine-tuning process
- Introduction of LEGAL-BERT models designed to assist in legal text analysis and processing of legal documents
- Tailored approaches needed when applying BERT models to specialized domains like law

## Summary

In their paper titled "LEGAL-BERT: The Muppets straight out of Law School," authors Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos discuss the application of BERT models in the legal domain. While BERT has shown impressive performance in various natural language processing (NLP) tasks, there has been limited exploration of its adaptation guidelines in specialized domains such as law. The authors specifically focus on the legal domain and investigate different approaches for applying BERT models to downstream legal tasks. They evaluate these approaches on multiple datasets and find that blindly following previous guidelines for pre-training and fine-tuning does not always yield satisfactory results in the legal domain. Therefore, they propose a systematic investigation of strategies when using BERT in specialized domains. The authors outline three main strategies: (a) using the original BERT model as is, (b) adapting BERT by additional pre-training on domain-specific corpora, and (c) pre-training BERT from scratch on domain-specific corpora. By exploring these strategies, they aim to provide better insights into effectively utilizing BERT models in the legal domain. Additionally, the authors suggest a broader hyper-parameter search space when fine-tuning BERT for downstream tasks. They emphasize the importance of considering specific requirements and characteristics of specialized domains during this process. To facilitate further research and applications in legal NLP, computational law, and legal technology, the authors introduce LEGAL-BERT—a family of BERT models designed to assist in these areas. This release aims to support advancements in legal text analysis and enable more accurate and efficient processing of legal documents. Overall, this paper highlights the need for tailored approaches when applying BERT models to specialized domains like law. The proposed strategies and LEGAL-BERT models contribute to advancing research efforts in legal NLP while addressing challenges specific to the legal domain.