# Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

**Authors:** Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, Fu Lee Wang  
**arXiv:** [2312.12148](https://arxiv.org/abs/2312.12148)  
**Date:** 2023-12-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.12148v1/

## Key points

- Pretrained Language Models (PLMs) and their fine-tuning through Parameter Efficient Fine-Tuning (PEFT) methods are discussed in the paper.
- PEFT reduces the number of fine-tuning parameters and memory usage while maintaining performance comparable to full fine-tuning, addressing challenges in resource-constrained environments.
- The development of PEFT methods has increased due to the demand for fine-tuning PLMs, especially Large Language Models (LLMs).
- Various PEFT methods are categorized into additive fine-tuning, partial fine-tuning, reparameterized fine-tuning, hybrid fine-tuning, and unified fine-tuning to establish a structured framework for understanding these approaches.
- The paper provides quantitative investigations and analyses of representative PEFT methods to understand their effectiveness in parameter efficiency and memory efficiency.

## Summary

, , , , 

In this paper, the authors delve into the realm of Pretrained Language Models (PLMs) and their fine-tuning through Parameter Efficient Fine-Tuning (PEFT) methods in Natural Language Processing (NLP). The exponential growth in the number of parameters in transformer-based PLMs, especially with the emergence of Large Language Models (LLMs), has led to a surge in successful NLP tasks. However, the sheer size and computational demands of these models present challenges when adapting them to specific downstream tasks, particularly in resource-constrained environments. PEFT offers a solution by reducing the number of fine-tuning parameters and memory usage while maintaining performance comparable to full fine-tuning. The demand for fine-tuning PLMs, especially LLMs, has resulted in an increase in the development of PEFT methods as depicted in Fig. 1. This paper provides a comprehensive review and systematic analysis of various PEFT methods for PLMs. The authors summarize these methods, discuss their applications, and outline future directions in Section III. Furthermore, they categorize PEFT methods into additive fine-tuning, partial fine-tuning, reparameterized fine-tuning, hybrid fine-tuning, and unified fine-tuning to establish a structured framework for understanding these approaches as shown in Fig. 2. In Section IV, quantitative investigations and analyses are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications of PEFT methods for PLMs in NLP tasks, this survey serves as a valuable resource for researchers and practitioners navigating the challenges and opportunities presented by PEFT. This study aims to provide a detailed exploration of PEFT methods for PLMs while also highlighting their significance in addressing computational resource constraints and enhancing performance on downstream tasks.