# AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

**Authors:** Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao  
**arXiv:** [2303.10512](https://arxiv.org/abs/2303.10512)  
**Date:** 2023-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.10512v2/

## Key points

- Fine-tuning large pre-trained language models on downstream tasks is crucial in NLP.
- Conventional fine-tuning of all parameters in a pre-trained model becomes impractical for numerous tasks.
- Various methods, such as AdaLoRA, facilitate incremental updates of pre-trained weights more efficiently.
- AdaLoRA dynamically allocates the parameter budget based on importance scores using singular value decomposition and pruning unimportant updates.
- Extensive experiments across various NLP tasks showed significant improvements with AdaLoRA, especially in limited budget scenarios.
- The code for AdaLoRA is available at https://github.com/QingruZhang/AdaLoRA.

## Summary

In the field of Natural Language Processing (NLP), fine-tuning large pre-trained language models on downstream tasks has emerged as a crucial practice. However, the conventional approach of fine-tuning all parameters in a pre-trained model becomes impractical when dealing with numerous downstream tasks. To address this challenge, various fine-tuning methods have been proposed to facilitate incremental updates of pre-trained weights in a more parameter-efficient manner, such as utilizing low-rank increments. One such method is AdaLoRA, which dynamically allocates the parameter budget among weight matrices based on their respective importance scores. By leveraging singular value decomposition and pruning unimportant updates, AdaLoRA effectively reduces the parameter budget allocated to less critical updates while avoiding computationally intensive exact SVD computations. Extensive experiments were conducted using several pre-trained models across tasks such as natural language processing, question answering, and natural language generation to evaluate the efficacy of AdaLoRA. The results demonstrated significant improvements over baseline methods, particularly in scenarios with limited budgets. The code for AdaLoRA is publicly accessible at https://github.com/QingruZhang/AdaLoRA. This study was authored by Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen and Tuo Zhao and was presented at the 11th International Conference on Learning Representations (ICLR 2023).