# FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition

**Authors:** Yuxuan Yan, Shunpu Tang, Zhiguo Shi, Qianqian Yang  
**arXiv:** [2404.18848](https://arxiv.org/abs/2404.18848)  
**Date:** 2024-04-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.18848v1/

## Key points

- FeDeRA is a groundbreaking study that addresses user privacy concerns in centralized training of Pre-trained Language Models (PLMs)
- Introduces FeDeRA as an improvement over existing methods like LoRA with exceptional performance in federated learning settings
- Utilizes Parameter-Efficient Fine-Tuning (PEFT) to tackle the burden on client computational resources in Federated Learning (FL)
- Proposes FeDeRA, which uses Singular Value Decomposition (SVD) on pre-trained matrices to select principal components for fine-tuning language models
- Outperformed all other PEFT methods and demonstrated comparable or superior performance to full parameter fine-tuning (FT) methods
- Significantly reduced training time by 95.9% to 97.9% compared to FT across different tasks using RoBERTa and DeBERTaV3

## Summary

FeDeRA: Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition is a groundbreaking study that addresses the challenges posed by user privacy concerns in centralized training of Pre-trained Language Models (PLMs). The study introduces FeDeRA as an improvement over existing methods like LoRA and showcases its exceptional performance while maintaining efficiency in federated learning settings. The escalating concerns surrounding extensive data collection have prompted the adoption of Federated Learning (FL) as a solution, where training occurs on client devices without sharing data. However, the substantial parameter size of PLMs places a burden on client computational resources and leads to costly communication expenses. To tackle this issue, Parameter-Efficient Fine-Tuning (PEFT) has been introduced into FL. Despite its effectiveness, non-IID data in federated learning results in performance gaps between PEFT and full parameter fine-tuning (FT). In response, the authors propose FeDeRA which utilizes an adapter module initialized through Singular Value Decomposition (SVD) on pre-trained matrices to select principal components. This innovative approach sets FeDeRA apart as a promising method for enhancing language model fine-tuning in FL scenarios. Extensive experiments were conducted using RoBERTa and DeBERTaV3 on three tasks across six datasets, comparing various methods including FT and other PEFT approaches. FeDeRA outperformed all other PEFT methods and demonstrated comparable or superior performance to FT methods. Additionally, federated learning was deployed on Jetson AGX Orin to compare training times for specific tasks. Compared to FT, FeDeRA significantly reduced training time by 95.9% to 97.9% across different tasks using RoBERTa and DeBERTaV3. In conclusion, FeDeRA offers an efficient solution for fine-tuning language models in federated learning settings. Its use of weight decomposition through SVD allows for improved performance while maintaining efficiency, making it a promising method for future FL applications.