# PELA: Learning Parameter-Efficient Models with Low-Rank Approximation

**Authors:** Yangyang Guo, Guangzhi Wang, Mohan Kankanhalli  
**arXiv:** [2310.10700](https://arxiv.org/abs/2310.10700)  
**Date:** 2023-10-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.10700v1/

## Key points

- Authors Yangyang Guo, Guangzhi Wang, and Mohan Kankanhalli address the challenge of using pre-trained large models in resource-constrained environments.
- They propose a novel method involving an intermediate pre-training stage to increase parameter efficiency of pre-trained models.
- The approach includes low-rank approximation, feature distillation, and weight perturbation regularization to compress the original large model.
- During pre-training, only the compressed model is updated while backbone parameters are frozen for efficient resource utilization.
- Results demonstrate significant efficiencies in terms of required parameters and computation time without compromising task performance significantly.

## Summary

, , , , 

In their paper titled "PELA: Learning Parameter-Efficient Models with Low-Rank Approximation," authors Yangyang Guo, Guangzhi Wang, and Mohan Kankanhalli address the challenge of applying pre-trained large models to downstream tasks in resource-constrained environments. They propose a novel method that introduces an intermediate pre-training stage aimed at increasing the parameter efficiency of pre-trained models. This approach involves low-rank approximation to compress the original large model, along with specialized modules for feature distillation and weight perturbation regularization. During pre-training, only the compressed model is updated while keeping the backbone parameters frozen, allowing for efficient utilization of limited resources. The results show that this method can achieve significant efficiencies in terms of required parameters and computation time without compromising task performance significantly. Overall, "PELA" presents a promising strategy for enhancing parameter efficiency in pre-trained models, offering potential benefits for resource-constrained scenarios.