# Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization

**Authors:** Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, Dongsoo Lee  
**arXiv:** [2305.14152](https://arxiv.org/abs/2305.14152)  
**Date:** 2023-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14152v1/

## Key points

- Parameter-efficient fine-tuning (PEFT) methods have been developed to address the high cost of full fine-tuning large language models (LLMs)
- A team of researchers has introduced Parameter-Efficient and Quantization-aware Adaptation (PEQA), a novel quantization-aware PEFT technique that enables model compression and accelerates inference
- PEQA operates through a dual-stage process: quantization into a matrix of low-bit integers and a scalar vector, followed by fine-tuning on the scalar vector for each downstream task
- This strategy significantly reduces the size of the model while maintaining performance upon deployment and reducing memory requirements
- Fast fine-tuning and efficient task switching become possible with PEQA
- The researchers conducted comprehensive experiments comparing PEQA with competitive baselines across natural language understanding to generation benchmarks using large language models up to 65 billion parameters
- The results demonstrate that PEQA is scalable, offers task-specific adaptation performance, and can follow instructions even in extremely low bit settings
- Overall, this new approach presents an effective solution for memory efficient fine tuning of compressed large language models via sub 4 bit integer quantization which facilitates more efficient deployment of LLMs in various applications such as natural language processing tasks.

## Summary

In recent years, parameter-efficient fine-tuning (PEFT) methods have been developed to address the high cost of full fine-tuning large language models (LLMs). To tackle this issue, a team of researchers has introduced Parameter-Efficient and Quantization-aware Adaptation (PEQA), a novel quantization-aware PEFT technique that enables model compression and accelerates inference. PEQA operates through a dual-stage process: first, the parameter matrix of each fully-connected layer undergoes quantization into a matrix of low-bit integers and a scalar vector; then, fine-tuning occurs on the scalar vector for each downstream task. This strategy significantly reduces the size of the model while maintaining performance upon deployment and reducing memory requirements. Additionally, fast fine-tuning and efficient task switching become possible with PEQA. The researchers conducted comprehensive experiments comparing PEQA with competitive baselines across natural language understanding to generation benchmarks using large language models up to 65 billion parameters. The results demonstrate that PEQA is scalable, offers task-specific adaptation performance, and can follow instructions even in extremely low bit settings. Overall, this new approach presents an effective solution for memory efficient fine tuning of compressed large language models via sub 4 bit integer quantization which facilitates more efficient deployment of LLMs in various applications such as natural language processing tasks.