# LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

**Authors:** Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, Yu Qiao  
**arXiv:** [2303.16199](https://arxiv.org/abs/2303.16199)  
**Date:** 2023-03-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.16199v1/

## Key points

- The LLaMA-Adapter is a novel adaptation method that fine-tunes the LLaMA language model into an instruction-following model.
- It introduces only 1.2 million learnable parameters and takes less than an hour for fine-tuning on eight A100 GPUs, making it a lightweight approach.
- The authors use 52K self-instruct demonstrations to train the model and adopt a set of learnable adaption prompts.
- They propose a zero-init attention mechanism with zero gating, which adaptively injects new instructional cues into LLaMA while preserving its pre-trained knowledge effectively.
- With efficient training, LLaMA-Adapter generates high-quality responses comparable to Alpaca with fully fine-tuned 7B parameters.
- This approach can be easily extended to multi-modal input such as images for image-conditioned LLaMA, which achieves superior reasoning capacity on ScienceQA.
- The authors have released their code at https://github.com/ZrrSkywalker/LLaMA-Adapter for public use.
- Overall, this study presents an innovative and efficient method for adapting language models into instruction following models using minimal additional parameters and training time.

## Summary

The LLaMA-Adapter is a novel adaptation method that efficiently fine-tunes the LLaMA language model into an instruction-following model. It introduces only 1.2 million learnable parameters upon the frozen LLaMA 7B model and takes less than an hour for fine-tuning on eight A100 GPUs, making it a lightweight approach. The authors use 52K self-instruct demonstrations to train the model and adopt a set of learnable adaption prompts, which they prepend to the input text tokens at higher transformer layers. They also propose a zero-init attention mechanism with zero gating, which adaptively injects new instructional cues into LLaMA while preserving its pre-trained knowledge effectively. With efficient training, LLaMA-Adapter generates high-quality responses comparable to Alpaca with fully fine-tuned 7B parameters. Furthermore, this approach can be easily extended to multi-modal input such as images for image-conditioned LLaMA, which achieves superior reasoning capacity on ScienceQA. The authors have released their code at https://github.com/ZrrSkywalker/LLaMA-Adapter for public use. Overall, this study presents an innovative and efficient method for adapting language models into instruction following models using minimal additional parameters and training time. This has significant implications for natural language processing tasks that require instruction following capabilities such as question answering systems or chatbots.