# Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

**Authors:** Haoyuan Wu, Haisheng Zheng, Bei Yu  
**arXiv:** [2401.02731](https://arxiv.org/abs/2401.02731)  
**Date:** 2024-01-05  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.02731v1/

## Key points

- Large Language Models (LLMs) have impressive proficiency in general NLP tasks
- LLMs often face performance limitations due to constrained model capacity
- Parameter-Efficient Sparsity Crafting (PESC) is a novel approach introduced in this paper
- PESC uses a Mixture of Experts (MoE) architecture to transition dense models into sparse models
- Adapters are integrated into the MoE layers of sparse models to differentiate experts without altering individual weights
- PESC significantly reduces computational costs and GPU memory requirements
- PESC allows for model capacity expansion with minimal increase in parameters via inserted adapters
- Empirical evaluation demonstrates the effectiveness of the PESC method, named Camelidae
- Camelidae outperforms other open-source sparse models and exhibits superior capabilities compared to GPT3.5 during instruction tuning
- This research addresses the challenge of expanding model capacity during instruction tuning by leveraging a parameter-efficient sparsity crafting approach using MoE architecture

## Summary

Large Language Models (LLMs) have shown impressive proficiency in general natural language processing (NLP) tasks. However, these models often face performance limitations across multiple tasks due to their constrained model capacity. Expanding this capacity during the instruction tuning phase presents significant challenges. To overcome this issue, a novel approach called Parameter-Efficient Sparsity Crafting (PESC) is introduced in this paper. PESC utilizes a Mixture of Experts (MoE) architecture to transition dense models into sparse models. By integrating adapters into the MoE layers of sparse models, PESC is able to differentiate experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, allowing for model capacity expansion through a minimal increase in parameters via the inserted adapters. The effectiveness of the PESC method is demonstrated through empirical evaluation. When used during instruction tuning, sparse models created using PESC, named Camelidae, outperform all other open-source sparse models and exhibit superior general capabilities compared to GPT3.5. Overall, this research addresses the challenge of expanding model capacity during instruction tuning by introducing a parameter-efficient sparsity crafting approach that leverages a Mixture of Experts architecture. The results show that this method improves performance and enables sparse models to achieve superior capabilities compared to existing models like GPT3.5 in various NLP tasks.