# LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

**Authors:** Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, Min Lin  
**arXiv:** [2307.13269](https://arxiv.org/abs/2307.13269)  
**Date:** 2023-07-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.13269v1/

## Key points

- LoraHub framework enables cross-task generalization and adaptability in large language models (LLMs)
- Low-rank adaptations (LoRA) modules are strategically assembled in LoraHub
- LoraHub learning performs comparably or better than gradient-dependent methods in few-shot scenarios
- Investigation of different LoRA modules for tasks in the Big-Bench Hard (BBH) benchmark
- Five tasks identified with substantial influence and effective for cross-task transfer
- These tasks require higher-level skills such as reading comprehension and reasoning
- Contribution to the development of a community for LoRA, where users can share trained modules and advance general intelligence and LLMs in production.

## Summary

This paper presents an analysis of the LoraHub framework which enables cross-task generalization and adaptability in large language models (LLMs) through the strategic assembly of Low-rank adaptations (LoRA) modules. The authors compare LoraHub learning with LoRA tuning and conventional fine-tuning methods and find that it performs comparably or even better than gradient-dependent methods in few-shot scenarios. They also investigate the effectiveness of different LoRA modules for tasks in the Big-Bench Hard (BBH) benchmark, identifying five tasks that have substantial influence and are particularly effective for cross-task transfer. These tasks require higher-level skills such as reading comprehension and reasoning. This research contributes to the development of a community for LoRA where users can share their trained modules and advance general intelligence and LLMs in production.