# Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks

**Authors:** Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, Rongrong Ji  
**arXiv:** [2204.07780](https://arxiv.org/abs/2204.07780)  
**Date:** 2022-04-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2204.07780v1/

## Key points

- Transformer model has impressive performance but criticized for large number of parameters and high computation cost
- Compressing the Transformer model is challenging due to complex layer designs
- Group-wise Transformation is introduced as a new approach to create a lightweight version of the Transformer model called LW-Transformer
- LW-Transformer successfully reduces parameters and computations while preserving efficient attention modeling and feature transformation
- LW-Transformer achieves competitive performance compared to original Transformer networks in vision-and-language tasks
- LW-Transformer's optimization strategy enhances performance in different models beyond vision-and-language tasks, such as Swin-Transformer in image classification.

## Summary

The Transformer model has gained attention for its impressive performance in various tasks. However, it has been criticized for its large number of parameters and high computation cost. Compressing the Transformer model has proven to be challenging due to the complex layer designs, specifically the Multi-Head Attention (MHA) and Feed-Forward Network (FFN). To tackle this issue, a new approach called Group-wise Transformation is introduced in this paper. The proposed method aims to create a lightweight version of the Transformer model for vision-and-language tasks, known as LW-Transformer. By applying Group-wise Transformation, LW-Transformer successfully reduces both the parameters and computations required by the Transformer model while preserving its efficient attention modeling on diverse subspaces of MHA and expanding-scaling feature transformation of FFN. The effectiveness of LW-Transformer is evaluated on three vision-and-language tasks and six benchmark datasets. Experimental results demonstrate that LW-Transformer achieves competitive performance compared to the original Transformer networks while significantly reducing parameters and computations. This highlights the potential of LW-Transformer as a more efficient alternative for vision-and-language tasks. Furthermore, the generalization ability of LW-Transformer is examined by applying the optimization strategy to another image classification model called Swin-Transformer. The results confirm that LW-Transformer's optimization strategy can also enhance the performance of different models beyond vision-and-language tasks. In conclusion, this paper presents an innovative approach, Group-wise Transformation, which enables the development of a lightweight version of the Transformer model called LW-Transformer. The experimental results showcase its effectiveness in reducing parameters and computations without compromising performance in vision-and-language tasks. Additionally, its optimization strategy proves beneficial for other models like Swin-Transformer in image classification.