# Not All Attention Is All You Need

**Authors:** Hongqiu Wu, Hai Zhao, Min Zhang  
**arXiv:** [2104.04692](https://arxiv.org/abs/2104.04692)  
**Date:** 2021-04-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.04692v3/

## Key points

- Pre-trained language models (PrLMs) have achieved remarkable success in natural language processing (NLP) tasks
- Large model size makes them susceptible to over-fitting which can affect their performance
- Dropout has been used as a solution but existing methods are not very effective on self-attention based models that are commonly used as the fundamental architecture of PrLMs
- Authors propose a novel dropout method called AttendOut that enables self-attention empowered PrLMs to be more robust in task-specific tuning
- State-of-the-art models with elaborate training design can achieve much stronger results using AttendOut
- The universality of the approach is verified through extensive experiments on different NLP tasks
- This paper provides valuable insights into improving the performance of PrLMs and highlights the importance of developing specialized techniques for specific architectures.

## Summary

In recent years, pre-trained language models (PrLMs) have achieved remarkable success in natural language processing (NLP) tasks. However, their large model size makes them susceptible to over-fitting which can affect their performance. To address this issue, dropout has been used as a solution but existing methods such as random-based, knowledge-based and search-based dropout are not very effective on self-attention based models that are commonly used as the fundamental architecture of PrLMs. In this paper titled "Not All Attention Is All You Need," authors Hongqiu Wu, Hai Zhao and Min Zhang propose a novel dropout method called AttendOut that enables self-attention empowered PrLMs to be more robust in task-specific tuning. The authors demonstrate that state-of-the-art models with elaborate training design can achieve much stronger results using AttendOut. The universality of the approach is verified through extensive experiments on different NLP tasks. This paper provides valuable insights into improving the performance of PrLMs and highlights the importance of developing specialized techniques for specific architectures.