# AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

**Authors:** Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha  
**arXiv:** [2108.05542](https://arxiv.org/abs/2108.05542)  
**Date:** 2021-08-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2108.05542v1/

## Key points

- The paper provides an overview of transformer-based pretrained language models (T-PTLMs) and their success in various natural language processing (NLP) tasks.
- T-PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks.
- The paper explains core concepts like pretraining, pretraining methods, pretraining tasks, embeddings, and downstream adaptation methods.
- The authors present a new taxonomy of T-PTLMs and give a brief overview of various benchmarks including both intrinsic and extrinsic evaluations.
- The paper presents a summary of various useful libraries for working with T-PTLMs.
- Future research directions that can further improve these models are highlighted such as improving the efficiency of training large scale models or developing better evaluation metrics for NLP tasks.
- This comprehensive survey paper serves as an excellent reference for anyone interested in understanding the core concepts behind T-PTLMs or staying updated with recent developments in this field.

## Summary

The paper "AMMUS: A Survey of Transformer-based Pretrained Models in Natural Language Processing" by Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha provides a comprehensive overview of transformer-based pretrained language models (T-PTLMs) and their success in various natural language processing (NLP) tasks. T-PTLMs are built on top of transformers, self-supervised learning, and transfer learning techniques. They learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. This approach provides good background knowledge to downstream tasks which avoids training them from scratch. The paper begins with a brief introduction to self-supervised learning followed by an explanation of various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings, and downstream adaptation methods. The authors then present a new taxonomy of T-PTLMs and give a brief overview of various benchmarks including both intrinsic and extrinsic evaluations. Additionally, the paper presents a summary of various useful libraries for working with T-PTLMs. The authors highlight some future research directions that can further improve these models such as improving the efficiency of training large scale models or developing better evaluation metrics for NLP tasks. This comprehensive survey paper serves as an excellent reference for anyone interested in understanding the core concepts behind T-PTLMs or staying updated with recent developments in this field. It provides valuable insights into how T-PTLMs have revolutionized NLP tasks and how they continue to evolve with ongoing research efforts. Overall, this paper is an essential resource for anyone looking to gain an understanding of the fundamentals behind T-PTLMs or stay abreast with recent advancements in this field.