# ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing

**Authors:** Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, Burkhard Rost  
**arXiv:** [2007.06225](https://arxiv.org/abs/2007.06225)  
**Date:** 2020-07-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2007.06225v3/

## Key points

- Authors Ahmed Elnaggar, Michael Heinzinger, and others explore computational biology and bioinformatics using Language Models (LMs) derived from Natural Language Processing (NLP).
- Experiments involved training auto-regressive models (Transformer-XL, XLNet) and auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD databases containing 393 billion amino acids.
- Training was conducted on the Summit supercomputer with 5616 GPUs and TPU Pod with up to 1024 cores.
- Protein LM-embeddings extracted from unlabeled data captured essential biophysical features in protein sequences.
- Impressive results were achieved in per-residue predictions of protein secondary structure (Q3=81%-87%), sub-cellular localization (Q10=81%), and distinguishing membrane-bound vs. water-soluble proteins (Q2=91%).
- ProtT5 embeddings showed transferability surpassing existing methods without relying on evolutionary information or costly database searches.
- The authors released their models on GitHub at https://github.com/agemagician/ProtTrans to support future research in this area.

## Summary

, , , , 

In their paper titled "ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing," authors Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost delve into the realm of computational biology and bioinformatics. They highlight the vast data gold-mines present in protein sequences that are ideal for Language Models (LMs) derived from Natural Language Processing (NLP). These LMs have the potential to push boundaries in new prediction tasks at low inference costs. The authors conducted experiments where they trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) using data sourced from UniRef and BFD databases containing a staggering 393 billion amino acids. This training was carried out on the powerful Summit supercomputer utilizing 5616 GPUs and TPU Pod with up to 1024 cores. Through dimensionality reduction techniques, they discovered that raw protein LM-embeddings extracted from unlabeled data managed to capture essential biophysical features inherent in protein sequences. Validation of these embeddings as exclusive inputs for various subsequent tasks showcased their effectiveness. The authors achieved impressive results in per-residue predictions of protein secondary structure with an accuracy range of Q3=81%-87%. Furthermore, they successfully predicted protein sub-cellular localization with a ten-state accuracy of Q10=81% and distinguished between membrane-bound and water-soluble proteins with a two-state accuracy of Q2=91%. Of particular note was the transferability of informative embeddings such as ProtT5 which surpassed existing state-of-the-art methods without relying on evolutionary information or costly database searches for per-residue predictions. These outcomes strongly suggested that protein LMs had acquired an understanding akin to the grammar embedded within the language of life. To aid future research endeavors in this domain, the authors generously released their models on GitHub at https://github.com/agemagician/ProtTrans. This comprehensive study spanning 17 pages featuring 9 figures and 4 tables sheds light on how self-supervised deep learning coupled with high-performance computing can potentially unlock profound insights into deciphering the intricate code governing life processes.