# Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

**Authors:** Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, Yinfei Yang  
**arXiv:** [2108.08877](https://arxiv.org/abs/2108.08877)  
**Date:** 2021-08-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2108.08877v3/

## Key points

- The paper explores three methods for extracting sentence embeddings from text-to-text transformers (T5).
- Sentence embeddings are useful for language processing tasks.
- The authors propose a new benchmark called SentGLUE to evaluate sentence embeddings.
- Encoder-only models outperform existing sentence embedding models on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS).
- Scaling up T5 from millions to billions of parameters produces further improvements.
- The encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings.
- The study provides valuable insights into producing high quality sentence embeddings using pre-trained text-to-text models like T5.
- The authors have released their models at https://tfhub.dev/google/collections/sentence-t5/1 for use by other researchers in natural language processing tasks.

## Summary

In the paper titled "Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models" by Jianmo Ni et al., the authors explore three methods for extracting sentence embeddings from text-to-text transformers (T5). They highlight that sentence embeddings are useful for language processing tasks and propose a new benchmark called SentGLUE to evaluate them. The authors' encoder-only models outperform existing sentence embedding models on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS). Furthermore, scaling up T5 from millions to billions of parameters produces further improvements. The encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings. This study provides valuable insights into producing high quality sentence embeddings using pre-trained text-to-text models like T5 and the authors have released their models at https://tfhub.dev/google/collections/sentence-t5/1 for use by other researchers in natural language processing tasks.