# Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks

**Authors:** Hyunjin Choi, Judong Kim, Seongho Joe, Youngjune Gwon  
**arXiv:** [2101.10642](https://arxiv.org/abs/2101.10642)  
**Date:** 2021-01-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2101.10642v1/

## Key points

- The paper evaluates the performance of BERT and ALBERT models in sentence embedding for downstream NLP tasks.
- A modified BERT network called Sentence-BERT (SBERT) is introduced, and BERT is replaced with ALBERT to create Sentence-ALBERT (SALBERT).
- An outer CNN sentence-embedding network is experimented with for both SBERT and SALBERT.
- Performance evaluation is done using semantic textual similarity (STS) and natural language inference (NLI) datasets.
- CNN architecture significantly improves ALBERT models more than BERT models in the STS benchmark.
- Despite having fewer model parameters, ALBERT sentence embeddings are highly competitive to BERT in downstream NLP evaluations.
- Comprehensive evaluation is conducted on STS tasks from 2012 to 2016 after fine-tuning with NLI and STSb train sets.
- CNN architecture improves ALBERT-based sentence embedding models substantially more than BERT-based ones.
- Contextualized representations from pre-trained language models are important for high performance in downstream NLP tasks.
- Further research is needed to explore optimal sentence embedding schemes in computational linguistics.

## Summary

The paper focuses on evaluating the performance of BERT and ALBERT models in sentence embedding for downstream NLP tasks. The authors introduce a modified BERT network called Sentence-BERT (SBERT) and replace BERT with ALBERT to create Sentence-ALBERT (SALBERT). They also experiment with an outer CNN sentence-embedding network for both SBERT and SALBERT. The performances of all the sentence-embedding models are evaluated using the semantic textual similarity (STS) and natural language inference (NLI) datasets. The results show that the CNN architecture significantly improves ALBERT models more than BERT models in the STS benchmark. Despite having fewer model parameters, ALBERT sentence embeddings are highly competitive to BERT in downstream NLP evaluations. Furthermore, a comprehensive evaluation is conducted on the STS tasks from 2012 to 2016 after fine-tuning with both the NLI and STSb train sets. The results demonstrate that CNN architecture improves ALBERT-based sentence embedding models substantially more than BERT-based ones. Overall, this study highlights the importance of contextualized representations from pre-trained language models for achieving high performance in downstream NLP tasks. It also suggests that further research is needed to explore optimal sentence embedding schemes in computational linguistics.