# Sabiá: Portuguese Large Language Models

**Authors:** Ramon Pires, Hugo Abonizio, Thales Rogério, Rodrigo Nogueira  
**arXiv:** [2304.07880](https://arxiv.org/abs/2304.07880)  
**Date:** 2023-04-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.07880v1/

## Key points

- Authors Ramon Pires, Hugo Abonizio, Thales Rogério, and Rodrigo Nogueira from Maritaca AI explore advancements in language models
- Monolingual pretraining on the target language (Portuguese) significantly enhances models trained on diverse corpora
- Pretrained GPT-J and LLaMA models using only 3% or less of their original pretraining budget on Portuguese texts
- Models outperform English-centric and multilingual counterparts by a significant margin in few-shot evaluations on Poeta datasets
- Best model Sabiá-65B performs comparably to GPT-3.5-turbo
- Language-specific pretraining helps capture linguistic nuances and structures unique to the target language
- Domain-specific knowledge acquired through monolingual pretraining contributes significantly to performance improvement
- Continuing pretraining with moderately-sized language-specific corpora can enhance a model's ability to capture cultural and knowledge richness inherent in individual languages

## Summary

In their paper "Sabiá: Portuguese Large Language Models," authors Ramon Pires, Hugo Abonizio, Thales Rogério, and Rodrigo Nogueira from Maritaca AI explore the advancements in language models. They challenge the prevailing practice of training single models on multiple languages simultaneously and demonstrate that monolingual pretraining on the target language - in this case Portuguese - significantly enhances models that have already been extensively trained on diverse corpora. Specifically, they further pretrain GPT-J and LLaMA models using only 3% or less of their original pretraining budget on Portuguese texts. Through few-shot evaluations on Poeta - a suite of 14 Portuguese datasets - the authors show that their models outperform English-centric and multilingual counterparts by a significant margin. Their best model, Sabiá-65B, performs comparably to GPT-3.5-turbo. By evaluating datasets originally conceived in Portuguese as well as translated ones, they analyze the impact of language-specific pretraining in capturing linguistic nuances and structures unique to the target language. The results indicate that domain-specific knowledge acquired through monolingual pretraining contributes significantly to the model's performance improvement. This enriches the model's knowledge about specific domains or cultures. The authors argue that while multilingual models have shown success in various tasks, continuing pretraining with moderately-sized language-specific corpora can enhance a model's ability to capture cultural and knowledge richness inherent in individual languages. This additional pretraining not only improves performance compared to multilingual models but also highlights the importance of tailoring language models to specific linguistic contexts for optimal results in natural language processing tasks.