# Finding the Optimal Vocabulary Size for Neural Machine Translation

**Authors:** Thamme Gowda, Jonathan May  
**arXiv:** [2004.02334](https://arxiv.org/abs/2004.02334)  
**Date:** 2020-04-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2004.02334v2/

## Key points

- Thamme Gowda and Jonathan May study neural machine translation (NMT) as a classification task within an autoregressive framework.
- Classifiers perform better when trained on balanced class distributions, but the Zipfian nature of languages introduces imbalanced classes in NMT.
- The researchers use two key statistics, Divergence (D) and Frequency at 95th% Class Rank (F95%), to quantify imbalance in class distributions.
- D measures deviation from a balanced distribution using Earth Mover Distance, while F95% identifies the least frequency in the 95th percentile of most frequent classes.
- Lower D values indicate more balanced class distribution, reducing errors due to class bias.
- F95% helps quantify the minimum number of training examples required for specific percentiles of classes while filtering out noise.
- The study explores the impact of various vocabulary sizes on NMT performance across multiple languages with varying data sizes.
- Insights are provided into why certain vocabulary sizes yield superior results and how imbalanced class distributions affect NMT outcomes.

## Summary

In their study titled "Finding the Optimal Vocabulary Size for Neural Machine Translation," Thamme Gowda and Jonathan May delve into the intricacies of neural machine translation (NMT) by framing it as a classification task within an autoregressive framework. They meticulously analyze the limitations posed by both classification and autoregression components, noting that classifiers typically perform better when trained on balanced class distributions. However, the Zipfian nature of languages introduces imbalanced classes, prompting the researchers to investigate its impact on NMT performance. To quantify this imbalance, Gowda and May employ two key statistics: Divergence (D) and Frequency at 95th% Class Rank (F95%). D measures the deviation from a balanced distribution using a simplified version of Earth Mover Distance. By calculating the total cost of moving probability mass between classes, they determine the imbalance measure D for K class distributions based on observed probabilities in training data. A lower value of D signifies a more balanced class distribution, reducing the likelihood of errors due to class bias. F95% identifies the least frequency in the 95th percentile of most frequent classes. This metric offers a straightforward approach to quantifying the minimum number of training examples required for specific percentiles of classes while filtering out noise from lower percentiles. Furthermore, Gowda and May explore the effect of various vocabulary sizes on NMT performance across multiple languages with varying data sizes. Through their analysis, they provide insights into why certain vocabulary sizes yield superior results compared to others. By systematically examining these factors, the researchers aim to enhance our understanding of how imbalanced class distributions impact NMT outcomes and offer valuable guidance for optimizing vocabulary size selection in neural machine translation systems.