# Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

**Authors:** Taku Kudo  
**arXiv:** [1804.10959](https://arxiv.org/abs/1804.10959)  
**Date:** 2018-04-29  
**Summary page:** https://www.summarizepaper.com/en/paper/1804.10959v1/

## Key points

- Subword units are effective for open vocabulary problems in neural machine translation (NMT)
- Subword segmentation can be ambiguous, leading to multiple possible segmentations
- The paper explores leveraging segmentation ambiguity as noise to enhance NMT model robustness
- Training the NMT model with multiple probabilistically sampled subword segmentations improves resilience and handling of unseen or out-of-domain words
- A new subword segmentation algorithm based on a unigram language model is introduced to improve sampling quality
- Experiments on multiple corpora consistently show improvements in translation performance, especially in low-resource and out-of-domain settings
- Harnessing subword segmentation ambiguity enhances robustness in NMT models and addresses open vocabulary challenges

## Summary

In the field of neural machine translation (NMT), subword units have proven to be an effective solution for dealing with open vocabulary problems. However, while sentences are typically converted into unique subword sequences, the process of subword segmentation can be ambiguous, leading to multiple possible segmentations even with the same vocabulary. This paper explores whether this segmentation ambiguity can be leveraged as a form of noise to enhance the robustness of NMT models. This approach involves training the NMT model with multiple subword segmentations that are probabilistically sampled during training. By exposing the model to different segmentations, it becomes more resilient to variations in input and is better able to handle unseen or out-of-domain words. Additionally, in order to improve the quality of subword sampling, the authors introduce a new subword segmentation algorithm based on a unigram language model. This algorithm helps ensure that the sampled segmentations are representative and capture meaningful linguistic patterns. The effectiveness of subword regularization and the new segmentation algorithm is evaluated through experiments on multiple corpora. The results consistently demonstrate improvements in translation performance, particularly in low-resource and out-of-domain settings where traditional NMT models often struggle. Overall, this paper highlights how harnessing the ambiguity inherent in subword segmentation can lead to enhanced robustness in NMT models and offers practical solutions for addressing open vocabulary challenges and improving translation quality across various domains and resource conditions.