# Neural Machine Translation with Byte-Level Subwords

**Authors:** Changhan Wang, Kyunghyun Cho, Jiatao Gu  
**arXiv:** [1909.03341](https://arxiv.org/abs/1909.03341)  
**Date:** 2019-09-07  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.03341v1/

## Key points

- Byte-level subwords as an alternative to character-based vocabularies in machine translation models
- Existing models have limitations with rare characters or character-rich languages
- Proposed solution: representing text at the byte level using a 256-byte set as the vocabulary
- Computational costs hinder widespread deployment and practical use of this approach
- Investigation of byte-level subwords known as byte-level BPE (BBPE)
- BBPE offers a more compact representation and eliminates out-of-vocabulary tokens compared to character vocabulary
- Contextualizing BBPE embeddings is necessary for optimal performance, achieved through convolutional or recurrent layers
- BBPE performs comparably to traditional BPE while significantly reducing its size (1/8th of BPE's size)
- BBPE facilitates effective vocabulary sharing across multiple languages in multilingual settings, improving translation quality
- BBPE enables model transfer between languages with non-overlapping character sets
- Byte-level subwords offer a promising solution for vocabulary limitations caused by rare characters or character-rich languages in machine translation models.

## Summary

In their paper titled "Neural Machine Translation with Byte-Level Subwords," authors Changhan Wang, Kyunghyun Cho, and Jiatao Gu explore the use of byte-level subwords as an alternative to character-based vocabularies in machine translation models. They highlight that existing models often suffer from limitations when dealing with rare characters or character-rich languages like Japanese and Chinese, which can unnecessarily occupy vocabulary slots and reduce compactness. To address this issue, the authors propose representing text at the byte level using a 256-byte set as the vocabulary. This approach has the potential to solve the problem but has been hindered by high computational costs, preventing widespread deployment and practical use. The authors specifically investigate byte-level subwords known as byte-level BPE (BBPE). BBPE offers a more compact representation compared to character vocabulary and eliminates out-of-vocabulary tokens. It is also more efficient than using pure bytes alone. However, they argue that contextualizing BBPE embeddings is necessary for optimal performance. This can be achieved through either a convolutional or recurrent layer. Through experiments, the authors demonstrate that BBPE performs comparably to traditional BPE while significantly reducing its size to only 1/8th of BPE's size. In multilingual settings, BBPE facilitates effective vocabulary sharing across multiple languages and leads to improved translation quality. Additionally, they show that BBPE enables model transfer between languages with non-overlapping character sets. Overall, this research presents byte-level subwords as a promising solution for machine translation models' vocabulary limitations caused by rare characters or character-rich languages. By utilizing BBPE and contextualizing its embeddings, researchers can achieve comparable performance to existing methods while significantly reducing computational costs and improving translation quality in multilingual scenarios.