# BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention

**Authors:** Zhebin Zhang, Sai Wu, Dawei Jiang, Gang Chen  
**arXiv:** [2011.04266](https://arxiv.org/abs/2011.04266)  
**Date:** 2020-11-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2011.04266v1/

## Key points

- BERT-enhanced neural machine translation (NMT) leverages BERT-encoded representations for translation tasks
- A recently proposed approach combines attention mechanisms with Transformer's encoder and decoder layers, using BERT's last-layer representation to improve performance
- This method lacks flexibility in distributing attention between the BERT representation and the encoder/decoder representation
- The work introduces a novel BERT-enhanced NMT model called BERT-JAM that addresses these limitations
- BERT-JAM incorporates joint-attention modules to dynamically allocate attention between different representations, enhancing the model's ability to capture relevant information from both BERT and the encoder/decoder layers
- BERT-JAM utilizes BERT's intermediate representations by composing them with a gated linear unit (GLU), allowing effective use of these representations by the encoder/decoder layers
- A three-phase optimization strategy is employed to train BERT-JAM, progressively unfreezing different components for better convergence and improved performance
- Experimental results show that BERT-JAM achieves state-of-the-art BLEU scores on multiple translation tasks
- The model's dynamic allocation of attention and effective utilization of both BERT and intermediate representations contribute to its superior performance compared to existing approaches

## Summary

BERT-enhanced neural machine translation (NMT) aims to leverage BERT-encoded representations for translation tasks. A recently proposed approach combines attention mechanisms with Transformer's encoder and decoder layers, using BERT's last-layer representation to improve performance. However, this method lacks flexibility in distributing attention between the BERT representation and the encoder/decoder representation. In response, this work introduces a novel BERT-enhanced NMT model called BERT-JAM that addresses these limitations. BERT-JAM improves upon existing models in two key ways. Firstly, it incorporates joint-attention modules that allow the encoder/decoder layers to dynamically allocate attention between different representations. This enables more flexible distribution of attention and enhances the model's ability to capture relevant information from both BERT and the encoder/decoder layers. Secondly, BERT-JAM utilizes BERT's intermediate representations by composing them with a gated linear unit (GLU). This allows the encoder/decoder layers to make effective use of these intermediate representations, further enhancing translation performance. To train BERT-JAM, a novel three-phase optimization strategy is employed. This strategy progressively unfreezes different components of BERT-JAM during training, enabling better convergence and improved performance. Experimental results demonstrate that BERT-JAM achieves state-of-the-art BLEU scores on multiple translation tasks. The model's ability to dynamically allocate attention and effectively utilize both BERT and intermediate representations contributes to its superior performance compared to existing approaches. Overall, this work presents a comprehensive solution for enhancing neural machine translation using BERT-encoded representations. The proposed BERT-JAM model demonstrates significant improvements in translation quality by leveraging joint-attention modules and incorporating intermediate representations from BERT through a gated linear unit composition.