# Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning

**Authors:** Duarte M. Alves, Nuno M. Guerreiro, João Alves, José Pombal, Ricardo Rei, José G. C. de Souza, Pierre Colombo, André F. T. Martins  
**arXiv:** [2310.13448](https://arxiv.org/abs/2310.13448)  
**Date:** 2023-10-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.13448v1/

## Key points

- Large language models (LLMs) have challenges in machine translation (MT)
- LLM-based MT systems are often brittle and require post-processing
- They rely heavily on few-shot examples
- Finetuning on translation instructions is computationally expensive
- The paper proposes adapter-based finetuning with LoRA as a solution
- This method reduces training parameters by a factor of 50
- It outperforms few-shot prompting and eliminates the need for post-processing or in-context examples
- Finetuning generally degrades few-shot performance, limiting adaptation capabilities
- The authors propose incorporating few-shot examples during finetuning to overcome this limitation
- Experimental results on 10 language pairs show successful recovery of few-shot capabilities while retaining benefits of finetuning
- The proposed approach improves effectiveness and efficiency of LLM-based MT systems.

## Summary

, , , , 


Large language models (LLMs) have shown promise in machine translation (MT), but current LLM-based MT systems face challenges. They are often brittle and require additional post-processing due to overgeneration, relying heavily on the choice of few-shot examples. Finetuning on translation instructions is a computationally expensive alternative, which may weaken in-context learning capabilities. To address these issues, this paper proposes a closer examination of the problem and introduces a novel approach. The authors first demonstrate that adapter-based finetuning with LoRA achieves comparable performance to traditional finetuning while reducing the number of training parameters by a factor of 50. This method also outperforms few-shot prompting and eliminates the need for post-processing or in-context examples. However, they find that finetuning generally degrades few-shot performance, limiting adaptation capabilities. To overcome this limitation, the authors propose a simple solution that incorporates few-shot examples during finetuning. Experimental results on 10 language pairs show that their proposed approach successfully recovers the original few-shot capabilities while retaining the benefits of finetuning. In summary, this paper addresses the challenges faced by LLM-based MT systems by introducing adapter-based finetuning with LoRA and incorporating few-shot examples during finetuning. The proposed approach achieves comparable performance to traditional finetuning while reducing training parameters and maintaining adaptation capabilities. These findings contribute to improving the effectiveness and efficiency of LLM-based MT systems.