Results of the summarizing process for the arXiv paper: 2304.03245v1

In their paper titled "Large language models effectively leverage document-level context for literary translation, but critical errors persist," authors Marzena Karpinska and Mohit Iyyer explore the ability of large language models (LLMs) to translate paragraphs and documents. While LLMs have shown competitive performance on sentence-level translation datasets, evaluating their performance on larger units of text such as paragraphs and documents has been challenging due to the cost and difficulty involved in conducting such evaluations. The authors conduct a rigorous human evaluation of Gpt-3.5 (text-davinci-003) LLM's ability to translate an entire literary paragraph from novels across 18 linguistically-diverse language pairs including Japanese, Polish, and English. The evaluation involves hiring translators fluent in both the source and target languages to provide span-level error annotations as well as preference judgments of which system's translations are better. The results show that asking the LLM to translate an entire literary paragraph at once results in higher-quality translations than standard sentence-by-sentence translation approaches. Discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. However, critical errors still abound, including occasional content omissions that require human translator intervention to ensure that the author's voice remains intact. The evaluation took approximately 350 hours of effort for annotation and analysis, and the authors publicly release their dataset and error annotations to spur future research on evaluation of document-level literary translation. Overall, this study highlights the potential of LLMs in translating larger units of text while also emphasizing the need for continued improvement in machine translation technology to minimize critical errors.

• - Large language models (LLMs) are effective in translating paragraphs and documents
• - Evaluating LLMs' performance on larger units of text is challenging due to cost and difficulty
• - Authors conducted a human evaluation of Gpt-3.5 LLM's ability to translate literary paragraphs across 18 diverse languages
• - Results show that discourse-level LLM translators commit fewer errors than sentence-level approaches
• - Critical errors still exist, including occasional content omissions that require human intervention
• - The evaluation took approximately 350 hours of effort for annotation and analysis
• - The authors publicly release their dataset and error annotations to spur future research on evaluation of document-level literary translation

Summary: Large language models (LLMs) are good at translating big pieces of text. It's hard to test how well they work on bigger pieces because it takes a lot of time and money. Some people tested one LLM called Gpt-3.5 by having humans check its translations in 18 different languages. They found that when the LLM looks at whole paragraphs instead of just sentences, it makes fewer mistakes. But sometimes it still misses important parts and needs a human to fix it. The people who did the test shared their information so other people can use it to make better LLMs. Definitions- Large language models (LLMs): computer programs that can understand and generate human language - Translate: change words from one language into another - Paragraphs: groups of sentences that talk about the same topic - Discourse-level: looking at a whole piece of text and how all the parts relate to each other - Sentence-level: looking at individual sentences without thinking about how they fit together with others - Annotations: notes or comments added to a piece of text to explain or highlight something

Large Language Models Effectively Leverage Document-Level Context for Literary Translation, But Critical Errors Persist

In their paper titled "Large language models effectively leverage document-level context for literary translation, but critical errors persist," authors Marzena Karpinska and Mohit Iyyer explore the ability of large language models (LLMs) to translate paragraphs and documents. LLMs have shown competitive performance on sentence-level translation datasets, but evaluating their performance on larger units of text such as paragraphs and documents has been difficult due to the cost and difficulty involved in conducting such evaluations. To address this challenge, Karpinska and Iyyer conducted a rigorous human evaluation of Gpt-3.5 (text-davinci-003) LLM's ability to translate an entire literary paragraph from novels across 18 linguistically diverse language pairs including Japanese, Polish, and English.

Evaluation Methodology

Karpinska and Iyyer hired translators fluent in both the source and target languages to provide span-level error annotations as well as preference judgments of which system's translations are better. The evaluation took approximately 350 hours of effort for annotation and analysis. The authors also publicly released their dataset and error annotations to spur future research on evaluation of document-level literary translation.

Results

The results show that asking the LLM to translate an entire literary paragraph at once results in higher quality translations than standard sentence-by-sentence translation approaches. Discourse level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence level approaches. However, critical errors still abound including occasional content omissions that require human translator intervention to ensure that the author's voice remains intact.

Conclusion

Overall, this study highlights the potential of LLMs in translating larger units of text while also emphasizing the need for continued improvement in machine translation technology to minimize critical errors.