# Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities

**Authors:** Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, Hung-yi Lee  
**arXiv:** [2310.04799](https://arxiv.org/abs/2310.04799)  
**Date:** 2023-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.04799v1/

## Key points

- Paper title: "Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities"
- Authors: Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, and Hung-yi Lee
- Focus on developing Large Language Models (LLMs) for non-English languages and aligning them with human preferences
- Introduction of chat vectors to enhance LLM performance by incorporating pre-existing knowledge and behaviors
- Replacement of traditional training paradigm with continual pre-training combined with chat vectors
- Empirical studies primarily on Traditional Chinese language models using LLaMA2 as the base model
- Evaluation of chat vectors effectiveness in terms of toxicity levels, accuracy in following instructions, and engagement in multi-turn dialogues
- Significant improvement in LLM chatting capabilities observed with incorporation of chat vectors
- Extension of experiments to Korean and Simplified Chinese models to validate adaptability across different languages

## Summary

The paper "Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities" by Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, and Hung-yi Lee delves into the advancements in conversational AI. The authors focus on developing Large Language Models (LLMs) for non-English languages and aligning them with human preferences. They introduce a novel method using chat vectors to enhance LLM performance by incorporating pre-existing knowledge and behaviors. This approach replaces the traditional training paradigm of continual pre-training followed by SFT and RLHF with continual pre-training combined with chat vectors. Empirical studies primarily concentrate on Traditional Chinese language models using LLaMA2 as the base model. The effectiveness of chat vectors is evaluated in terms of toxicity levels, accuracy in following instructions, and engagement in multi-turn dialogues. Results show that incorporating chat vectors significantly improves LLM chatting capabilities. Experiments are extended to Korean and Simplified Chinese models to validate the adaptability of this approach across different languages. Overall, this paper presents an efficient solution for aligning Large Language Models with human preferences in non-English language contexts through the use of chat vectors.