In recent years, there has been a surge in the development of large language models (LLMs) that have been used as starting points for building chatbots and other applications. After the release of ChatGPT, the LLaMA model opened up opportunities for research on efficient fine-tuning, longer prompt context, retrieval augmented generation (RAG), and quantization. Several open access text-based LLMs have been introduced since then, including MosaicML's MPT, Together AI's RedPajama-INCITE, TII's Falcon, Meta's Llama 2, and Mistral 7B. Zephyr-7B is built upon Mistral 7B due to its strong performance. Researchers have also focused on improving small model performance by distilling knowledge from larger models. The self-instruct method and Alpaca model were among the early works in this area. Other models like Vicuna and WizardLM explored different approaches to distillation. In this work, the authors not only focus on distilling supervised fine-tuning (SFT) but also preference optimization. They compare their approach with Xwin-LM which uses PPO for preference optimization. To evaluate the performance of language models, various benchmarking tools have been developed. These tools use powerful LLMs such as GPT-4 and Claude to score or rank model responses in pairwise settings. Examples include the LMSYS chatbot arena benchmarks, AlpacaEval leaderboard, MTBench for multi-turn instructions evaluation across task categories, HuggingFace Open LLM leaderboard, Chain-of-Thought Hub, ChatEval, and FastEval. The goal of this work is to align an open-source large language model with user intent. The approach involves using a larger teacher model πT for prompted generation queries and producing a student model πθ through several stages similar to InstructGPT. Step 1 involves distilled supervised fine-tuning (dSFT), where the goal is to maximize the log likelihood of the student model's outputs compared to a dataset C consisting of input–output pairs (x , y). The detailed longer summary provides an overview of related work in the field including the development of open LLMs and approaches for improving small model performance through distillation. It also highlights benchmarking tools used for evaluating language models.
- Many people are working on creating chatbots and other applications that can understand and use language well.
- Researchers have been studying ways to make these language models more efficient and effective, like using longer prompts and retrieval augmented generation.
- Some new text-based language models have been made available for anyone to use, like MPT, RedPajama-INCITE, Falcon, Llama 2, Mistral 7B.
- Zephyr-7B is a model that was built based on the Mistral 7B model because it performed well.
- People are also trying to improve smaller models by distilling knowledge from larger models using methods like self-instruct and Alpaca model.
Definitions
- Language Models: Programs or systems that can understand and generate human language.
- Chatbots: Computer programs designed to simulate conversation with human users.
- Fine-tuning: Adjusting or improving a model's performance by making small changes or optimizations.
- Prompt context: The information or instructions given to a language model before it generates a response.
- Retrieval augmented generation (RAG): A technique where a language model retrieves relevant information from a database before generating its response.
- Open access: Available for anyone to use or access without restrictions or limitations.
- Benchmarking tools: Tools used to evaluate and compare the performance of different models or systems in specific tasks or areas.
Exploring the Potential of Large Language Models for Aligning with User Intent
In recent years, there has been a surge in the development of large language models (LLMs) that have opened up opportunities for research and applications such as chatbots. Several open access text-based LLMs have been introduced since then, including MosaicML's MPT, Together AI's RedPajama-INCITE, TII's Falcon, Meta's Llama 2, Mistral 7B and Zephyr-7B. Researchers have also focused on improving small model performance by distilling knowledge from larger models. This work focuses on aligning an open source large language model with user intent through distilled supervised fine-tuning (dSFT) and preference optimization.
Background: Open Access Text-Based LLMs
MosaicML’s MPT is an open access text-based LLM that was released after ChatGPT. It enables efficient fine-tuning and longer prompt context retrieval augmented generation (RAG). Following this release were other popular open access text based LLMs such as Together AI’s RedPajama INCITE, TII’s Falcon, Meta’s Llama 2 and Mistral 7B which Zephyr 7B was built upon due to its strong performance.
Improving Small Model Performance Through Distillation
The self instruct method and Alpaca model were among the early works in this area which explored different approaches to distillation for improving small model performance by extracting knowledge from larger models. Vicuna and WizardLM are two other examples of models that explore different approaches to distillation. In this work however, the authors focus not only on distilling supervised fine tuning but also preference optimization using Xwin LM which uses PPO for preference optimization.
Evaluating Language Models
Various benchmarking tools have been developed to evaluate language models using powerful LLMs such as GPT 4 or Claude to score or rank model responses in pairwise settings. Examples include LMSYS chatbot arena benchmarks; Alpaca Eval leaderboard; MTBench for multi turn instructions evaluation across task categories; HuggingFace Open LLM leaderboard; Chain of Thought Hub; ChatEval; FastEval etcetera .
Goal
The goal of this work is to align an open source large language model with user intent through distilled supervised fine tuning (dSFT) and preference optimization using Xwin LM which uses PPO for preference optimization .
Conclusion
This article provided an overview of related work in the field including the development of open access text based LLMs , approaches used for improving small model performance through distillation , benchmarking tools used for evaluating language models , as well as a discussion about aligning an open source large language model with user intent through dSFT & preference optimization .
We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.
