# RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment

**Authors:** Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, Yuandong Tian  
**arXiv:** [2307.12950](https://arxiv.org/abs/2307.12950)  
**Date:** 2023-07-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.12950v1/

## Key points

- The paper proposes a method called Reinforcement Learning from Contrast Distillation (RLCD) for aligning language models with natural language principles without human feedback.
- RLCD trains a preference model using simulated preference pairs generated through contrasting positive and negative prompts.
- The preference model is used to enhance an unaligned base language model through reinforcement learning.
- RLCD is compared against two baselines: RLAIF and context distillation, in three alignment tasks: harmlessness, helpfulness, and story outline generation.
- RLCD outperforms the baseline methods in all tested alignment tasks.
- RLCD shows promise in improving the quality and alignment of language models by leveraging contrast distillation and reinforcement learning techniques.

## Summary

The paper titled "RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment" proposes a method called Reinforcement Learning from Contrast Distillation (RLCD) to align language models with natural language principles without relying on human feedback. The authors introduce RLCD as a technique that trains a preference model using simulated preference pairs which consist of both high-quality and low-quality examples generated through contrasting positive and negative prompts. This preference model is then utilized to enhance an unaligned base language model through reinforcement learning. In their empirical evaluation, the authors compare RLCD against two baselines: RLAIF (Bai et al., 2022b) and context distillation (Huang et al., 2022). They conduct experiments across three diverse alignment tasks namely harmlessness, helpfulness, and story outline generation. Additionally, they evaluate the performance of RLCD on both 7B and 30B model scales for preference data simulation. The results demonstrate that RLCD outperforms the baseline methods in all tested alignment tasks. This suggests that RLCD effectively aligns language models with natural language principles without requiring human feedback. Overall, the proposed approach shows promise in improving the quality and alignment of language models by leveraging contrast distillation and reinforcement learning techniques.