# Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

**Authors:** Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan  
**arXiv:** [2305.03047](https://arxiv.org/abs/2305.03047)  
**Date:** 2023-05-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.03047v1/

## Key points

- Recent AI-assistant agents rely heavily on supervised fine-tuning and reinforcement learning from human feedback to align with human intentions.
- This dependence can significantly constrain the true potential of AI-assistant agents due to high cost, quality, reliability, diversity, self-consistency and undesirable biases.
- A novel approach called SELF-ALIGN has been proposed to address these challenges.
- SELF-ALIGN combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.
- The approach encompasses four stages: synthetic prompt generation, in-context learning from demonstrations using a small set of principles, fine-tuning on original LLM with high-quality self-aligned responses, and refinement step addressing issues of overly brief or indirect responses.
- Applying SELF ALIGN to the LLaMA 65b base language model has resulted in an AI assistant named Dromedary that significantly surpasses several state-of-the-art AI systems like Text Davinci 003 and Alpaca on benchmark datasets with various settings.
- Verbose cloning has also been tested where verbose versions of successful prompts are used as additional training data for LLMs Anthropic LM LLaMA 65B Alpaca.
- In conclusion, SELF ALIGN offers a promising approach for self-alignment of AI agents with minimal human supervision.

## Summary

Recent AI-assistant agents, such as ChatGPT, rely heavily on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to ensure that the output of large language models (LLMs) aligns with human intentions. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and related issues on quality, reliability, diversity, self-consistency and undesirable biases. To address these challenges, a novel approach called SELF-ALIGN has been proposed. SELF-ALIGN combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. The approach encompasses four stages: firstly, an LLM generates synthetic prompts using a topic-guided method to augment prompt diversity; secondly, a small set of human-written principles guides the LLM through in-context learning from demonstrations to produce helpful, ethical and reliable responses to user queries; thirdly, fine-tuning is done on the original LLM with high quality self-aligned responses so that it can generate desirable responses directly without principles or demonstrations anymore; finally a refinement step addresses issues of overly brief or indirect responses. Applying SELF ALIGN to the LLaMA 65b base language model has resulted in an AI assistant named Dromedary. With fewer than 300 lines of human annotations including less than 200 seed prompts 16 generic principles and 5 exemplars for in context learning Dromedary significantly surpasses several state of the art AI systems like Text Davinci 003 and Alpaca on benchmark datasets with various settings. The principle engraved synthetic prompts generated by SELF ALIGN have shown promising results in preliminary testing. Verbose cloning has also been tested where verbose versions of successful prompts are used as additional training data for LLMs Anthropic LM LLaMA 65B Alpaca has been used to demonstrate the effectiveness of SELF ALIGN. In conclusion SELF ALIGN offers a promising approach for self alignment of AI agents with minimal human supervision. The approach combines principle driven reasoning and the generative power of LLMs to produce helpful ethical and reliable responses to user queries. Future work could explore the use of verbose cloning and other techniques to further improve performance of AI assistant agents.