# Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

**Authors:** Quanyi Li, Zhenghao Peng, Bolei Zhou  
**arXiv:** [2202.10341](https://arxiv.org/abs/2202.10341)  
**Date:** 2022-02-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.10341v1/

## Key points

- Researchers Quanyi Li, Zhenghao Peng, and Bolei Zhou introduce the Human-AI Copilot Optimization (HACO) method to enhance reinforcement learning.
- HACO incorporates human expertise into training to improve learning speed and safety.
- The method leverages trial-and-error exploration and partial human demonstrations to train a high-performing agent.
- Experimental results show that HACO achieves higher sample efficiency in safe driving benchmarks compared to traditional methods.
- HACO allows effective exploration in risky environments while maintaining safety through human expert intervention.
- Ablation study validates the effectiveness of HACO in optimizing agent performance with minimal human interventions.

## Summary

In the study "Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization," researchers Quanyi Li, Zhenghao Peng, and Bolei Zhou introduce a novel human-in-the-loop learning method called Human-AI Copilot Optimization (HACO). The aim of this method is to incorporate human expertise into the training loop of reinforcement learning to enhance learning speed and ensure training safety. The key challenge addressed in this work is determining when and how human experts should interact with the learning agent during training, given the limited budget for human intervention. <HACO>The proposed HACO method leverages data from both trial-and-error exploration and partial human demonstrations to train a high-performing agent. By extracting proxy state-action values from partial human demonstrations, HACO optimizes the agent to improve these values while reducing the need for human interventions.</HACO>

Experimental results demonstrate that HACO achieves significantly higher sample efficiency in a safe driving benchmark compared to traditional reinforcement learning and imitation learning baselines. The method successfully trains agents to navigate unseen traffic scenarios with minimal human intervention budget, ensuring high levels of safety and generalizability. Furthermore, comparisons with vanilla RL methods such as PPO and SAC, as well as safe RL baselines like CPO, PPO-Lagrangian, and SAC-Lagrangian, highlight the superior performance of HACO in terms of autonomy and efficiency. <Human Expertise>The proposed HACO method allows for effective exploration by the agent in risky environments while maintaining safety by enabling the human expert to take control and demonstrate how to avoid potentially dangerous situations or trivial behaviors.</Human Expertise>

The study also includes an ablation study that further validates the effectiveness of HACO in optimizing agent performance while minimizing human interventions. Overall, the research presents a promising approach for integrating human expertise into reinforcement learning processes effectively, leading to improved training outcomes in complex tasks such as safe driving. Additional details including code and demo videos are available on the project website: https://decisionforce.github.io/HACO/.