# Eureka: Human-Level Reward Design via Coding Large Language Models

**Authors:** Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar  
**arXiv:** [2310.12931](https://arxiv.org/abs/2310.12931)  
**Date:** 2023-10-19  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.12931v1/

## Key points

- Large Language Models (LLMs) struggle with complex low-level manipulation tasks
- Eureka is a human-level reward design algorithm powered by LLMs
- Eureka performs evolutionary optimization over reward code using state-of-the-art LLMs like GPT-4
- Eureka generates rewards for acquiring complex skills through reinforcement learning without task-specific prompting or predefined templates
- Eureka surpasses human experts on 83% of tasks and achieves an average normalized improvement of 52%
- Eureka enables gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF)
- Eureka incorporates human inputs to enhance the quality and safety of generated rewards without model updating
- Eureka rewards applied in a curriculum learning setting enable a simulated Shadow Hand to perform pen spinning tricks with rapid speed and precision
- Eureka's human-level reward design algorithm leverages LLMs for complex low-level manipulation tasks
- The results demonstrate the superiority of Eureka over expert human-engineered rewards and its potential for improving reinforcement learning with human feedback.

## Summary

In their paper titled "Eureka: Human-Level Reward Design via Coding Large Language Models," Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan and Anima Anandkumar discuss the challenges of using Large Language Models (LLMs) for complex low-level manipulation tasks. While LLMs have been successful in high-level semantic planning for sequential decision-making tasks, they struggle with tasks such as dexterous pen spinning. To address this gap, the authors introduce Eureka - a human-level reward design algorithm powered by LLMs. Eureka leverages the capabilities of state-of-the-art LLMs like GPT-4 to perform evolutionary optimization over reward code. It exploits zero-shot generation, code writing and in-context improvement abilities to generate rewards that can be used for acquiring complex skills through reinforcement learning without any task specific prompting or pre defined templates. To evaluate Eureka's performance, the authors conducted experiments in 29 open source RL environments with 10 different robot morphologies. They found that Eureka surpasses human experts on 83% of the tasks and achieves an average normalized improvement of 52%. This highlights the generality and effectiveness of Eureka in various scenarios. Additionally, Eureka enables a gradient free in context learning approach to reinforcement learning from human feedback (RLHF). It incorporates human inputs to enhance the quality and safety of generated rewards without requiring model updating. Finally, the authors demonstrate the practical application of Eureka rewards in a curriculum learning setting by showcasing a simulated Shadow Hand capable of performing pen spinning tricks with rapid speed and precision. Overall this research presents an innovative solution to leveraging LLMs for complex low level manipulation tasks through Eureka's human level reward design algorithm. The results demonstrate its superiority over expert human engineered rewards and its potential for improving reinforcement learning with human feedback.