# FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing

**Authors:** Kyle Stachowicz, Dhruv Shah, Arjun Bhorkar, Ilya Kostrikov, Sergey Levine  
**arXiv:** [2304.09831](https://arxiv.org/abs/2304.09831)  
**Date:** 2023-04-19  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2304.09831v1/

## Key points

- FastRLAP is a reinforcement learning framework for autonomous driving
- It trains autonomously in the real world without human intervention or simulations
- The system uses a large prior dataset for initialization of RL policy and value function representations
- It uses a single low-speed demonstration to determine the desired driving course and extracts checkpoints
- FastRLAP can learn to drive over various racing courses with less than 20 minutes of online training
- The resulting policies demonstrate aggressive driving skills and approach human driver performance
- Demonstration lap is crucial for fast learning and achieving low time-to-first-lap (T2F)
- Pre-training with generic ImageNet encoder leads to quick first lap but poor asymptotic performance
- High-speed navigation requires task-specific features learned through pre-training
- Learning directly from visual observations outperforms variations with access to privileged state information
- Obstacles have similar representations across positions and environments, while state-based agents must learn free space representation through trial and error
- Learned policy makes decisions based on environmental visual features rather than memorizing actions.

## Summary

The FastRLAP system is a reinforcement learning (RL) framework that enables an autonomous small-scale RC car to drive aggressively based on visual observations. Unlike other systems, FastRLAP trains autonomously in the real world without any human interventions or simulations. The system incorporates several important components to achieve this. It initializes the representations for the RL policy and value function from a large prior dataset of other robots navigating in different environments at low speed, which provides relevant navigation information. Using a sample-efficient online RL method, FastRLAP uses a single low-speed user-provided demonstration to determine the desired driving course and extracts navigational checkpoints. It then autonomously practices driving through these checkpoints, resetting automatically on collision or failure. Surprisingly, with appropriate initialization and algorithm choice, FastRLAP can learn to drive over various racing courses with less than 20 minutes of online training. The resulting policies demonstrate emergent aggressive driving skills such as timing braking and acceleration around turns and avoiding obstacles that impede motion. These policies approach the performance of a human driver using a similar first-person interface over the course of training. In simulated environments, additional baselines were considered, showing that the demonstration lap is crucial for fast learning and achieving low time-to-first-lap (T2F). Removing pseudo-resets leads to extended durations of being stuck and slow first laps even with a demonstration. Analyzing the role of pre-training with offline RL, it was found that FastRLAP initialized with a generic ImageNet encoder completes its first lap relatively quickly but has comparably poor asymptotic performance. This suggests that while general-purpose visual features are sufficient for low-speed navigation, high-speed navigation requires task-specific features like depth or obstacle detection learned through task-specific pre-training. Learning directly from visual observations outperformed variations with access to privileged state information in both simulated and real environments. This indicates that features learned by pre-trained encoders are more informative than simple localization estimates because they generalize better. Obstacles have similar representations across different positions and environments, while state-based agents must learn a free space representation of the entire environment through trial and error. Qualitative analysis shows that the learned policy genuinely learns from visual cues in the environment rather than memorizing a sequence of actions. The critic network assigns low values to actions steering towards obstacles and higher values to actions keeping to less restrictive areas like paths instead of tall grass. This suggests a correlation between visual cues in the environment and the decisions made by the learned policy, indicating that it makes decisions based on environmental visual features rather than memorizing actions.