# Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

**Authors:** Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio  
**arXiv:** [1406.2572](https://arxiv.org/abs/1406.2572)  
**Date:** 2014-06-10  
**Summary page:** https://www.summarizepaper.com/en/paper/1406.2572v1/

## Key points

- Authors: Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio
- Focus: Challenges in minimizing non-convex error functions over continuous, high-dimensional spaces
- Main Point: Saddle points are a more significant challenge than local minima in practical high-dimensional problems
- Issue: Saddle points surrounded by high error plateaus impede learning progress and slow down optimization algorithms like gradient descent and quasi-Newton methods
- Solution Proposed: Saddle-free Newton method for second-order optimization to navigate high-dimensional saddle points efficiently
- Validation: Applied the proposed method to training deep or recurrent neural networks with superior performance compared to conventional methods
- Impact: Offers a promising avenue for enhancing optimization algorithms in various scientific and engineering domains

## Summary

In their paper "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization," authors Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio delve into the challenges faced in minimizing non-convex error functions over continuous, high-dimensional spaces. The authors argue that a more significant difficulty arises from the proliferation of saddle points rather than local minima in practical high-dimensional problems. These saddle points are surrounded by high error plateaus that can significantly impede learning progress. This creates an illusion of a local minimum and can drastically slow down optimization algorithms like gradient descent and quasi-Newton methods. To address this challenge, the authors propose a novel approach called the saddle-free Newton method for second-order optimization. This method aims to swiftly navigate high-dimensional saddle points that hinder traditional optimization techniques. They validate their proposal by applying it to training deep or recurrent neural networks and provide numerical evidence showcasing its superior performance compared to conventional methods. By tackling the issue of saddle points in high-dimensional non-convex optimization landscapes, this research offers a promising avenue for enhancing optimization algorithms across various scientific and engineering domains.