# On the Impact of the Activation Function on Deep Neural Networks Training

**Authors:** Soufiane Hayou, Arnaud Doucet, Judith Rousseau  
**arXiv:** [1902.06853](https://arxiv.org/abs/1902.06853)  
**Date:** 2019-02-19  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1902.06853v2/

## Key points

- Authors focus on optimizing deep neural network architectures for improved learning capabilities and performance
- Consequences of inappropriate selection: information loss during forward propagation, gradient vanishing/exploding during back-propagation
- Conduct comprehensive theoretical analysis to explore tuning initialization parameters and activation functions for accelerated training processes and enhanced performance outcomes
- Aim to identify optimal configurations based on properties of untrained random networks for efficient training and successful network performance
- Study contributes valuable insights into optimizing deep neural networks for improved results in various applications

## Summary

The authors of this study focus on the optimization of deep neural network architectures to improve learning capabilities and overall performance. They highlight the consequences of inappropriate selection in terms of information loss during forward propagation and gradient vanishing/exploding during back-propagation. Building upon previous research on hyperparameters for successful training, they conduct a comprehensive theoretical analysis to explore how tuning initialization parameters and activation functions can accelerate training processes and enhance performance outcomes. By understanding the properties of untrained random networks, they aim to identify optimal configurations that lead to efficient training and successful network performance. This study contributes valuable insights into optimizing deep neural networks for improved results in various applications.