Dynamic ReLU

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   DPReLU: Dynamic Parametric Rectified Linear Unit

   Kien Mai Ngoc, Dong-Hun Yang, Iksoo Shin et al. · 2020 · International Conference on Smart Media and Applications · cité 5 fois · Pas encore résumé

   Résumé scientifique

   As one of the essential elements of deep learning, the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, because of the dying ReLU problem and bias shift effect, deep learning models with ReLU may miss the potential benefits of negative values. To alleviate this problem, numerous ReLU variants have been proposed. In this paper, we propose Dynamic Parametric ReLU (DPReLU) that can control the overall functional shape of ReLU with four learnable parameters and is based on the ideas of the Parametric ReLU (PReLU) and Flexible ReLU (FReLU). The parameters of DPReLU are determined by training rather than by humans, thereby m…

2. Très proche Pas pertinent

   DPReLU: Dynamic Parametric Rectified Linear Unit and Its Proper Weight Initialization Method

   Dong-Hun Yang, Kien Mai Ngoc, Iksoo Shin et al. · 2023 · International Journal of Computational Intelligence Systems · cité 20 fois · Pas encore résumé

   Résumé scientifique

   Activation functions are essential in deep learning, and the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, owing to the dying ReLU problem and bias shift effect, deep learning models using ReLU cannot exploit the potential benefits of negative values. Numerous ReLU variants have been proposed to address this issue. In this study, we propose Dynamic Parametric ReLU (DPReLU), which can dynamically control the overall functional shape of ReLU with four learnable parameters. The parameters of DPReLU are determined by training rather than by humans, thereby making the formulation more suitable and flexible for each model…

3. Très proche Pas pertinent

   Flexible Rectified Linear Units for Improving Convolutional Neural Networks

   Suo Qiu, Bo-Lun Cai · 2017 · arXiv.org · preprint, non relu par les pairs · cité 13 fois · Pas encore résumé

   Résumé scientifique

   Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU). FReLU improves the flexibility of ReLU by a learnable rectified point. FReLU achieves a faster convergence and higher performance. Furthermore, FReLU does not rely on strict assumptions by self-adaption. FReLU is also simple and effective without using exponential function. We evaluate FReLU on two standard image classification dataset, including CIFAR-10 and CIFAR-100. Experimental results show the strengths of the proposed method.

4. Très proche Pas pertinent

   FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

   Suo Qiu, Xiang-Min Xu, Bo-Lun Cai · 2018 · International Conference on Pattern Recognition · cité 96 fois · Pas encore résumé

   Résumé scientifique

   Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. However, because of the zero-hard rectification, ReLU networks lose the benefits from negative values. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU) to further explore the effects of negative values. By redesigning the rectified point of ReLU as a learnable parameter, FReLU expands the states of the activation output. When a network is successfully trained, FReLU tends to converge to a negative value, which improves the expressiveness and thus the performance. Furthermore, FReLU is designed to be simple and effective without exponenti…

5. Très proche Pas pertinent

   Adaptive Convolutional ReLUs

   Hongyang Gao, Lei Cai, Shuiwang Ji · 2020 · AAAI Conference on Artificial Intelligence · cité 17 fois · Pas encore résumé

   Résumé scientifique

   Rectified linear units (ReLUs) are currently the most popular activation function used in neural networks. Although ReLUs can solve the gradient vanishing problem and accelerate training convergence, it suffers from the dying ReLU problem in which some neurons are never activated if the weights are not updated properly. In this work, we propose a novel activation function, known as the adaptive convolutional ReLU (ConvReLU), that can better mimic brain neuron activation behaviors and overcome the dying ReLU problem. With our novel parameter sharing scheme, ConvReLUs can be applied to convolution layers that allow each input neuron to be activated by different trainable thresholds without inv…

6. Très proche* Pas pertinent

   Deep Learning using Rectified Linear Units (ReLU)

   Abien Fred Agarap · 2018 · arXiv.org · preprint, non relu par les pairs · cité 4 021 fois · Pas encore résumé
7. Très proche Pas pertinent

   Deep neural networks with Elastic Rectified Linear Units for object recognition

   Xiao-Heng Jiang, Yan-Wei Pang, Xue-Long Li et al. · 2018 · Neurocomputing · cité 129 fois · Pas encore résumé

   Résumé scientifique

   Abstract Rectified Linear Unit (ReLU) is crucial to the recent success of deep neural networks (DNNs). In this paper, we propose a novel Elastic Rectified Linear Unit (EReLU) that focuses on processing the positive part of input. Unlike previous variants of ReLU that typically adopt linear or piecewise linear functions to represent the positive part, EReLU is characterized by that each positive value scales within a moderate range like a spring during training stage. On test time, EReLU becomes standard ReLU. EReLU improves model fitting with no extra parameters and little overfitting risk. Furthermore, we propose Elastic Parametric Rectified Linear Unit (EPReLU) by taking advantage of EReLU…

8. Très proche Pas pertinent

   A ReLU Dense Layer to Improve the Performance of Neural Networks

   Alireza M. Javid, Sandipan Das, M. Skoglund et al. · 2020 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cité 45 fois · Pas encore résumé

   Résumé scientifique

   We propose ReDense as a simple and low complexity way to improve the performance of trained neural networks. We use a combination of random weights and rectified linear unit (ReLU) activation function to add a ReLU dense (ReDense) layer to the trained neural network such that it can achieve a lower training loss. The lossless flow property (LFP) of ReLU is the key to achieve the lower training loss while keeping the generalization error small. ReDense does not suffer from vanishing gradient problem in the training due to having a shallow structure. We experimentally show that ReDense can improve the training and testing performance of various neural network architectures with different optim…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Pas pertinent

   Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

   Djork-Arné Clevert, T. Unterthiner, S. Hochreiter · 2015 · International Conference on Learning Representations · cité 6 493 fois · Pas encore résumé

   Résumé scientifique

   We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit na…

2. Proche Pas pertinent

   Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

   Ludovic Trottier, Philippe Gigu�re, B. Chaib-draa · 2016 · International Conference on Machine Learning and Applications · cité 226 fois · Pas encore résumé

   Résumé scientifique

   Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a …

3. Proche Pas pertinent

   Swish: a Self-Gated Activation Function

   Prajit Ramachandran, Barret Zoph, Quoc V. Le · 2017 · arXiv: Neural and Evolutionary Computing · preprint, non relu par les pairs · cité 728 fois · Pas encore résumé

   Résumé scientifique

   The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose a new activation function, named Swish, which is simply $f(x) = x \cdot \text{sigmoid}(x)$. Our experiments show that Swish tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for M…

4. Apparenté Une méthode qu’il utilise Pas pertinent

   BlockDrop: Dynamic Inference Paths in Residual Networks

   Zu-Xuan Wu, Tushar Nagarajan, Abhishek Kumar et al. · 2017 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · cité 563 fois · Pas encore résumé

   Résumé scientifique

   Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual reward of utilizing a minimal nu…

5. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Dynamic Convolution: Attention Over Convolution Kernels

   Yinpeng Chen, Xi-Yang Dai, Meng-Chen Liu et al. · 2019 · Computer Vision and Pattern Recognition · cité 1 542 fois · Pas encore résumé

   Résumé scientifique

   Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, …

Les articles qui le citent

1. Proche Utilise cet article en profondeur Pas pertinent

   PWLU: Learning Specialized Activation Functions With the Piecewise Linear Unit

   Zezhou Zhu, Yu-Cong Zhou, Yuan Dong et al. · 2023 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cité 15 fois · Pas encore résumé

   Résumé scientifique

   The choice of activation functions is crucial to deep neural networks. ReLU is a popular hand-designed activation function. Swish, the automatically searched activation function, outperforms ReLU on many challenging datasets. However, the search method has two main drawbacks. First, the tree-based search space is highly discrete and restricted, which is difficult to search. Second, the sample-based search method is inefficient in finding specialized activation functions for each dataset or neural architecture. To overcome these drawbacks, we propose a new activation function called Piecewise Linear Unit (PWLU), incorporating a carefully designed formulation and learning method. PWLU can lear…

2. Proche Utilise cet article en profondeur Pas pertinent

   RepAct: The Re-parameterizable Adaptive Activation Function

   Xian Wu, Qingchuan Tao, Shuang Wang · 2024 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé

   Résumé scientifique

   Addressing the imperative need for efficient artificial intelligence in IoT and edge computing, this study presents RepAct, a re-parameterizable adaptive activation function tailored for optimizing lightweight neural networks within the computational limitations of edge devices. By employing a multi-branch structure with learnable adaptive weights, RepAct enriches feature processing and enhances cross-layer interpretability. When evaluated on tasks such as image classification and object detection, RepAct notably surpassed conventional activation functions in lightweight networks, delivering up to a 7.92% accuracy boost on MobileNetV3-Small for the ImageNet100 dataset, while maintaining comp…

3. Proche Pas pertinent

   VeLU: Variance-enhanced Learning Unit for Deep Neural Networks

   Ashkan Shakarami, Yousef Yeganeh, Azade Farshad et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Activation functions play a critical role in deep neural networks by shaping gradient flow, optimization stability, and generalization. While ReLU remains widely used due to its simplicity, it suffers from gradient sparsity and dead-neuron issues and offers no adaptivity to input statistics. Smooth alternatives such as Swish and GELU improve gradient propagation but still apply a fixed transformation regardless of the activation distribution. In this paper, we propose VeLU, a Variance-enhanced Learning Unit that introduces variance-aware and distributionally aligned nonlinearity through a principled combination of ArcTan-ArcSin transformations, adaptive scaling, and Wasserstein-2 regularizat…

4. Apparenté Utilise cet article en profondeur Pas pertinent

   MicroNet: Improving Image Recognition with Extremely Low FLOPs

   Yun-Sheng Li, Yinpeng Chen, Xi-Yang Dai et al. · 2021 · IEEE International Conference on Computer Vision · cité 114 fois · Pas encore résumé

   Résumé scientifique

   This paper aims at addressing the problem of substantial performance degradation at extremely low computational cost (e.g. 5M FLOPs on ImageNet classification). We found that two factors, sparse connectivity and dynamic activation function, are effective to improve the accuracy. The former avoids the significant reduction of network width, while the latter mitigates the detriment of reduction in network depth. Technically, we propose micro-factorized convolution, which factorizes a convolution matrix into low rank matrices, to integrate sparse connectivity into convolution. We also present a new dynamic activation function, named Dynamic Shift Max, to improve the non-linearity via maxing out…

5. Proche Pas pertinent

   Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

   Zhiqiang Zhang, Xiaoming Li, Xinyi Xu et al. · 2024 · International Journal of Web Information Systems · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Purpose The purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets. Design/methodology/approach This research integrates three novel trai…