Exploring the Performance of Perforated Backpropagation through Further Experiments

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Proche Prolonge cet article Cite cet article Utilise cet article en profondeur Pas pertinent

   Perforated Neural Networks for Keyword Spotting

   V. Gopal, Aris Ilias Goutis, Ralph Crewe et al. · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé

   Perforated Neural Networks applique Perforated Backpropagation à la détection de mots-clés sur appareil périphérique, tandis que cet article teste l’algorithme sur plusieurs projets.

   Résumé scientifique

   Edge machine learning presents a unique set of constraints not encountered in cloud-scale model deployment: strict memory budgets, limited compute, and non-negotiable accuracy thresholds must all be satisfied simultaneously. Existing compression and optimization techniques can trade one resource for another, but rarely improve both accuracy and model size at the same time. This paper presents the application of Perforated Backpropagation to keyword spotting on the Edge Impulse platform, an experiment that won the Best Model award at the Edge Impulse 2025 Hackathon in December 2025. By adding artificial Dendrite Nodes to a standard convolutional neural network trained on the Edge Impulse keyw…

2. Apparenté Contexte plus large Pas pertinent

   The Cascaded Forward Algorithm for Neural Network Training

   Gongpei Zhao, Tao Wang, Yidong Li et al. · 2023 · Computer Vision and Pattern Recognition · preprint, non relu par les pairs · cité 28 fois · Résumé IA disponible

   The Cascaded Forward Algorithm propose une autre méthode d’entraînement neuronal, sans lien indiqué avec les expériences de compression par Perforated Backpropagation.

   Résumé scientifique

   Backpropagation algorithm has been widely used as a mainstream learning procedure for neural networks in the past decade, and has played a significant role in the development of deep learning. However, there exist some limitations associated with this algorithm, such as getting stuck in local minima and experiencing vanishing/exploding gradients, which have led to questions about its biological plausibility. To address these limitations, alternative algorithms to backpropagation have been preliminarily explored, with the Forward-Forward (FF) algorithm being one of the most well-known. In this paper we propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algori…

3. Apparenté Contexte plus large Pas pertinent

   Scaling MLPs: A Tale of Inductive Bias

   Gregor Bachmann, Sotiris Anagnostidis, Thomas Hofmann · 2023 · Machine Learning · cité 73 fois · Résumé IA disponible

   Scaling MLPs étudie les limites des perceptrons multicouches en vision, alors que cet article expérimente une méthode d’entraînement sur divers projets.

   Résumé scientifique

   In this work we revisit the most fundamental building block in deep learning, the multi-layer perceptron (MLP), and study the limits of its performance on vision tasks. Empirical insights into MLPs are important for multiple reasons. (1) Given the recent narrative "less inductive bias is better", popularized due to transformers eclipsing convolutional models, it is natural to explore the limits of this hypothesis. To that end, MLPs offer an ideal test bed, being completely free of any inductive bias. (2) MLPs have almost exclusively been the main protagonist in the deep learning theory literature due to their mathematical simplicity, serving as a proxy to explain empirical phenomena observed…

4. Apparenté Contexte plus large Pas pertinent

   DeepPCR: Parallelizing Sequential Operations in Neural Networks

   Federico Danieli, Miguel Sarabia, Xavier Suau et al. · 2023 · Machine Learning · cité 14 fois · Résumé IA disponible

   DeepPCR parallélise des opérations séquentielles des réseaux neuronaux, tandis que cet article évalue une méthode d’apprentissage inspirée des dendrites.

   Résumé scientifique

   Parallelization techniques have become ubiquitous for accelerating inference and training of deep neural networks. Despite this, several operations are still performed in a sequential manner. For instance, the forward and backward passes are executed layer-by-layer, and the output of diffusion models is produced by applying a sequence of denoising steps. This sequential approach results in a computational cost proportional to the number of steps involved, presenting a potential bottleneck as the number of steps increases. In this work, we introduce DeepPCR, a novel algorithm which parallelizes typically sequential operations used in inference and training of neural networks. DeepPCR is based…

5. Apparenté* Article de revue Pas pertinent

   Navigating beyond backpropagation: on alternative training methods for deep neural networks

   Roshan Birjais, Kevin I-Kai Wang, Waleed H. Abdulla · 2025 · Knowledge and Information Systems · cité 4 fois · Pas encore résumé

   Navigating beyond backpropagation examine des méthodes d’entraînement alternatives aux réseaux profonds, sujet général des expériences sur Perforated Backpropagation.

6. Apparenté Contexte plus large Pas pertinent

   Git Re-Basin: Merging Models modulo Permutation Symmetries

   Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa · 2022 · Machine Learning · cité 598 fois · Résumé IA disponible

   Git Re-Basin aligne des unités de modèles pour fusionner leurs paramètres, tandis que cet article étudie compression et précision par une méthode d’apprentissage.

   Résumé scientifique

   The success of deep learning is thanks to our ability to solve certain massive non-convex optimization problems with relative ease. Despite non-convex optimization being NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes contain (nearly) a single basin, after accounting for all possible permutation symmetries of hidden units. We introduce three algorithms to permute the units of one model to bring them into alignment with units of a reference model. This transformation produces a functionally equivalent set of weights that lie in an approxima…

7. Apparenté Même méthode, autre problème Pas pertinent

   Backpropagation Neural Tree

   V. Ojha, Giuseppe Nicosia · 2022 · Neural Networks · cité 21 fois · Pas encore résumé

   Backpropagation Neural Tree intègre des non-linéarités dendritiques dans un modèle neuronal, comme Perforated Backpropagation s’inspire du rôle computationnel des dendrites.

   Résumé scientifique

   We propose a novel algorithm called Backpropagation Neural Tree (BNeuralT), which is a stochastic computational dendritic tree. BNeuralT takes random repeated inputs through its leaves and imposes dendritic nonlinearities through its internal connections like a biological dendritic tree would do. Considering the dendritic-tree like plausible biological properties, BNeuralT is a single neuron neural tree model with its internal sub-trees resembling dendritic nonlinearities. BNeuralT algorithm produces an ad hoc neural tree which is trained using a stochastic gradient descent optimizer like gradient descent (GD), momentum GD, Nesterov accelerated GD, Adagrad, RMSprop, or Adam. BNeuralT trainin…

8. Apparenté Autre approche du problème Pas pertinent

   Hebbian Deep Learning Without Feedback

   Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo et al. · 2022 · Neural and Evolutionary Computing · cité 97 fois · Résumé IA disponible

   Hebbian Deep Learning Without Feedback entraîne des réseaux profonds sans signaux d’erreur ni rétroaction, contrairement à la méthode de rétropropagation étudiée ici.

   Résumé scientifique

   Recent approximations to backpropagation (BP) have mitigated many of BP's computational inefficiencies and incompatibilities with biology, but important limitations still remain. Moreover, the approximations significantly decrease accuracy in benchmarks, suggesting that an entirely different approach may be more fruitful. Here, grounded on recent theory for Hebbian learning in soft winner-take-all networks, we present multilayer SoftHebb, i.e. an algorithm that trains deep neural networks, without any feedback, target, or error signals. As a result, it achieves efficiency by avoiding weight transport, non-local plasticity, time-locking of layer updates, iterative equilibria, and (self-) supe…

Les travaux antérieurs sur lesquels il s’appuie

1. Apparenté Contexte plus large Pas pertinent

   RoBERTa: A Robustly Optimized BERT Pretraining Approach

   Yinhan Liu, Myle Ott, Naman Goyal et al. · 2019 · Computation and Language · preprint, non relu par les pairs · cité 32 368 fois · Pas encore résumé

   RoBERTa mesure les effets des choix d’entraînement et des données sur le préentraînement linguistique, tandis que cet article teste la compression de modèles.

   Résumé scientifique

   Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight …

2. Apparenté Contexte plus large Pas pertinent

   Deep Residual Learning for Image Recognition

   Kaiming He, X. Zhang, Shaoqing Ren et al. · 2015 · Computer Vision and Pattern Recognition · cité 245 842 fois · Résumé IA disponible

   Deep Residual Learning introduit des réseaux résiduels pour faciliter l’optimisation et accroître la profondeur, contrairement à l’étude de compression par rétropropagation perforée.

   Résumé scientifique

   Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers - 8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error o…

3. Apparenté Contexte plus large Pas pertinent

   Searching for MobileNetV3

   Andrew G. Howard, Ruo-Ming Pang, Hartwig Adam et al. · 2019 · IEEE International Conference on Computer Vision · cité 10 730 fois · Pas encore résumé

   Searching for MobileNetV3 optimise des architectures pour les processeurs mobiles, tandis que cet article teste une méthode pouvant réduire la taille des modèles.

   Résumé scientifique

   We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resourc…

4. Apparenté Contexte plus large Pas pertinent

   Generating Wikipedia by Summarizing Long Sequences

   Peter J. Liu, Mohammad Saleh, Etienne Pot et al. · 2018 · International Conference on Learning Representations · cité 887 fois · Résumé IA disponible

   Generating Wikipedia by Summarizing Long Sequences traite du résumé neuronal de documents, un objet différent des expériences d’optimisation de réseaux.

   Résumé scientifique

   We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores an…

5. Apparenté* Données, outil ou méthode en commun Pas pertinent

   ImageNet: A large-scale hierarchical image database

   Jia Deng, Wei Dong, R. Socher et al. · 2009 · 2009 IEEE Conference on Computer Vision and Pattern Recognition · cité 78 335 fois · Pas encore résumé

   ImageNet fournit une base de données hiérarchique d’images, ressource citée dans l’article consacré aux expériences sur Perforated Backpropagation.