Hypernetworks for Continual Semi-Supervised Learning

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Proche Pas pertinent

   Nonparametric Bayesian Structure Adaptation for Continual Learning

   Abhishek Kumar, Sunabha Chatterjee, Piyush Rai · 2019 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Continual Learning is a learning paradigm where machine learning models are trained with sequential or streaming tasks. Two notable directions among the recent advances in continual learning with neural networks are (i) variational Bayes based regularization by learning priors from previous tasks, and, (ii) learning the structure of deep networks to adapt to new tasks. So far, these two approaches have been orthogonal. We present a principled non-parametric Bayesian approach for learning the structure of feed-forward neural networks, addressing the shortcomings of both these approaches. In our model, the number of nodes in each hidden layer can automatically grow with the introduction of eac…

2. Proche Pas pertinent

   A Comprehensive Survey of Continual Learning: Theory, Method and Application

   Liyuan Wang, Xingxing Zhang, Hang Su et al. · 2023 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   To cope with real-world dynamics, an intelligent agent needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime. This ability, known as continual learning, provides a foundation for AI systems to develop themselves adaptively. In a general sense, continual learning is explicitly limited by catastrophic forgetting, where learning a new task usually results in a dramatic performance drop of the old tasks. Beyond this, increasingly numerous advances have emerged in recent years that largely extend the understanding and application of continual learning. The growing and widespread interest in this direction demonstrates its realistic significance as well …

3. Proche Pas pertinent

   Continual Lifelong Learning with Neural Networks: A Review

   German I. Parisi, Ronald Kemker, Jose L. Part et al. · 2018 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Humans and animals have the ability to continually acquire, fine-tune, and transfer knowledge and skills throughout their lifespan. This ability, referred to as lifelong learning, is mediated by a rich set of neurocognitive mechanisms that together contribute to the development and specialization of our sensorimotor skills as well as to long-term memory consolidation and retrieval. Consequently, lifelong learning capabilities are crucial for autonomous agents interacting in the real world and processing continuous streams of information. However, lifelong learning remains a long-standing challenge for machine learning and neural network models since the continual acquisition of incrementally…

4. Proche Pas pertinent

   Nested Learning: The Illusion of Deep Learning Architectures

   Ali Behrouz, Meisam Razaviyayn, Peilin Zhong et al. · 2025 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Despite the recent progresses, particularly in developing Language Models, there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improve, and find effective solutions. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a machine learning model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own context flow. Through the lenses of NL, existing deep learning methods learns from data through compressing their own context flow, and in-context learning naturally emerges in large models. NL suggests a philosophy to design more expre…

5. Proche Pas pertinent

   Online Continual Learning with Maximally Interfered Retrieval

   Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky et al. · 2019 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work we consider a control…

6. Apparenté Pas pertinent

   Large Language Model Can Continue Evolving From Mistakes

   Haokun Zhao, Haixia Han, Jie Shi et al. · 2024 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Large Language Models (LLMs) demonstrate impressive performance in various downstream tasks. However, they may still generate incorrect responses in certain scenarios due to the knowledge deficiencies and the flawed pre-training data. Continual Learning (CL) is a commonly used method to address this issue. Traditional CL is task-oriented, using novel or factually accurate data to retrain LLMs from scratch. However, this method requires more task-related training data and incurs expensive training costs. To address this challenge, we propose the Continue Evolving from Mistakes (CEM) method, inspired by the 'summarize mistakes' learning skill, to achieve iterative refinement of LLMs. Specifica…

7. Apparenté Pas pertinent

   Learning to Learn Neural Networks

   Tom Bosc · 2016 · Apprentissage automatique · Pas encore résumé

   Résumé scientifique

   Meta-learning consists in learning learning algorithms. We use a Long Short Term Memory (LSTM) based network to learn to compute on-line updates of the parameters of another neural network. These parameters are stored in the cell state of the LSTM. Our framework allows to compare learned algorithms to hand-made algorithms within the traditional train and test methodology. In an experiment, we learn a learning algorithm for a one-hidden layer Multi-Layer Perceptron (MLP) on non-linearly separable datasets. The learned algorithm is able to update parameters of both layers and generalise well on similar datasets.

8. Apparenté Pas pertinent

   Continual Pre-Training Mitigates Forgetting in Language and Vision

   Andrea Cossu, Tinne Tuytelaars, Antonio Carta et al. · 2022 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Pre-trained models are nowadays a fundamental component of machine learning research. In continual learning, they are commonly used to initialize the model before training on the stream of non-stationary data. However, pre-training is rarely applied during continual learning. We formalize and investigate the characteristics of the continual pre-training scenario in both language and vision environments, where a model is continually pre-trained on a stream of incoming data and only later fine-tuned to different downstream tasks. We show that continually pre-trained models are robust against catastrophic forgetting and we provide strong empirical evidence supporting the fact that self-supervis…