Meta-Learning Symmetries by Reparameterization

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Another approach to the problem Not relevant

   Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance

   Jinwoo Kim, T. Nguyen, Ayhan Suleymanzade et al. · 2023 · Neural Information Processing Systems · cited 35 times · Not summarized yet

   Learning Probabilistic Symmetrization makes an arbitrary base model equivariant by learning a probabilistic symmetrization, rather than learning parameter-sharing patterns from data.

   Abstract

   We present a novel framework to overcome the limitations of equivariant architectures in learning functions with group symmetries. In contrary to equivariant architectures, we use an arbitrary base model (such as an MLP or a transformer) and symmetrize it to be equivariant to the given group by employing a small equivariant network that parameterizes the probabilistic distribution underlying the symmetrization. The distribution is end-to-end trained with the base model which can maximize performance while reducing sample complexity of symmetrization. We show that this approach ensures not only equivariance to given group but also universal approximation capability in expectation. We implemen…

2. Very close Builds on this paper Cites this paper Not relevant

   Equivariance Discovery by Learned Parameter-Sharing

   Raymond A. Yeh, Yuan-Ting Hu, Mark A. Hasegawa-Johnson et al. · 2022 · International Conference on Artificial Intelligence and Statistics · cited 23 times · Not summarized yet

   Equivariance Discovery by Learned Parameter-Sharing formulates symmetry discovery as optimization over parameter-sharing schemes and cites this paper.

   Abstract

   Designing equivariance as an inductive bias into deep-nets has been a prominent approach to build effective models, e.g., a convolutional neural network incorporates translation equivariance. However, incorporating these inductive biases requires knowledge about the equivariance properties of the data, which may not be available, e.g., when encountering a new domain. To address this, we study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. We propose to use the partition distance to empirically quantify the accuracy of the recovered equivariance. Also, we theoreticall…

3. Very close Another approach to the problem Not relevant

   Symmetry From Scratch: Group Equivariance as a Supervised Learning Task

   Haozhe Huang, L. K. Cheng, Kai Chen et al. · 2024 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Symmetry From Scratch induces equivariance by teaching general architectures symmetries as a supervised task, while this paper learns parameter-sharing patterns from data.

   Abstract

   In machine learning datasets with symmetries, the paradigm for backward compatibility with symmetry-breaking has been to relax equivariant architectural constraints, engineering extra weights to differentiate symmetries of interest. However, this process becomes increasingly over-engineered as models are geared towards specific symmetries/asymmetries hardwired of a particular set of equivariant basis functions. In this work, we introduce symmetry-cloning, a method for inducing equivariance in machine learning models. We show that general machine learning architectures (i.e., MLPs) can learn symmetries directly as a supervised learning task from group equivariant architectures and retain/brea…

4. Very close Groundwork for this paper Cited by this paper Not relevant

   Equivariance Through Parameter-Sharing

   Siamak Ravanbakhsh, J. Schneider, B. Póczos · 2017 · International Conference on Machine Learning · cited 436 times · Not summarized yet

   Equivariance Through Parameter-Sharing establishes how parameter symmetries and sharing schemes induce group equivariance, which this paper cites.

   Abstract

   We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group G that acts discretely on the input and output of a standard neural network layer ϕW : ℝM → ℝN, we show that ϕW is equivariant with respect to G-action iff G explains the symmetries of the network parameters W. Inspired by this observation, we then propose two parameter-sharing schemes to induce the desirable symmetry on W. Our procedure for tying the parameters achieves G-equivariance and, under some conditions on the action of G, it guarantees sensitivity to all other permutation groups outside G.

5. Very close Builds on this paper Cites this paper Not relevant

   Equivariance-aware Architectural Optimization of Neural Networks

   Kaitlin Maile, Dennis G. Wilson, P. Forré · 2022 · International Conference on Learning Representations · cited 10 times · Not summarized yet

   Equivariance-aware Architectural Optimization adapts equivariance constraints across groups and cites this paper's work on learning symmetries.

   Abstract

   Incorporating equivariance to symmetry groups as a constraint during neural network training can improve performance and generalization for tasks exhibiting those symmetries, but such symmetries are often not perfectly nor explicitly present. This motivates algorithmically optimizing the architectural constraints imposed by equivariance. We propose the equivariance relaxation morphism, which preserves functionality while reparameterizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the [G]-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization. We further present evolutionary…

6. Very close Builds on this paper Cites this paper Not relevant

   Learning Layer-wise Equivariances Automatically using Gradients

   Alexander Immer, T. V. D. van der Ouderaa, Mark van der Wilk · 2023 · Neural Information Processing Systems · cited 31 times · Not summarized yet

   Learning Layer-wise Equivariances Automatically also learns symmetry constraints and weight connectivity from data using gradients, and cites this paper.

   Abstract

   Convolutions encode equivariance symmetries into neural networks leading to better generalisation performance. However, symmetries provide fixed hard constraints on the functions a network can represent, need to be specified in advance, and can not be adapted. Our goal is to allow flexible symmetry constraints that can automatically be learned from data using gradients. Learning symmetry and associated weight connectivity structures from scratch is difficult for two reasons. First, it requires efficient and flexible parameterisations of layer-wise equivariances. Secondly, symmetries act as constraints and are therefore not encouraged by training losses measuring data fit. To overcome these c…

7. Very close Builds on this paper Cites this paper Not relevant

   Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensors

   E. Bekkers, Alejandro García-Castellanos, Thijs P. Kuipers et al. · 2024 · Neural Information Processing Systems · cited 6 times · Not summarized yet

   Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensors discovers symmetries through weight sharing and cites this paper.

   Abstract

   Group equivariance has emerged as a valuable inductive bias in deep learning, enhancing generalization, data efficiency, and robustness. Classically, group equivariant methods require the groups of interest to be known beforehand, which may not be realistic for real-world data. Additionally, baking in fixed group equivariance may impose overly restrictive constraints on model architecture. This highlights the need for methods that can dynamically discover and apply symmetries as soft constraints. For neural network architectures, equivariance is commonly achieved through group transformations of a canonical weight tensor, resulting in weight sharing over a given group $G$. In this work, we p…

8. Very close Broader context Not relevant

   A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models

   Yuqing Xie, T. Smidt · 2025 · Neural Information Processing Systems · cited 10 times · Not summarized yet

   A Tale of Two Symmetries analyzes optimization and loss landscapes in equivariant models, rather than discovering equivariances from data.

   Abstract

   Equivariant neural networks have proven to be effective for tasks with known underlying symmetries. However, optimizing equivariant networks can be tricky and best training practices are less established than for standard networks. In particular, recent works have found small training benefits from relaxing equivariance constraints. This raises the question: do equivariance constraints introduce fundamental obstacles to optimization? Or do they simply require different hyperparameter tuning? In this work, we investigate this question through a theoretical analysis of the loss landscape geometry. We focus on networks built using permutation representations, which we can view as a subset of un…

Prior work it builds on

1. Related Groundwork for this paper Used in depth by this paper A method it uses Not relevant

   Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

   Chelsea Finn, Pieter Abbeel, Sergey Levine · 2017 · Machine Learning · cited 15,598 times · AI summary available

   Model-Agnostic Meta-Learning introduces a general gradient-based meta-learning approach that this paper cites.

   Abstract

   We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model…

2. Close Same method, another problem Not relevant

   Harmonic Networks: Deep Translation and Rotation Equivariance

   Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov et al. · 2016 · Computer Vision and Pattern Recognition · cited 897 times · Not summarized yet

   Harmonic Networks encode translation and rotation equivariance through structured filters, applying equivariant parameterization to image processing.

   Abstract

   Translating or rotating an input image should not affect the results of many computer vision tasks. Convolutional neural networks (CNNs) are already translation equivariant: input image translations produce proportionate feature map translations. This is not the case for rotations. Global rotation equivariance is typically sought through data augmentation, but patch-wise equivariance is more difficult. We present Harmonic Networks or H-Nets, a CNN exhibiting equivariance to patch-wise translation and 360-rotation. We achieve this by replacing regular CNN filters with circular harmonics, returning a maximal response and orientation for every receptive field patch. H-Nets use a rich, parameter…

3. Close Same method, another problem A method it uses Not relevant

   Modular meta-learning

   Ferran Alet, Tomas Lozano-Perez, L. Kaelbling · 2018 · Conference on Robot Learning · cited 138 times · Not summarized yet

   Modular meta-learning learns reusable neural components across tasks, applying meta-learning to modular composition rather than symmetry discovery.

   Abstract

   Many prediction problems, such as those that arise in the context of robotics, have a simplifying underlying structure that, if known, could accelerate learning. In this paper, we present a strategy for learning a set of neural network modules that can be combined in different ways. We train different modular structures on a set of related tasks and generalize to new tasks by composing the learned modules in new ways. By reusing modules to generalize we achieve combinatorial generalization, akin to the "infinite use of finite means" displayed in language. Finally, we show this improves performance in two robotics-related problems.

4. Close Same method, another problem Not relevant

   Meta-Curvature

   Eunbyung Park, Junier B. Oliva · 2019 · Neural Information Processing Systems · cited 144 times · Not summarized yet

   Meta-Curvature learns gradient transformations for fast task adaptation, using meta-learning for a different model component.

   Abstract

   We propose meta-curvature (MC), a framework to learn curvature information for better generalization and fast model adaptation. MC expands on the model-agnostic meta-learner (MAML) by learning to transform the gradients in the inner optimization such that the transformed gradients achieve better generalization performance to a new task. For training large scale neural networks, we decompose the curvature matrix into smaller matrices in a novel scheme where we capture the dependencies of the model's parameters with a series of tensor products. We demonstrate the effects of our proposed method on several few-shot learning tasks and datasets. Without any task specific techniques and architectur…

5. Related Same method, another problem A method it uses Not relevant

   Learning to Convolve: A Generalized Weight-Tying Approach

   Nichita Diaconu, Daniel E. Worrall · 2019 · International Conference on Machine Learning · cited 88 times · Not summarized yet

   Learning to Convolve learns filter transformations and weight tying for roto-translation equivariance, rather than discovering sharing patterns from data.

   Abstract

   Recent work (Cohen & Welling, 2016) has shown that generalizations of convolutions, based on group theory, provide powerful inductive biases for learning. In these generalizations, filters are not only translated but can also be rotated, flipped, etc. However, coming up with exact models of how to rotate a 3 x 3 filter on a square pixel-grid is difficult. In this paper, we learn how to transform filters for use in the group convolution, focussing on roto-translation. For this, we learn a filter basis and all rotated versions of that filter basis. Filters are then encoded by a set of rotation invariant coefficients. To rotate a filter, we switch the basis. We demonstrate we can produce featur…

Papers that cite it

1. Close Builds on this paper Uses this paper in depth Not relevant

   Generative Adversarial Symmetry Discovery

   Jianwei Yang, R. Walters, Nima Dehmamy et al. · 2023 · International Conference on Machine Learning · cited 58 times · Not summarized yet

   Generative Adversarial Symmetry Discovery learns equivariances from data through adversarially learned transformations and cites this paper.

   Abstract

   Despite the success of equivariant neural networks in scientific applications, they require knowing the symmetry group a priori. However, it may be difficult to know which symmetry to use as an inductive bias in practice. Enforcing the wrong symmetry could even hurt the performance. In this paper, we propose a framework, LieGAN, to automatically discover equivariances from a dataset using a paradigm akin to generative adversarial training. Specifically, a generator learns a group of transformations applied to the data, which preserve the original distribution and fool the discriminator. LieGAN represents symmetry as interpretable Lie algebra basis and can discover various symmetries such as …

2. Close Same method, another problem Not relevant

   Introducing Symmetries to Black Box Meta Reinforcement Learning

   Louis Kirsch, Sebastian Flennerhag, Hado van Hasselt et al. · 2021 · AAAI Conference on Artificial Intelligence · cited 36 times · Not summarized yet

   Introducing Symmetries to Black Box Meta Reinforcement Learning studies symmetry's role in meta-RL generalization, rather than learning equivariant parameter sharing.

   Abstract

   Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction. In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network. These methods are very flexible, but they tend to underperform compared to human-engineered RL algorithms in terms of generalisation to new, unseen environments. In this paper, we explore the role of symmetries in meta-generalisation. We show that a recent successful meta RL approach that meta-learns an objective for backpropagation-based learning exhibits certain symmetries (specifically the reuse of the learning rule, and invariance to input and output …

3. Close Builds on this paper Not relevant

   On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory

   Andrea Perin, Stéphane Deny · 2024 · Journal of machine learning research · cited 14 times · Not summarized yet

   On the Ability of Deep Networks to Learn Symmetries from Data studies data-driven symmetry learning theoretically and cites this paper.

   Abstract

   Symmetries (transformations by group actions) are present in many datasets, and leveraging them holds considerable promise for improving predictions in machine learning. In this work, we aim to understand when and how deep networks -- with standard architectures trained in a standard, supervised way -- learn symmetries from data. Inspired by real-world scenarios, we study a classification paradigm where data symmetries are only partially observed during training: some classes include all transformations of a cyclic group, while others -- only a subset. In the infinite-width limit, where kernel analogies apply, we derive a neural kernel theory of symmetry learning. The group-cyclic nature of …

4. Close Same method, another problem Not relevant

   Learning Symmetric Embeddings for Equivariant World Models

   Jung Yeon Park, Ondrej Biza, Linfeng Zhao et al. · 2022 · International Conference on Machine Learning · cited 64 times · Not summarized yet

   Learning Symmetric Embeddings learns representations whose features transform predictably, applying symmetry learning to equivariant world models.

   Abstract

   Incorporating symmetries can lead to highly data-efficient and generalizable models by defining equivalence classes of data samples related by transformations. However, characterizing how transformations act on input data is often difficult, limiting the applicability of equivariant models. We propose learning symmetric embedding networks (SENs) that encode an input space (e.g. images), where we do not know the effect of transformations (e.g. rotations), to a feature space that transforms in a known manner under these operations. This network can be trained end-to-end with an equivariant task network to learn an explicitly symmetric representation. We validate this approach in the context of…

5. Close Broader context Not relevant

   Using and Abusing Equivariance

   Tom Edixhoven, A. Lengyel, J. V. van Gemert · 2023 · 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 6 times · Not summarized yet

   Using and Abusing Equivariance studies how approximate equivariance affects generalization, rather than learning symmetries through parameter sharing.

   Abstract

   In this paper we show how Group Equivariant Convolutional Neural Networks use subsampling to learn to break equivariance to the rotation and reflection symmetries. We focus on the 2D rotations and reflections and investigate the impact of the broken equivariance on network performance. We show that a change in the input dimension of a network as small as a single pixel can be enough for commonly used architectures to become approximately equivariant, rather than exactly. We investigate the impact of networks not being exactly equivariant and find that approximately equivariant networks generalise significantly worse to unseen symmetries compared to their exactly equivariant counterparts. How…