# An Adaptive Tangent Feature Perspective of Neural Networks

**Authors:** Daniel LeJeune, Sina Alemohammad  
**arXiv:** [2308.15478](https://arxiv.org/abs/2308.15478)  
**Date:** 2023-08-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2308.15478v1/

## Key points

- Authors propose a framework for understanding feature learning in neural networks
- Linear models in tangent feature space are studied
- Features can be transformed during training and linear transformations of features are considered
- Joint optimization problem over parameters and transformations with a bilinear interpolation constraint is formulated
- Specialized analysis on neural network structures provides insights into how features and kernel function change
- Experiments conducted on real neural networks using a simple regression problem
- Adaptive feature implementation of tangent feature classification evaluated on MNIST and CIFAR-10 datasets
- Results show that adaptive feature model has lower sample complexity compared to fixed tangent feature model
- Framework introduces understanding of feature adaptivity in neural networks and insights into evolution of features and kernel functions during training
- Further research needed to fully characterize real neural networks and understand extent of adaptivity in practice.

## Summary

In this paper, the authors propose a framework for understanding feature learning in neural networks by studying linear models in tangent feature space. They allow the features to be transformed during training and consider linear transformations of features. This leads to a joint optimization problem over parameters and transformations with a bilinear interpolation constraint that can be equivalently formulated as a linearly constrained optimization with structured regularization. The authors specialize their analysis to neural network structures and gain insights into how the features and kernel function change, providing additional nuance to the phenomenon of kernel alignment when the target function is poorly represented using tangent features. To verify their theoretical observations, they conduct experiments on real neural networks using a simple regression problem and evaluate an adaptive feature implementation of tangent feature classification on MNIST and CIFAR-10 datasets. The results show that the adaptive feature model has an order of magnitude lower sample complexity compared to the fixed tangent feature model. Overall, this work introduces a framework for understanding feature adaptivity in neural networks and provides insights into how features and kernel functions evolve during training. The empirical results demonstrate its effectiveness in improving sample complexity compared to fixed tangent features; however further research is needed to fully characterize real neural networks and understand the extent of adaptivity in practice.