# Linear Transformers with Learnable Kernel Functions are Better In-Context Models

**Authors:** Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov  
**arXiv:** [2402.10644](https://arxiv.org/abs/2402.10644)  
**Date:** 2024-02-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.10644v1/

## Key points

- Advancing subquadratic architectures for Language Models (LMs) is crucial in the field of natural language processing
- State Space Models have surpassed Transformer performance on language modeling tasks
- Based model emerged as a hybrid solution blending Linear Transformer with a kernel inspired by Taylor expansion and convolutional networks
- Attention models excel on longer sequences, outperforming non-attention counterparts like Based on Multi-Query Associative Recall (MQAR) task
- Need for further research to bridge performance gap between attention-based methods and alternatives like Based
- Challenges persist with Based model compared to traditional Transformers when handling long sequences with smaller models
- Proposed alteration to Based kernel aims to address scalability issues and improve efficiency in similarity calculations between queries and keys
- Focus on refining the Based model's kernel function to enhance In-Context Learning abilities and performance on tasks like MQAR and overall language modeling processes
- Effectiveness of the method in scenarios involving intensive copying or recalling previous context requires further investigation
- Experiments limited to academic-scale models provide valuable insights but may impact generalizability to larger models.

## Summary

Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, have surpassed Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally excels. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions and augmented by convolutional networks. Mirroring the Transformer's adeptness in-context learning, it became a strong contender in the field. Our findings reveal a significant disparity in handling the Multi-Query Associative Recall (MQAR) task between attention-based models and others such as Based, particularly as sequence lengths increase. Attention models excel on longer sequences and significantly outperform their non-attention counterparts. These results underscore the need for further research into strategies that could bridge this gap to achieve the performance of attention-based methods. Future studies could explore ways to match or exceed the superior aspects of attention mechanisms on tasks requiring associative recall. Despite improvements made with our proposed alteration to the Based kernel, challenges persist compared to traditional Transformers when handling long sequences with smaller models. Addressing this issue remains a primary focus of our work. To fully understand the Based architecture, it is essential to discuss Linear Transformers and their reliance on an attention mechanism that incurs computational inefficiencies with growing sequence lengths. To address scalability issues, Katharopoulos et al. (2020) proposed using a non-linear kernel function ϕ(·) as an approximation for similarity calculations between queries and keys. This approach led to linear complexity with sequence length and improved efficiency compared to the original Transformer model. The Based model introduced by Arora et al. (2023) utilized a novel kernel function inspired by the Taylor series expansion of exponential functions to enhance In-Context Learning abilities. Our study focuses on refining this kernel further to amplify its performance on tasks like MQAR and overall language modeling processes evaluated on datasets like Pile. While our method shows promise across various NLP tasks typically handled by Transformers, its effectiveness in scenarios involving intensive copying or recalling previous context requires further investigation. Additionally, our experiments are limited to academic-scale models, which may impact generalizability to larger models but still provide valuable insights into potential efficacy. In conclusion, our work highlights the importance of advancing subquadratic architectures for LMs and bridging performance gaps between attention-based methods and alternatives like Based through innovative approaches like refined kernel functions. Future research directions could lead to improved models capable of processing long sequences efficiently across diverse NLP tasks.