# AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders

**Authors:** Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, Christopher Potts  
**arXiv:** [2501.17148](https://arxiv.org/abs/2501.17148)  
**Date:** 2025-01-28  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2501.17148v2/

## Key points

- Fine-grained steering of language model outputs is crucial for safety and reliability in natural language processing.
- Common techniques used for steering include prompting and finetuning, as well as representation-based methods such as sparse autoencoders (SAEs), linear artificial tomography, supervised steering vectors, linear probes, and representation finetuning.
- A lack of a standardized benchmark for comparing these approaches led to the introduction of AxBench, a large-scale benchmark specifically designed for steering and concept detection tasks.
- Prompting outperformed all existing methods in steering performance, followed by finetuning. However, representation-based methods like difference-in-means showed superior performance in concept detection tasks.
- Sparse autoencoders (SAEs) were not competitive in either evaluation.
- A novel weakly-supervised representational method called Rank-1 Representation Finetuning (ReFT-r1) was introduced and proved to be competitive on both steering and concept detection tasks while offering interpretability advantages over prompting.
- SAE-scale feature dictionaries were trained and publicly released for ReFT-r1 and DiffMean by the researchers.

## Summary

, , , , 

In the field of natural language processing, fine-grained steering of language model outputs is crucial for ensuring safety and reliability. Prompting and finetuning are commonly used techniques to achieve this, but researchers have also explored representation-based methods such as sparse autoencoders (SAEs), linear artificial tomography, supervised steering vectors, linear probes, and representation finetuning. However, there has been a lack of a standardized benchmark for directly comparing these different approaches. To address this gap, a team of researchers introduced AxBench, a large-scale benchmark specifically designed for steering and concept detection tasks. Through experiments conducted on Gemma-2-2B and 9B datasets, they found that prompting outperformed all existing methods in terms of steering performance, followed by finetuning. On the other hand, for concept detection tasks, representation-based methods like difference-in-means showed superior performance compared to others. Interestingly, sparse autoencoders (SAEs) were not competitive in either evaluation. In addition to evaluating existing methods, the researchers introduced a novel weakly-supervised representational method called Rank-1 Representation Finetuning (ReFT-r1). This new approach proved to be competitive on both steering and concept detection tasks while offering interpretability advantages that prompting lacks. Along with AxBench, the researchers also trained and publicly released SAE-scale feature dictionaries for ReFT-r1 and DiffMean. Overall, the study highlights the importance of fine-grained steering in language models and provides valuable insights into the effectiveness of different steering techniques through comprehensive evaluations on AxBench. The introduction of ReFT-r1 as a promising new method further enriches the landscape of interpretability-focused approaches in language model steering research.