# From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

**Authors:** Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, Mihai Surdeanu  
**arXiv:** [2404.07544](https://arxiv.org/abs/2404.07544)  
**Date:** 2024-04-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.07544v1/

## Key points

- Investigated capabilities of large language models (LLMs) in regression tasks without additional training
- Pre-trained models like GPT-4 and Claude 3 can rival or outperform traditional supervised methods in regression tasks
- Claude 3 showed superior performance on challenging datasets like Friedman #2 compared to other supervised methods
- Performance of LLMs scales with number of in-context exemplars, achieving sub-linear regret based on online learning principles
- Addressed limitations including reliance on proprietary models, potential data contamination issues, and released results from open-weight models for comparison
- Empirical evidence supports effectiveness of LLMs in regression tasks but lacks theoretical explanations
- Study provides valuable insights into regression capabilities of LLMs without needing additional training or updates

## Summary

In this study, we investigated the capabilities of large language models (LLMs) in performing regression tasks without any additional training. Our analysis focused on various pre-trained models such as GPT-4 and Claude 3, highlighting their ability to rival or even outperform traditional supervised methods like Random Forest and Gradient Boosting in regression tasks. Notably, Claude 3 demonstrated superior performance on challenging datasets like Friedman #2 compared to other supervised methods. Furthermore, we explored how the performance of LLMs scales with the number of in-context exemplars, demonstrating their capacity to achieve sub-linear regret based on online learning principles. We also addressed several limitations in our study, including the reliance on proprietary models whose performance may vary over time and potential data contamination issues due to opaque training datasets for LLMs. To mitigate these challenges, we included results from leading open-weight models alongside proprietary ones and conducted experiments with multiple random seeds on new datasets. Additionally, we released intermediate results and considered a publicly available model like Falcon 40B for comparison. While our empirical evidence supports the effectiveness of LLMs in regression tasks, we acknowledge the lack of theoretical explanations to underpin these observations. Overall, our study provides valuable insights into the regression capabilities of large language models when presented with in-context examples, highlighting their potential as proficient regressors without the need for additional training or updates.