# FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

**Authors:** Lingjiao Chen, Matei Zaharia, James Zou  
**arXiv:** [2305.05176](https://arxiv.org/abs/2305.05176)  
**Date:** 2023-05-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.05176v1/

## Key points

- Increasing number of large language models (LLMs) available for querying
- Varying pricing structures and significant differences in fees between LLM APIs
- Three strategies proposed to reduce inference cost associated with using LLMs:
- Prompt adaptation: Modifying prompts for more accurate results with fewer queries
- LLM approximation: Using simpler models as substitutes for certain queries
- LLM cascade: Combining multiple LLMs flexibly to optimize cost and accuracy based on query types
- FrugalGPT presented as an example implementation of the LLM cascade strategy
- FrugalGPT achieves up to 98% cost reduction while maintaining or improving accuracy compared to GPT-4
- Strategies provide a foundation for sustainable and efficient use of LLMs, optimizing cost savings without compromising performance.

## Summary

In their paper titled "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance," authors Lingjiao Chen, Matei Zaharia, and James Zou address the issue of the increasing number of large language models (LLMs) that users can query for a fee. They specifically focus on popular LLM APIs such as GPT-4, ChatGPT, and J1-Jumbo. The authors begin by reviewing the cost associated with querying these LLM APIs and find that they have varying pricing structures. In fact, the fees can differ significantly by two orders of magnitude. This discrepancy in pricing becomes particularly problematic when using LLMs on large collections of queries and text, as it can quickly become expensive. Motivated by this issue, the authors propose three strategies that users can employ to reduce the inference cost associated with using LLMs. The first strategy is prompt adaptation, which involves modifying the prompts given to LLMs to achieve more accurate results with fewer queries. The second strategy is LLM approximation, where simpler models are used as substitutes for certain queries instead of relying solely on expensive LLMs. Lastly, they introduce the concept of LLM cascade, which involves combining multiple LLMs in a flexible manner to optimize cost and accuracy based on different types of queries. To demonstrate their ideas in practice, the authors present FrugalGPT as an example implementation of the LLM cascade strategy. FrugalGPT learns which combinations of LLMs to use for different queries in order to minimize cost while maintaining or even improving accuracy compared to individual LLMs like GPT-4. Through experiments, they show that FrugalGPT can achieve up to 98% cost reduction while matching GPT-4's performance or improve accuracy over GPT-4 by 4% at the same cost. Overall, this research lays a foundation for using LLMs sustainably and efficiently by providing strategies to reduce inference costs. By utilizing prompt adaptation, LLM approximation, and the LLM cascade approach exemplified by FrugalGPT , users can optimize their use of LLMs and achieve cost savings without compromising performance.