# Efficient Benchmarking (of Language Models)

**Authors:** Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Leshem Choshen  
**arXiv:** [2308.11696](https://arxiv.org/abs/2308.11696)  
**Date:** 2023-08-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.11696v1/

## Key points

- Increasing versatility of language models (LMs) has led to comprehensive benchmarks
- Benchmarks have significant computational costs, requiring thousands of GPU hours per model
- Little discussion in literature about efficiency aspect of evaluating LMs
- Authors address problem by reducing computation costs without compromising reliability
- Use HELM benchmark as test case to investigate design choices' impact on computation and reliability tradeoff
- Introduce new measure called Decision Impact on Reliability (DIoR) to evaluate reliability of design choices
- Small changes in benchmark design can have significant impact on ranking
- Only a handful of examples needed for accurate benchmark ranking
- Slight variations in HELM scenarios can lead to widely varying rankings
- Authors provide concrete recommendations for more efficient benchmark design and utilization practices
- Recommendations can lead to dramatic cost savings while minimizing loss of benchmark reliability, reducing computation by up to 100x or more
- Study highlights importance of considering efficiency in LM evaluation benchmarks and offers practical solutions for optimizing benchmarking processes.

## Summary

The increasing versatility of language models (LMs) has led to the development of comprehensive benchmarks that assess a wide range of capabilities. However, these benchmarks come with significant computational costs, often requiring thousands of GPU hours per model. Despite this, there has been little discussion in the literature about the efficiency aspect of evaluating LMs. In this study, the authors address the problem of efficient benchmarking by intelligently reducing the computation costs without compromising reliability. They use the HELM benchmark as a test case to investigate how different design choices affect the tradeoff between computation and reliability. To evaluate the reliability of these decisions, they propose a new measure called Decision Impact on Reliability (DIoR). The authors find that even small changes in benchmark design can have a significant impact on ranking. For example, removing a low-ranked model from the benchmark can change its current leader on HELM. They also observe that only a handful of examples are needed to obtain an accurate benchmark ranking. On the other hand, slight variations in HELM scenarios can lead to widely varying rankings. Based on their findings, the authors provide concrete recommendations for more efficient benchmark design and utilization practices. These recommendations can lead to dramatic cost savings while minimizing loss of benchmark reliability, often reducing computation by up to 100x or more. Overall, this study highlights the importance of considering efficiency in LM evaluation benchmarks and provides valuable insights into how different design choices can impact computation-reliability tradeoffs. The proposed recommendations offer practical solutions for researchers and practitioners looking to optimize their benchmarking processes.