# Stop Comparing LLM Agents Without Disclosing the Harness

**Authors:** Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, Chandan K. Reddy  
**arXiv:** [2605.23950](https://arxiv.org/abs/2605.23950)  
**Date:** 2026-05-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2605.23950v1/

## Key points

- The execution harness plays a crucial role in determining agent performance for long-horizon tasks
- Performance variability is primarily driven by harness configuration rather than model choice (Binding Constraint Thesis)
- Small changes in the harness can lead to significant shifts in performance beyond what can be achieved by switching models
- Harness-induced variance can exceed model-induced variance, impacting model rankings
- A new evaluation framework is proposed that emphasizes disclosure of harness specifications and includes a variance decomposition protocol
- Researchers are encouraged to consider the impact of harnesses on experimental outcomes
- Benchmark designers are urged to incorporate harness variation into evaluation dimensions
- Practitioners are advised to view model selection as part of a larger optimization loop that includes consideration of the execution environment

## Summary

, , , , 

This position paper argues that in the realm of long-horizon tasks evaluated across models with comparable frontier capability, the execution harness plays a crucial role in determining agent performance. The infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model is often more influential than the model itself. The authors introduce the Binding Constraint Thesis, which posits that performance variability is primarily driven by harness configuration rather than model choice. They provide three lines of support for this thesis. Firstly, they formalize the concept by treating the harness as a controller of a closed-loop dynamical system and the language model as the policy it governs. This framework explains how small changes in the harness can lead to significant shifts in performance beyond what can be achieved by switching models. Secondly, through analysis of published benchmarks and industry deployments, they demonstrate that harness-induced variance can exceed model-induced variance, sometimes resulting in unexpected changes in model rankings. Lastly, they propose a new evaluation framework that emphasizes disclosure of harness specifications and includes a variance decomposition protocol. The paper addresses three key audiences: researchers are encouraged to consider the impact of harnesses on experimental outcomes; benchmark designers are urged to incorporate harness variation into evaluation dimensions; and practitioners are advised to view model selection as part of a larger optimization loop that includes consideration of the execution environment. The authors raise several open questions for further exploration within the research community. These include defining comparable frontier models without confounding factors related to harnesses, establishing metrics for measuring distance between different harness configurations, determining optimal levels of disclosure versus locked-harness evaluation, and developing trajectory-level diagnostics for assessing agent performance. In conclusion, the paper asserts that the execution harness is a critical factor influencing long-horizon language model-agent performance when benchmark leaderboards guide research and product decisions. They advocate for routine disclosure of Harness Cards detailing harness configurations, variance decompositions in evaluations where feasible, and trajectory-level metrics to assess recovery, drift, and control lag. Until these practices become standard, comparisons based on leaderboard rankings should be viewed as incomplete and potentially misleading.