# Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

**Authors:** Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, Martin Vechev  
**arXiv:** [2602.11988](https://arxiv.org/abs/2602.11988)  
**Date:** 2026-02-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2602.11988v1/

## Key points

- Context files are commonly used in software development to tailor coding agents to repositories either manually or automatically.
- A comprehensive evaluation was conducted by the researchers using two settings: established SWE-bench tasks with LLM-generated context files and a novel collection of issues with developer-committed context files.
- The study found that context files tended to decrease task success rates and increase inference cost by over 20% compared to providing no repository context.
- Both LLM-generated and developer-provided context files encouraged broader exploration but ultimately made tasks harder due to unnecessary requirements.
- The study introduced AGENTBENCH, a benchmark for assessing the impact of context files on coding agents' performance in real-world tasks.
- Evaluation on AGENTBENCH and SWE-BENCH LITE revealed that LLM-generated context files generally decreased agent performance, while developer-written context files showed a slight improvement.
- Context files led to more thorough testing and exploration by coding agents according to an in-depth analysis of agent traces.
- Human-written context files should only include minimal requirements to avoid unnecessarily challenging coding agents.

## Summary

By Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev, the authors investigate the effectiveness of context files in software development. Context files are commonly used to tailor coding agents to repositories either manually or automatically. Despite being encouraged by agent developers, there has been a lack of rigorous investigation into their actual impact on real-world tasks. The researchers conducted a comprehensive evaluation using two settings: established SWE-bench tasks from popular repositories with LLM-generated context files based on agent-developer recommendations and a novel collection of issues from repositories with developer-committed context files. Surprisingly, the study found that context files tended to decrease task success rates compared to providing no repository context while also increasing inference cost by over 20%. Both LLM-generated and developer-provided context files encouraged broader exploration by coding agents but ultimately made tasks harder due to unnecessary requirements. The key contributions of the study include the introduction of AGENTBENCH, a curated benchmark for assessing the impact of context files on coding agents' performance in real-world tasks. The evaluation involved different coding agents and models on AGENTBENCH and SWE-BENCH LITE, revealing that LLM-generated context files generally decreased agent performance across various models or prompts used for generation. However, developer-written context files showed a slight improvement in agent performance. Furthermore, an in-depth analysis of agent traces demonstrated that context files led to more thorough testing and exploration by coding agents. The findings suggest that human-written context files should only include minimal requirements to avoid making tasks unnecessarily challenging for coding agents. Overall, this research sheds light on the importance of carefully considering the role of repository-level context files in optimizing coding agent performance in software development tasks.