# SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

**Authors:** Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, Bing Zhao  
**arXiv:** [2603.03823](https://arxiv.org/abs/2603.03823)  
**Date:** 2026-03-04  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2603.03823v1/

## Key points

- Large Language Model (LLM)-powered agents have shown impressive capabilities in automating tasks like static bug fixing
- Traditional one-shot repair paradigms are insufficient for handling long-term software development with requirement changes and feature iterations
- SWE-CI benchmark shifts the evaluation paradigm from short-term functional correctness to long-term maintainability in code generation
- SWE-CI is the first repository-level benchmark based on Continuous Integration, assessing agents' ability to sustain code quality over extended periods of evolution
- The benchmark comprises 100 tasks from real-world repositories with an average evolution history of 233 days and 71 consecutive commits
- SWE-CI focuses on dynamic maintainability, offering insights into an agent's ability to adapt and evolve code over time
- EvoScore in SWE-CI allows nuanced assessment of coding capabilities by measuring performance on future modifications
- State-of-the-art models excel in functional correctness but struggle with sustaining code quality over prolonged evolution periods

## Summary

, , , , 

In the realm of software engineering, Large Language Model (LLM)-powered agents have showcased impressive capabilities in automating tasks such as static bug fixing. This has been demonstrated by benchmarks like SWE-bench. However, the real-world scenario of developing mature software involves intricate requirement changes and long-term feature iterations. Traditional one-shot repair paradigms fail to capture this process. To address this gap, the innovative SWE-CI benchmark has been introduced, marking a shift in the evaluation paradigm for code generation from short-term functional correctness to long-term maintainability. SWE-CI stands out as the first repository-level benchmark built upon the Continuous Integration loop. Its aim is to assess how well agents can sustain code quality throughout extended periods of evolution. The benchmark comprises 100 tasks derived from real-world code repositories with an average evolution history spanning 233 days and 71 consecutive commits. SWE-CI challenges agents to systematically resolve these tasks through multiple rounds of analysis and coding iterations. By focusing on dynamic maintainability rather than static fixes, SWE-CI offers valuable insights into an agent's ability to adapt and evolve code over time. The motivation behind designing benchmarks like SWE-CI stems from the understanding that software quality naturally degrades over time as maintenance progresses. With maintenance activities accounting for a significant portion of total software lifecycle costs, there is a pressing need to evaluate models based on their capacity to maintain code effectively. The existing snapshot-style evaluation protocols used in benchmarks like HumanEval and LiveCodeBench overlook the crucial aspect of long-term code evolution. Agents that produce quick fixes may pass initial tests but struggle when faced with evolving requirements and changing interfaces. Through extensive experiments involving more than 10 billion tokens, it was observed that while state-of-the-art models excel in functional correctness tasks, they encounter challenges in sustaining code quality over prolonged evolution periods. The introduction of EvoScore as a proxy metric in SWE-CI enables a nuanced assessment of an agent's coding capabilities by measuring its performance on future modifications. This comprehensive evaluation approach sheds light on the distinctive diagnostic value of SWE-CI in gauging an agent's ability to maintain codebase integrity amidst evolving requirements. In conclusion, SWE-CI represents a groundbreaking initiative in evaluating LLM-based agents' long-term coding proficiency through continuous integration processes. By emphasizing maintainability alongside functional correctness, this benchmark offers valuable insights into how well agents can adapt and evolve codebases over extended periods of time.