SEBGen: A Self-Evolving Benchmark Generation Framework for Software Engineering Agents

Source abstract

Static software engineering benchmarks such as SWE-bench suffer from three fundamental limitations: task sets are fixed and become saturated as model capabilities rapidly improve; expert construction is costly and slow; and difficulty levels cannot adapt to evolving model abilities. We present SEBGen, a self-evolving benchmark generation framework that synthesizes evaluation tasks from code repositories and improves its generation strategies through an evolution loop. SEBGen introduces three key innovations: an explicit five-dimension exploration value evaluator that quantitatively scores generation strategies across quality, novelty, difficulty coverage, skill diversity, and evolution potential, replacing implicit LLM judgments; a five-layer hybrid validation pipeline combining mechanical checks with LLM-as-judge evaluation; and a self-evolving loop with archive-based experience accumulation, sigmoid-scaled parent selection, and LLM-driven strategy mutation. On a synthetic corpus, SEBGen achieves a 96 percent validation pass rate with stable quality across five evolution generations. On a real repository, it transforms all six tqdm bug-fix commits into validated tasks, and a comparison against SWE-bench Lite shows task quality of 0.76 approaching expert baselines of 0.80. These results indicate that dynamic, self-evolving benchmarks are a viable alternative to static benchmarks.