Preprint · 2026

NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents

Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, Ge Zhang

Abstract

Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.

BibTeX Copy

@misc{ding_nl2repo-bench_2026,
	title = {{NL2Repo}-{Bench}: {Towards} {Long}-{Horizon} {Repository} {Generation} {Evaluation} of {Coding} {Agents}},
	shorttitle = {{NL2Repo}-{Bench}},
	url = {http://arxiv.org/abs/2512.12730},
	doi = {10.48550/arXiv.2512.12730},
	abstract = {Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40\% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.},
	urldate = {2026-11-06},
	publisher = {arXiv},
	author = {Ding, Jingzhe and Long, Shengda and Pu, Changxin and Zhou, Huan and Gao, Hongwan and Gao, Xiang and He, Chao and Hou, Yue and Hu, Fei and Li, Zhaojian and Shi, Weiran and Wang, Zaiyuan and Zan, Daoguang and Zhang, Chenchen and Zhang, Xiaoxu and Chen, Qizhi and Cheng, Xianfu and Deng, Bo and Gu, Qingshui and Hua, Kai and Lin, Juntao and Liu, Pai and Li, Mingchen and Pan, Xuanguang and Peng, Zifan and Qin, Yujia and Shan, Yong and Tan, Zhewen and Xie, Weihao and Wang, Zihan and Yuan, Yishuo and Zhang, Jiayu and Zhao, Enduo and Zhao, Yunfei and Zhu, He and Zhu, Liya and Zou, Chenyang and Ding, Ming and Jiao, Jianpeng and Liu, Jiaheng and Liu, Minghao and Liu, Qian and Tao, Chongyang and Yang, Jian and Yang, Tong and Zhang, Zhaoxiang and Chen, Xinjie and Huang, Wenhao and Zhang, Ge},
	month = jan,
	year = {2026},
	note = {arXiv:2512.12730 [cs.CL]},
	keywords = {Computer Science - Computation and Language},
}