Preprint · 2025

RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback

Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang, Bowei He, Xinni Zhang, Dianzhi Yu, Hanchen Yang, Hoang H. Nguyen, Yue Zhou, Jie Yang, Jizhou Guo, Wenzhe Fan, Chin-Yuan Yeh, Panpan Meng, Liancheng Fang, Jinhu Qi, Wei-Chieh Huang, Zhengyao Gu, Yuwei Han, Langzhou He, Yuyao Yang, Yinghui Li, Hai-Tao Zheng, Xue Liu, Irwin King, Philip S. Yu

Abstract

Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLM agents through multi-turn interactions with LLM-simulated human feedback. It includes structured instructions,unit tests, and a five-level feedback hierarchy to reflect realistic researcher-agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experiments with leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation

BibTeX Copy

@misc{miao_recode-h_2025,
	title = {{RECODE}-{H}: {A} {Benchmark} for {Research} {Code} {Development} with {Interactive} {Human} {Feedback}},
	shorttitle = {{RECODE}-{H}},
	url = {http://arxiv.org/abs/2510.06186},
	doi = {10.48550/arXiv.2510.06186},
	abstract = {Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLM agents through multi-turn interactions with LLM-simulated human feedback. It includes structured instructions,unit tests, and a five-level feedback hierarchy to reflect realistic researcher-agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experiments with leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation},
	urldate = {2026-11-05},
	publisher = {arXiv},
	author = {Miao, Chunyu and Zou, Henry Peng and Li, Yangning and Chen, Yankai and Wang, Yibo and Wang, Fangxin and Li, Yifan and Yang, Wooseong and He, Bowei and Zhang, Xinni and Yu, Dianzhi and Yang, Hanchen and Nguyen, Hoang H. and Zhou, Yue and Yang, Jie and Guo, Jizhou and Fan, Wenzhe and Yeh, Chin-Yuan and Meng, Panpan and Fang, Liancheng and Qi, Jinhu and Huang, Wei-Chieh and Gu, Zhengyao and Han, Yuwei and He, Langzhou and Yang, Yuyao and Li, Yinghui and Zheng, Hai-Tao and Liu, Xue and King, Irwin and Yu, Philip S.},
	month = oct,
	year = {2025},
	note = {arXiv:2510.06186 [cs.CL]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
}