Preprint · 2025

ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code

Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, Nick Haber

Abstract

Large language models (LLMs) have shown promise in transforming machine learning research, yet their capability to faithfully implement novel ideas from recent research papers-ideas unseen during pretraining-remains unclear. We introduce ResearchCodeBench, a benchmark of 212 coding challenges that evaluates LLMs' ability to translate cutting-edge ML contributions from top 2024-2025 research papers into executable code. We assessed 30+ proprietary and open-source LLMs, finding that even the best models correctly implement less than 40% of the code. We find Gemini-2.5-Pro-Preview to perform best at 37.3% success rate, with O3 (High) and O4-mini (High) following behind at 32.3% and 30.8% respectively. We present empirical findings on performance comparison, contamination, and error patterns. By providing a rigorous and community-driven evaluation platform, ResearchCodeBench enables continuous understanding and advancement of LLM-driven innovation in research code generation.

BibTeX Copy

@misc{hua_researchcodebench_2025,
	title = {{ResearchCodeBench}: {Benchmarking} {LLMs} on {Implementing} {Novel} {Machine} {Learning} {Research} {Code}},
	shorttitle = {{ResearchCodeBench}},
	url = {http://arxiv.org/abs/2506.02314},
	doi = {10.48550/arXiv.2506.02314},
	abstract = {Large language models (LLMs) have shown promise in transforming machine learning research, yet their capability to faithfully implement novel ideas from recent research papers-ideas unseen during pretraining-remains unclear. We introduce ResearchCodeBench, a benchmark of 212 coding challenges that evaluates LLMs' ability to translate cutting-edge ML contributions from top 2024-2025 research papers into executable code. We assessed 30+ proprietary and open-source LLMs, finding that even the best models correctly implement less than 40\% of the code. We find Gemini-2.5-Pro-Preview to perform best at 37.3\% success rate, with O3 (High) and O4-mini (High) following behind at 32.3\% and 30.8\% respectively. We present empirical findings on performance comparison, contamination, and error patterns. By providing a rigorous and community-driven evaluation platform, ResearchCodeBench enables continuous understanding and advancement of LLM-driven innovation in research code generation.},
	urldate = {2026-11-06},
	publisher = {arXiv},
	author = {Hua, Tianyu and Hua, Harper and Xiang, Violet and Klieger, Benjamin and Truong, Sang T. and Liang, Weixin and Sun, Fan-Yun and Haber, Nick},
	month = jun,
	year = {2025},
	note = {arXiv:2506.02314 [cs.AI]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
}