01 Research replication & science

Shared by Changkun Ou with everyone

Year ↓	Title added	Authors	Venue
2026	AI Coding Agents Can Reproduce Social Science Findings	Alizadeh et al.	arXiv
2026	AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery	Liu et al.	arXiv
2026	AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite	Bragg et al.	arXiv
2026	AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage	Zhao et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	Can Coding Agents Reproduce Findings in Computational Materials Science?	Huang et al.	arXiv
2026	Coding-agents can replicate scientific machine learning papers	Hans, Bilionis	arXiv
2026	CompMat-Bench: Benchmarking AI Agents for Computational Materials Science	Zhang et al.	arXiv
2026	CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark	Siegel et al.	arXiv
2026	EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights	Geng et al.	arXiv
2026	FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights	Wang et al.	arXiv
2026	Into the Unknown: Benchmarking AI agents on open scientific research problems	Kim et al.	arXiv
2026	Life After Benchmark Saturation: A Case Study of CORE-Bench	Nadgir et al.	arXiv
2026	MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility	Gaddipati et al.	arXiv
2026	PAPERBENCHX: Can AI Reproduce Science Across Disciplines?	Wu et al.	arXiv
2026	PRBench: End-to-end Paper Reproduction in Physics Research	Qiu et al.	arXiv
2026	Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results	Kohler et al.	arXiv
2026	RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?	Salunkhe et al.	arXiv
2026	ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences	Nguyen et al.	Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
2026	Revibing Code from Papers: Reimplementing HCI Artifacts	Adar et al.	Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology
2026	RExBench: Can coding agents autonomously implement AI research extensions?	Edwards et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	Terminal-Bench-Science 0.1	team
2026	The AI Replication Engine: First Experiments and What's Next	(I4R)
2026	What We Learned by Reproducing 2,200 Papers from ICML	Face
2025	AstaBench: Rigorous benchmarking of AI agents with a holistic scientific research suite	Ai2
2025	BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology	Mitchener et al.	arXiv
2025	EXP-Bench: Can AI Conduct AI Research Experiments?	Kon et al.	arXiv
2025	From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking	Kim et al.	arXiv
2025	LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research	Yan et al.	arXiv
2025	MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?	Zhang et al.	arXiv
2025	PaperBench: Evaluating AI's Ability to Replicate AI Research	Starace et al.	arXiv
2025	ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?	Ye et al.	arXiv
2025	REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?	Hu et al.	Findings of the Association for Computational Linguistics: ACL 2025
2025	ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code	Hua et al.	arXiv
2025	ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery	Chen et al.	arXiv
2025	SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers	Xiang et al.	arXiv
2024	DiscoveryBench: Towards Data-Driven Discovery with Large Language Models	Majumder et al.	arXiv
2024	DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents	Jansen et al.	arXiv
2024	LAB-Bench: Measuring Capabilities of Language Models for Biology Research	Laurent et al.	arXiv
2024	SciCode: A Research Coding Benchmark Curated by Scientists	Tian et al.	arXiv
2024	SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories	Bogin et al.	arXiv
2022	ScienceWorld: Is your Agent Smarter than a 5th Grader?	Wang et al.	arXiv

/ filter · j k or ↑ ↓ select · Enter open page · c copy BibTeX