ReplicHAI › Agentic reproduction: systems & benchmarks

Shared by Changkun Ou with everyone

Year ↓	Title added	Authors	Venue
2026	AI Coding Agents Can Reproduce Social Science Findings	Alizadeh et al.	arXiv
2026	AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science	Brodeur et al.	Proceedings of the National Academy of Sciences
2026	An Agentic Approach Towards Replication Package Quality Evaluation	Mbida, Angermeir	arXiv
2026	ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review	Riehl et al.	arXiv
2026	Automated reproducibility assessments in the social and behavioral sciences using large language models	Holtdirk et al.	arXiv
2026	Automated Reproducibility Has a Problem Statement Problem	Snelleman et al.	RAI Workshop at AAAI 2026
2026	Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches	Shah et al.	arXiv
2026	AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage	Zhao et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	Can Coding Agents Reproduce Findings in Computational Materials Science?	Huang et al.	arXiv
2026	CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents	Jansen et al.	arXiv
2026	Coding-agents can replicate scientific machine learning papers	Hans, Bilionis	arXiv
2026	Does It Reproduce? An open, auditable archive of reproducibility assessments of scientific publications	Hamburg-Eppendorf
2026	FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights	Wang et al.	Proceedings of the 43rd International Conference on Machine Learning
2026	Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation	Kapoor et al.	International Conference on Learning Representations (ICLR)
2026	Life After Benchmark Saturation: A Case Study of CORE-Bench	Nadgir et al.	arXiv
2026	MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility	Gaddipati et al.	arXiv
2026	On the Computational Reproducibility of Human-Computer Interaction	Iarygina et al.	Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
2026	Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning	Seo et al.	International Conference on Learning Representations (ICLR)
2026	PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers	Zhang et al.	arXiv
2026	PRBench: End-to-end Paper Reproduction in Physics Research	Qiu	arXiv
2026	Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results	Kohler et al.	arXiv
2026	Reimagining research papers as interactive and reliable AI agents	Miao et al.	Nature
2026	ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences	Nguyen et al.	Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26)
2026	Reproducing ICML 2026 — Open Reproductions	Face, alphaXiv
2026	ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues	Li et al.	arXiv
2026	Revibing Code from Papers: Reimplementing HCI Artifacts	Adar et al.	Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology
2026	Revibing Code from Papers: Reimplementing HCI Artifacts	Adar et al.	Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26)
2026	RExBench: Can coding agents autonomously implement AI research extensions?	Edwards et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis	Xu, Yang	arXiv
2026	SciCoQA: Quality Assurance for Scientific Paper--Code Alignment	Baumgärtner, Gurevych	arXiv
2026	The AI Replication Engine V2: Reading the Page, Mapping the Literature	Barbarioli	Institute for Replication blog
2026	The AI Replication Engine: First Experiments and What's Next	Barbarioli	Institute for Replication (I4R)
2026	The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research	Bai et al.	arXiv
2026	Towards a Science of AI Agent Reliability	Rabanser et al.	Proceedings of the 43rd International Conference on Machine Learning (ICML)
2026	Towards Automating Scientific Review with Google's Paper Assistant Tool	Jayaram et al.	arXiv
2026	VERITAS: Towards a General-Purpose Replication Tool for Scientific Research	Liu et al.	arXiv
2026	What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations	Luo et al.	arXiv
2026	What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction	Li et al.	arXiv
2026	What We Learned by Reproducing 2,200 papers from ICML	Abid et al.	Hugging Face Blog
2025	Exploring the use of AI authors and reviewers at Agents4Science	Bianchi et al.	arXiv
2025	From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking	Kim et al.	arXiv
2025	LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research	Yan et al.	Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
2025	PaperBench: Evaluating AI's Ability to Replicate AI Research	Starace et al.	Proceedings of the 42nd International Conference on Machine Learning
2025	RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback	Miao et al.	arXiv
2025	Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification	Zhou et al.	arXiv
2025	ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?	Ye et al.	arXiv
2025	REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?	Hu et al.	Findings of the Association for Computational Linguistics: ACL 2025
2025	ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code	Hua et al.	Advances in Neural Information Processing Systems
2025	SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers	Xiang et al.	Conference on Language Modeling (COLM)
2025	The 2025 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results	Belz et al.	Proceedings of the 4th Workshop on Generation, Evaluation and Metrics
2024	AI Agents That Matter	Kapoor et al.	arXiv
2024	CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark	Siegel et al.	Transactions on Machine Learning Research
2024	SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories	Bogin et al.	Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

/ filter · j k or ↑ ↓ select · Enter open page · c copy BibTeX