ReplicHAI

Shared by Changkun Ou with everyone

Year ↓	Title added	Authors	Venue
2026	Agentic Coding 2.0 leaderboard	AI
2026	Agents Are Systems, Not Models: Rethinking Agentic Evaluation	Wiedmann et al.	arXiv
2026	Agents' Last Exam	Sun et al.	arXiv
2026	Agents' Last Exam leaderboard	team
2026	AI Coding Agents Can Reproduce Social Science Findings	Alizadeh et al.	arXiv
2026	AI Coding Agents Can Reproduce Social Science Findings	Alizadeh et al.	arXiv
2026	AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable	Alizadeh et al.	arXiv
2026	AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science	Brodeur et al.	Proceedings of the National Academy of Sciences
2026	AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery	Liu et al.	arXiv
2026	AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications	Zhao et al.	arXiv
2026	An Agentic Approach Towards Replication Package Quality Evaluation	Mbida, Angermeir	arXiv
2026	ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review	Riehl et al.	arXiv
2026	ARC-AGI-3	Foundation
2026	Artificial Analysis Intelligence Index v5	Analysis
2026	AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite	Bragg et al.	arXiv
2026	Automated reproducibility assessments in the social and behavioral sciences using large language models	Holtdirk et al.	arXiv
2026	Automated Reproducibility Has a Problem Statement Problem	Snelleman et al.	RAI Workshop at AAAI 2026
2026	Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches	Shah et al.	arXiv
2026	AutomationBench	Shepard, Salimans	arXiv
2026	AutomationBench leaderboard	Zapier
2026	AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage	Zhao et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage	Zhao et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents	Khanal et al.	arXiv
2026	BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business	Zhao et al.	arXiv
2026	Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents	Ma et al.	arXiv
2026	Can Coding Agents Reproduce Findings in Computational Materials Science?	Huang et al.	arXiv
2026	Can Coding Agents Reproduce Findings in Computational Materials Science?	Huang et al.	arXiv
2026	CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents	Jansen et al.	arXiv
2026	Coding-agents can replicate scientific machine learning papers	Hans, Bilionis	arXiv
2026	Coding-agents can replicate scientific machine learning papers	Hans, Bilionis	arXiv
2026	Company Knowledge Bench	Kapa.ai
2026	CompMat-Bench: Benchmarking AI Agents for Computational Materials Science	Zhang et al.	arXiv
2026	Continual Learning Bench (site and leaderboard)	Berkeley
2026	Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments	Asawa et al.	arXiv
2026	CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark	Siegel et al.	arXiv
2026	CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale	Wang et al.	arXiv
2026	DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints	Zhang et al.	arXiv
2026	DeepSWE v1.1	Datacurve
2026	Demystifying evals for AI agents	Anthropic
2026	Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations	Srinivasan	arXiv
2026	DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle	Tang et al.	arXiv
2026	Digital Work Index	AI
2026	Does It Reproduce? An open, auditable archive of reproducibility assessments of scientific publications	Hamburg-Eppendorf
2026	EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents	Huang et al.	arXiv
2026	EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights	Geng et al.	arXiv
2026	Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions	Hu et al.	arXiv
2026	FAIR-D: An Extension to the FAIR Principles for Open Hardware Replicability in Human-Computer Interaction	Simon et al.	Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems
2026	FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights	Wang et al.	arXiv
2026	FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights	Wang et al.	Proceedings of the 43rd International Conference on Machine Learning
2026	From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence	Theiler et al.	arXiv
2026	FrontierCode	Cognition
2026	Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking	Messing	arXiv
2026	Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking	Messing	arXiv
2026	Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation	Kapoor et al.	International Conference on Learning Representations (ICLR)
2026	How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines	Yagubyan	arXiv
2026	Human-AI Collaboration for Estimating Scientific Replicability	Chakravorti et al.	arXiv
2026	Humanity's Last Exam	Phan et al.	Nature
2026	Interaction Models: A Scalable Approach to Human-AI Collaboration		Thinking Machines Lab: Connectionism
2026	Into the Unknown: Benchmarking AI agents on open scientific research problems	Kim et al.	arXiv
2026	Investigating the analytical robustness of the social and behavioural sciences	Aczel et al.	Nature
2026	Investigating the replicability of the social and behavioural sciences	Tyner et al.	Nature
2026	Investigating the reproducibility of the social and behavioural sciences	Miske et al.	Nature
2026	It's Not Just the Prompt: Model Choice Dominates LLM Creative Output	Haase et al.	Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems
2026	KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards	Li et al.	arXiv
2026	LibraryDesignBench leaderboard	AI
2026	Life After Benchmark Saturation: A Case Study of CORE-Bench	Nadgir et al.	arXiv
2026	Life After Benchmark Saturation: A Case Study of CORE-Bench	Nadgir et al.	arXiv
2026	Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading	Li et al.	arXiv
2026	Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory	Zhou et al.	arXiv
2026	Many AI analysts, one dataset: Navigating the agentic data science multiverse	Bertran et al.	Proceedings of the National Academy of Sciences
2026	MBABench 2.0 leaderboard	AI
2026	MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers	Bandi et al.	arXiv
2026	Measuring AI Ability to Complete Long Software Tasks	Kwa et al.	arXiv
2026	Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents	Shen et al.	arXiv
2026	MirrorCode	AI
2026	MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility	Gaddipati et al.	arXiv
2026	MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility	Gaddipati et al.	arXiv
2026	NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents	Ding et al.	arXiv
2026	On the Computational Reproducibility of Human-Computer Interaction	Iarygina et al.	Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
2026	OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks	Yuan et al.	arXiv
2026	Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning	Seo et al.	International Conference on Learning Representations (ICLR)
2026	PAPERBENCHX: Can AI Reproduce Science Across Disciplines?	Wu et al.	arXiv
2026	PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers	Zhang et al.	arXiv
2026	PRBench: End-to-end Paper Reproduction in Physics Research	Qiu	arXiv
2026	PRBench: End-to-end Paper Reproduction in Physics Research	Qiu et al.	arXiv
2026	Predicting results of social science experiments using large language models	Ashokkumar et al.	Nature
2026	Progressive Autonomy as Preference Learning: A Formalization of Trust Calibration for Agentic Tool Use	Ou	arXiv
2026	Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results	Kohler et al.	arXiv
2026	Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results	Kohler et al.	arXiv
2026	RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?	Salunkhe et al.	arXiv
2026	Reimagining research papers as interactive and reliable AI agents	Miao et al.	Nature
2026	ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences	Nguyen et al.	Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
2026	ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences	Nguyen et al.	Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26)
2026	Reproducibility and robustness of economics and political science research	Brodeur et al.	Nature
2026	Reproducing ICML 2026 — Open Reproductions	Face, alphaXiv
2026	ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues	Li et al.	arXiv
2026	Revibing Code from Papers: Reimplementing HCI Artifacts	Adar et al.	Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology
2026	Revibing Code from Papers: Reimplementing HCI Artifacts	Adar et al.	Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology
2026	Revibing Code from Papers: Reimplementing HCI Artifacts	Adar et al.	Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26)
2026	RExBench: Can coding agents autonomously implement AI research extensions?	Edwards et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	RExBench: Can coding agents autonomously implement AI research extensions?	Edwards et al.	Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2026	Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis	Xu, Yang	arXiv
2026	SciCoQA: Quality Assurance for Scientific Paper--Code Alignment	Baumgärtner, Gurevych	arXiv
2026	SearchHorizonBench: A Benchmark for Search Value in AI Agents	Ning et al.	arXiv
2026	Senior SWE-bench leaderboard	AI
2026	SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks	Zhong et al.	arXiv
2026	SlopCode Bench leaderboard	AI
2026	SnorkelFinance 2.0 leaderboard	AI
2026	SnorkelLegal leaderboard	AI
2026	SnorkelManufacturing leaderboard	AI
2026	SnorkelRevOps leaderboard	AI
2026	SnorkelUnderwrite 2.0 leaderboard	AI
2026	Survey on Evaluation of LLM-based Agents	Yehudai et al.	arXiv
2026	SWE-bench CLI leaderboard	AI
2026	SWE-bench leaderboards	team
2026	SWE-Bench Pro V2	AI
2026	Terminal-Bench (site and leaderboard)	(Stanford
2026	Terminal-Bench 3.0	team
2026	Terminal-Bench 4.0 leaderboard	Institute
2026	Terminal-Bench-Science 0.1	team
2026	Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces	Merrill et al.	arXiv
2026	The AI Replication Engine V2: Reading the Page, Mapping the Literature	Barbarioli	Institute for Replication blog
2026	The AI Replication Engine: First Experiments and What's Next	Barbarioli	Institute for Replication (I4R)
2026	The AI Replication Engine: First Experiments and What's Next	(I4R)
2026	The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark	Spence et al.	arXiv
2026	The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research	Bai et al.	arXiv
2026	The threat of analytic flexibility in using large language models to simulate human data	Cummins	Advances in Methods and Practices in Psychological Science
2026	The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution	Li et al.	arXiv
2026	This human study did not involve human subjects: Validating LLM simulations as behavioral evidence	Hullman et al.	arXiv
2026	TOP 2025: An update to the Transparency and Openness Promotion Guidelines	Grant et al.	Research Integrity and Peer Review
2026	Towards a Science of AI Agent Reliability	Rabanser et al.	Proceedings of the 43rd International Conference on Machine Learning (ICML)
2026	Towards Automating Scientific Review with Google's Paper Assistant Tool	Jayaram et al.	arXiv
2026	Using prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making	Chandrashekar et al.	Royal Society Open Science
2026	VERITAS: Towards a General-Purpose Replication Tool for Scientific Research	Liu et al.	arXiv
2026	WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks	Miyai et al.	arXiv
2026	What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations	Luo et al.	arXiv
2026	What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction	Li et al.	arXiv
2026	What We Learned by Reproducing 2,200 Papers from ICML	Face
2026	What We Learned by Reproducing 2,200 papers from ICML	Abid et al.	Hugging Face Blog
2026	When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents	Mishra, Mishra	arXiv
2026	Why we no longer evaluate SWE-bench Verified	OpenAI
2026	Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks	Haase et al.	arXiv
2026	WorkplaceAgents leaderboard	AI
2025	"Seems Complicated and Unachievable": A Collaborative Autoethnography of Design Replication in Undergraduate Research	Rios et al.	Proceedings of the 7th Annual Symposium on HCI Education (EduCHI 2025)
2025	$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment	Barres et al.	arXiv
2025	A large-scale replication of scenario-based experiments in psychology and management using large language models	Cui et al.	Nature Computational Science
2025	ACEBench: Who Wins the Match Point in Tool Usage?	Chen et al.	arXiv
2025	AgentBench: Evaluating LLMs as Agents	Liu et al.	arXiv
2025	AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments	Schmidgall et al.	arXiv
2025	AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents	Andriushchenko et al.	arXiv
2025	An Illusion of Progress? Assessing the Current State of Web Agents	Xue et al.	arXiv
2025	AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents	Rawles et al.	arXiv
2025	APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay	Prabhakar et al.	arXiv
2025	ARE: Scaling Up Agent Environments and Evaluations	Froger et al.	arXiv
2025	AstaBench: Rigorous benchmarking of AI agents with a holistic scientific research suite	Ai2
2025	BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games	Paglieri et al.	arXiv
2025	BEARCUBS: A benchmark for computer-using web agents	Song et al.	arXiv
2025	Berkeley Function-Calling Leaderboard	Gorilla
2025	BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions	Zhuo et al.	arXiv
2025	BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology	Mitchener et al.	arXiv
2025	BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems	Zhang et al.	arXiv
2025	BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent	Chen et al.	arXiv
2025	BrowseComp: a benchmark for browsing agents	OpenAI
2025	BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents	Wei et al.	arXiv
2025	CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings	Quan et al.	arXiv
2025	Context is Key for Reproducibility of Empirical Studies in Human-Computer Interaction	Vanderdonckt, Vatavu	Proceedings of the 3rd ACM Conference on Reproducibility and Replicability
2025	CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments	Huang et al.	arXiv
2025	CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities	Zhu et al.	arXiv
2025	Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models	Zhang et al.	arXiv
2025	DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents	Du et al.	arXiv
2025	Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering	Si et al.	arXiv
2025	DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?	Jing et al.	arXiv
2025	Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making	Li et al.	arXiv
2025	EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents	Yang et al.	arXiv
2025	Environments Hub	Intellect
2025	Epoch AI Benchmarking Hub	AI
2025	Establishing Best Practices for Building Rigorous Agentic Benchmarks	Zhu et al.	arXiv
2025	Evaluation and Benchmarking of LLM Agents: A Survey	Mohammadi et al.	Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
2025	EXP-Bench: Can AI Conduct AI Research Experiments?	Kon et al.	arXiv
2025	Exploring the use of AI authors and reviewers at Agents4Science	Bianchi et al.	arXiv
2025	Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks	Bigeard et al.	arXiv
2025	From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking	Kim et al.	arXiv
2025	From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking	Kim et al.	arXiv
2025	GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks	Patwardhan et al.	arXiv
2025	GDPval: measuring the performance of our models on real-world tasks	OpenAI
2025	Harbor: framework for agent evals and RL environments	Institute
2025	HCAST: Human-Calibrated Autonomy Software Tasks	Rein et al.	arXiv
2025	Holistic Agent Leaderboard	SAgE
2025	Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation	Kapoor et al.	arXiv
2025	Identifying Non-Replicable Social Science Studies with Language Models	Saynova et al.	arXiv
2025	IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems	Levi, Kadar	arXiv
2025	KernelBench: Can LLMs Write Efficient GPU Kernels?	Ouyang et al.	arXiv
2025	Large language models surpass human experts in predicting neuroscience results	Luo et al.	Nature Human Behaviour
2025	Large language models that replace human participants can harmfully misportray and flatten identity groups	Wang et al.	Nature Machine Intelligence
2025	LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?	Zheng et al.	arXiv
2025	LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research	Yan et al.	Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
2025	LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research	Yan et al.	arXiv
2025	LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory	Wu et al.	arXiv
2025	macOSWorld: A Multilingual Interactive Benchmark for GUI Agents	Yang et al.	arXiv
2025	MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers	Wang et al.	arXiv

/ filter · j k or ↑ ↓ select · Enter open page · c copy BibTeX