Changkun Ou/Library

ReplicHAI › Agent Evals

Shared by Changkun Ou with everyone

256 entries Export .bib
Year ↓ Title added Authors Venue
Webpage 2026 Agentic Coding 2.0 leaderboard AI
Preprint 2026 Agents Are Systems, Not Models: Rethinking Agentic Evaluation Wiedmann et al. arXiv
Preprint 2026 Agents' Last Exam Sun et al. arXiv
Webpage 2026 Agents' Last Exam leaderboard team
Preprint 2026 AI Coding Agents Can Reproduce Social Science Findings Alizadeh et al. arXiv
Preprint 2026 AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery Liu et al. arXiv
Preprint 2026 AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications Zhao et al. arXiv
Blog post 2026 ARC-AGI-3 Foundation
Webpage 2026 Artificial Analysis Intelligence Index v5 Analysis
Preprint 2026 AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite Bragg et al. arXiv
Preprint 2026 AutomationBench Shepard, Salimans arXiv
Webpage 2026 AutomationBench leaderboard Zapier
Conference paper 2026 AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage Zhao et al. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Preprint 2026 BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business Zhao et al. arXiv
Preprint 2026 Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents Ma et al. arXiv
Preprint 2026 Can Coding Agents Reproduce Findings in Computational Materials Science? Huang et al. arXiv
Preprint 2026 Coding-agents can replicate scientific machine learning papers Hans, Bilionis arXiv
Blog post 2026 Company Knowledge Bench Kapa.ai
Preprint 2026 CompMat-Bench: Benchmarking AI Agents for Computational Materials Science Zhang et al. arXiv
Webpage 2026 Continual Learning Bench (site and leaderboard) Berkeley
Preprint 2026 Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments Asawa et al. arXiv
Preprint 2026 CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark Siegel et al. arXiv
Preprint 2026 CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale Wang et al. arXiv
Preprint 2026 DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints Zhang et al. arXiv
Blog post 2026 DeepSWE v1.1 Datacurve
Blog post 2026 Demystifying evals for AI agents Anthropic
Preprint 2026 DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle Tang et al. arXiv
Webpage 2026 Digital Work Index AI
Preprint 2026 EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents Huang et al. arXiv
Preprint 2026 EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights Geng et al. arXiv
Preprint 2026 Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions Hu et al. arXiv
Preprint 2026 FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights Wang et al. arXiv
Blog post 2026 FrontierCode Cognition
Preprint 2026 Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking Messing arXiv
Journal article 2026 Humanity's Last Exam Phan et al. Nature
Preprint 2026 Into the Unknown: Benchmarking AI agents on open scientific research problems Kim et al. arXiv
Preprint 2026 KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards Li et al. arXiv
Webpage 2026 LibraryDesignBench leaderboard AI
Preprint 2026 Life After Benchmark Saturation: A Case Study of CORE-Bench Nadgir et al. arXiv
Preprint 2026 Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading Li et al. arXiv
Webpage 2026 MBABench 2.0 leaderboard AI
Preprint 2026 MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers Bandi et al. arXiv
Preprint 2026 Measuring AI Ability to Complete Long Software Tasks Kwa et al. arXiv
Preprint 2026 Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents Shen et al. arXiv
Webpage 2026 MirrorCode AI
Preprint 2026 MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility Gaddipati et al. arXiv
Preprint 2026 NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents Ding et al. arXiv
Preprint 2026 OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks Yuan et al. arXiv
Preprint 2026 PAPERBENCHX: Can AI Reproduce Science Across Disciplines? Wu et al. arXiv
Preprint 2026 PRBench: End-to-end Paper Reproduction in Physics Research Qiu et al. arXiv
Preprint 2026 Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results Kohler et al. arXiv
Preprint 2026 RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers? Salunkhe et al. arXiv
Conference paper 2026 ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences Nguyen et al. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Conference paper 2026 Revibing Code from Papers: Reimplementing HCI Artifacts Adar et al. Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology
Conference paper 2026 RExBench: Can coding agents autonomously implement AI research extensions? Edwards et al. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Preprint 2026 SearchHorizonBench: A Benchmark for Search Value in AI Agents Ning et al. arXiv
Webpage 2026 Senior SWE-bench leaderboard AI
Preprint 2026 SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks Zhong et al. arXiv
Webpage 2026 SlopCode Bench leaderboard AI
Webpage 2026 SnorkelFinance 2.0 leaderboard AI
Webpage 2026 SnorkelLegal leaderboard AI
Webpage 2026 SnorkelManufacturing leaderboard AI
Webpage 2026 SnorkelRevOps leaderboard AI
Webpage 2026 SnorkelUnderwrite 2.0 leaderboard AI
Preprint 2026 Survey on Evaluation of LLM-based Agents Yehudai et al. arXiv
Webpage 2026 SWE-bench CLI leaderboard AI
Webpage 2026 SWE-bench leaderboards team
Blog post 2026 SWE-Bench Pro V2 AI
Webpage 2026 Terminal-Bench (site and leaderboard) (Stanford
Blog post 2026 Terminal-Bench 3.0 team
Webpage 2026 Terminal-Bench 4.0 leaderboard Institute
Webpage 2026 Terminal-Bench-Science 0.1 team
Preprint 2026 Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces Merrill et al. arXiv
Blog post 2026 The AI Replication Engine: First Experiments and What's Next (I4R)
Preprint 2026 The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark Spence et al. arXiv
Preprint 2026 The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution Li et al. arXiv
Preprint 2026 WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks Miyai et al. arXiv
Blog post 2026 What We Learned by Reproducing 2,200 Papers from ICML Face
Preprint 2026 When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents Mishra, Mishra arXiv
Blog post 2026 Why we no longer evaluate SWE-bench Verified OpenAI
Webpage 2026 WorkplaceAgents leaderboard AI
Preprint 2025 $τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment Barres et al. arXiv
Preprint 2025 ACEBench: Who Wins the Match Point in Tool Usage? Chen et al. arXiv
Preprint 2025 AgentBench: Evaluating LLMs as Agents Liu et al. arXiv
Preprint 2025 AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments Schmidgall et al. arXiv
Preprint 2025 AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents Andriushchenko et al. arXiv
Preprint 2025 An Illusion of Progress? Assessing the Current State of Web Agents Xue et al. arXiv
Preprint 2025 AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Rawles et al. arXiv
Preprint 2025 APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay Prabhakar et al. arXiv
Preprint 2025 ARE: Scaling Up Agent Environments and Evaluations Froger et al. arXiv
Blog post 2025 AstaBench: Rigorous benchmarking of AI agents with a holistic scientific research suite Ai2
Preprint 2025 BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games Paglieri et al. arXiv
Preprint 2025 BEARCUBS: A benchmark for computer-using web agents Song et al. arXiv
Webpage 2025 Berkeley Function-Calling Leaderboard Gorilla
Preprint 2025 BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions Zhuo et al. arXiv
Preprint 2025 BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology Mitchener et al. arXiv
Preprint 2025 BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems Zhang et al. arXiv
Preprint 2025 BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent Chen et al. arXiv
Blog post 2025 BrowseComp: a benchmark for browsing agents OpenAI
Preprint 2025 BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents Wei et al. arXiv
Preprint 2025 CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings Quan et al. arXiv
Preprint 2025 CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments Huang et al. arXiv
Preprint 2025 CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities Zhu et al. arXiv
Preprint 2025 Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models Zhang et al. arXiv
Preprint 2025 DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents Du et al. arXiv
Preprint 2025 Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering Si et al. arXiv
Preprint 2025 DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? Jing et al. arXiv
Preprint 2025 Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making Li et al. arXiv
Preprint 2025 EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents Yang et al. arXiv
Blog post 2025 Environments Hub Intellect
Webpage 2025 Epoch AI Benchmarking Hub AI
Preprint 2025 Establishing Best Practices for Building Rigorous Agentic Benchmarks Zhu et al. arXiv
Conference paper 2025 Evaluation and Benchmarking of LLM Agents: A Survey Mohammadi et al. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Preprint 2025 EXP-Bench: Can AI Conduct AI Research Experiments? Kon et al. arXiv
Preprint 2025 Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks Bigeard et al. arXiv
Preprint 2025 From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking Kim et al. arXiv
Preprint 2025 GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks Patwardhan et al. arXiv
Blog post 2025 GDPval: measuring the performance of our models on real-world tasks OpenAI
Computer program 2025 Harbor: framework for agent evals and RL environments Institute
Preprint 2025 HCAST: Human-Calibrated Autonomy Software Tasks Rein et al. arXiv
Webpage 2025 Holistic Agent Leaderboard SAgE
Preprint 2025 Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation Kapoor et al. arXiv
Preprint 2025 IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems Levi, Kadar arXiv
Preprint 2025 KernelBench: Can LLMs Write Efficient GPU Kernels? Ouyang et al. arXiv
Preprint 2025 LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? Zheng et al. arXiv
Preprint 2025 LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research Yan et al. arXiv
Preprint 2025 LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory Wu et al. arXiv
Preprint 2025 macOSWorld: A Multilingual Interactive Benchmark for GUI Agents Yang et al. arXiv
Preprint 2025 MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers Wang et al. arXiv
Preprint 2025 MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers Luo et al. arXiv
Preprint 2025 MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use Wu et al. arXiv
Blog post 2025 Measuring AI Ability to Complete Long Tasks METR
Preprint 2025 MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents Jiang et al. arXiv
Preprint 2025 MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents Tan et al. arXiv
Preprint 2025 Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge Gou et al. arXiv
Preprint 2025 MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering Chan et al. arXiv
Preprint 2025 MLGym: A New Framework and Benchmark for Advancing AI Research Agents Nathani et al. arXiv
Preprint 2025 MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? Zhang et al. arXiv
Preprint 2025 Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving Zan et al. arXiv
Preprint 2025 MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents Zhu et al. arXiv
Preprint 2025 NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security Shao et al. arXiv
Preprint 2025 OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents Kuntz et al. arXiv
Webpage 2025 OSWorld site and OSWorld-Verified leaderboard Lab
Preprint 2025 PaperBench: Evaluating AI's Ability to Replicate AI Research Starace et al. arXiv
Preprint 2025 R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents Jain et al. arXiv
Preprint 2025 RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts Wijk et al. arXiv
Preprint 2025 REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards Stojanovski et al. arXiv
Webpage 2025 Remote Labor Index CAIS
Preprint 2025 Remote Labor Index: Measuring AI Automation of Remote Work Mazeika et al. arXiv
Preprint 2025 ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? Ye et al. arXiv
Conference paper 2025 REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research? Hu et al. Findings of the Association for Computational Linguistics: ACL 2025
Preprint 2025 ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code Hua et al. arXiv
Preprint 2025 ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery Chen et al. arXiv
Preprint 2025 SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers Xiang et al. arXiv
Preprint 2025 ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use Li et al. arXiv
Preprint 2025 Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows Lei et al. arXiv
Preprint 2025 StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models Guo et al. arXiv
Preprint 2025 SWE-bench Goes Live! Zhang et al. arXiv
Preprint 2025 SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? Deng et al. arXiv
Preprint 2025 SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? Miserendino et al. arXiv
Preprint 2025 SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents Rashid et al. arXiv
Preprint 2025 SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents Badertdinov et al. arXiv
Preprint 2025 SWE-smith: Scaling Data for Software Engineering Agents Yang et al. arXiv
Preprint 2025 SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents Mündler et al. arXiv
Conference paper 2025 The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models Patil et al. Proceedings of the 42nd International Conference on Machine Learning
Preprint 2025 The BrowserGym Ecosystem for Web Agent Research Chezelles et al. arXiv
Preprint 2025 The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason Liang et al. arXiv
Preprint 2025 TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks Xu et al. arXiv
Preprint 2025 ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities Lu et al. arXiv
Preprint 2025 Training Software Engineering Agents and Verifiers with SWE-Gym Pan et al. arXiv
Webpage 2025 Vending-Bench 2 Labs
Preprint 2025 Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents Backlund, Petersson arXiv
Preprint 2025 Why Do Multi-Agent LLM Systems Fail? Cemri et al. arXiv
Preprint 2025 WideSearch: Benchmarking Agentic Broad Info-Seeking Wong et al. arXiv
Preprint 2025 WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks Boisvert et al. arXiv
Preprint 2024 $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains Yao et al. arXiv
Preprint 2024 Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations Miller arXiv
Preprint 2024 AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents Ma et al. arXiv
Preprint 2024 AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents Debenedetti et al. arXiv
Preprint 2024 AgentGym: Evolving Large Language Model-based Agents across Diverse Environments Xi et al. arXiv
Preprint 2024 AI Agents That Matter Kapoor et al. arXiv
Webpage 2024 Aider LLM Leaderboards (Polyglot) Aider
Preprint 2024 AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents Xu et al. arXiv
Preprint 2024 AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents Trivedi et al. arXiv
Preprint 2024 AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? Yoran et al. arXiv
Preprint 2024 BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation Li et al. arXiv
Preprint 2024 Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Chiang et al. arXiv
Preprint 2024 Commit0: Library Generation from Scratch Zhao et al. arXiv
Preprint 2024 DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models Huang et al. arXiv
Preprint 2024 DiscoveryBench: Towards Data-Driven Discovery with Large Language Models Majumder et al. arXiv
Preprint 2024 DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents Jansen et al. arXiv
Preprint 2024 Evaluating Very Long-Term Conversational Memory of LLM Agents Maharana et al. arXiv
Preprint 2024 Identifying the Risks of LM Agents with an LM-Emulated Sandbox Ruan et al. arXiv
Preprint 2024 InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks Hu et al. arXiv
Preprint 2024 InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents Zhan et al. arXiv
Computer program 2024 Inspect Institute
Computer program 2024 Inspect Evals contributors
Blog post 2024 Introducing SWE-bench Verified OpenAI
Preprint 2024 LAB-Bench: Measuring Capabilities of Language Models for Biology Research Laurent et al. arXiv
Preprint 2024 LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code Jain et al. arXiv

1–200 of 256 Next ›

/ filter · j k or ↑ ↓ select · Enter open page · c copy BibTeX