02 Software engineering & coding

Shared by Changkun Ou with everyone

Year ↓	Title added	Authors	Venue
2026	Agentic Coding 2.0 leaderboard	AI
2026	DeepSWE v1.1	Datacurve
2026	DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle	Tang et al.	arXiv
2026	FrontierCode	Cognition
2026	LibraryDesignBench leaderboard	AI
2026	Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading	Li et al.	arXiv
2026	MirrorCode	AI
2026	NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents	Ding et al.	arXiv
2026	Senior SWE-bench leaderboard	AI
2026	SlopCode Bench leaderboard	AI
2026	SWE-bench CLI leaderboard	AI
2026	SWE-bench leaderboards	team
2026	SWE-Bench Pro V2	AI
2026	Terminal-Bench (site and leaderboard)	(Stanford
2026	Terminal-Bench 3.0	team
2026	Terminal-Bench 4.0 leaderboard	Institute
2026	Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces	Merrill et al.	arXiv
2026	Why we no longer evaluate SWE-bench Verified	OpenAI
2025	BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions	Zhuo et al.	arXiv
2025	CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings	Quan et al.	arXiv
2025	Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering	Si et al.	arXiv
2025	KernelBench: Can LLMs Write Efficient GPU Kernels?	Ouyang et al.	arXiv
2025	LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?	Zheng et al.	arXiv
2025	Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving	Zan et al.	arXiv
2025	SWE-bench Goes Live!	Zhang et al.	arXiv
2025	SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?	Deng et al.	arXiv
2025	SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?	Miserendino et al.	arXiv
2025	SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents	Rashid et al.	arXiv
2025	SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents	Badertdinov et al.	arXiv
2025	SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents	Mündler et al.	arXiv
2025	The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason	Liang et al.	arXiv
2024	Aider LLM Leaderboards (Polyglot)	Aider
2024	Commit0: Library Generation from Scratch	Zhao et al.	arXiv
2024	Introducing SWE-bench Verified	OpenAI
2024	LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code	Jain et al.	arXiv
2024	Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study	Li et al.	arXiv
2024	Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet	Anthropic
2024	SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?	Yang et al.	arXiv
2024	SWE-Bench+: Enhanced Coding Benchmark for LLMs	Aleithan et al.	arXiv
2024	SWE-bench: Can Language Models Resolve Real-World GitHub Issues?	Jimenez et al.	arXiv
2023	CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion	Ding et al.	arXiv
2023	InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback	Yang et al.	arXiv
2023	RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems	Liu et al.	arXiv
2022	Competition-Level Code Generation with AlphaCode	Li et al.	Science
2022	DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation	Lai et al.	arXiv
2021	Evaluating Large Language Models Trained on Code	Chen et al.	arXiv
2021	Measuring Coding Challenge Competence With APPS	Hendrycks et al.	arXiv
2021	Program Synthesis with Large Language Models	Austin et al.	arXiv

/ filter · j k or ↑ ↓ select · Enter open page · c copy BibTeX