Preprint · 2026
Agents' Last Exam
| Venue | arXiv |
|---|---|
| DOI | 10.48550/arXiv.2606.05405 |
| URL | http://arxiv.org/abs/2606.05405 |
| Topics | ReplicHAI / Agent Evals / 05 Computer use & GUI |
| Tags | Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning |
| archiveID | arXiv:2606.05405 |
| extra | arXiv:2606.05405 [cs.AI] |
| libraryCatalog | arXiv.org |
Abstract
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.
BibTeX
@misc{sun_agents_2026,
title = {Agents' {Last} {Exam}},
url = {http://arxiv.org/abs/2606.05405},
doi = {10.48550/arXiv.2606.05405},
abstract = {Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1\%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.},
urldate = {2026-11-06},
publisher = {arXiv},
author = {Sun, Yiyou and Han, Xinyang and Zhang, Weichen and Pang, Yuanbo and Wang, Tianyu and Cao, Yuhan and Huang, Yixiao and Liu, Xiang and Di, Zonglin and Duroiu, Chris and Zhang, Haoyun and Lin, Jeffrey and Zhang, Weishu and Zeng, Tyler and Yan, Ying and Liu, Bo and Wen, Hanson and Xu, Mingyang and Liu, Xiaoyuan and Chen, Zimeng and Zhang, Zhaowei and Shi, Weiyan and Dsouza, Amanda and Chen, Vincent Sunn and Xie, Jingxu and Lei, Dewen and Zhang, Hanxuan and Wang, Yinhe and Zhang, Xinran and Zhou, Chao and Huang, Sheldon and Bryant, Patrick and Boettiger, Carl and Rangan, Yamini and Rothenberg, Bradley and Steinfeld, Kyle and Rao, Arvind and Schneider, Tapio and Yannakakis, Georgios and Zanna, Laure and Ozbay, Kaan and Sim, Ida and Zohdi, Tarek and Karniadakis, George Em and Gallant, Jack and Head-Gordon, Teresa and Liu, Chris Yuhao and Hu, Rongwang and Cheng, Yafei and Bacho, Aras and Qin, Zengyi and Yuille, Alan and Chen, Yixiong and Liu, Hao and Zeng, Lin and Chen, Bo and Li, Zhendong and Gong, Litian and Bharadwaj, Shashank Muralidhar and Jansen, Thorben and Tang, Haixiang and Tu, Jianhong and Liu, Jixian and Song, Maojia and Wang, Ruheng and Cao, Shengcao and Zhang, Yanxin and Yang, Yingxuan and Zhang, Zongzheng and Bao, Honglin and Kim, Hyunjun and Xu, Peiran and Lu, Shuo and Zhang, Zheng and Wang, Zhonghua and Prakash, Aditya and Lyu, Bohan and Ma, Chang and Chen, Haoxuan and Fan, Hengduan and Tao, Molei and Sitelew, Roman and Poldrack, Russell Alan and Liu, Yang and Liu, Yang and Wu, Yichen and Lin, Yuanze and Li, Zhenglu and Wei, Zibu and Chen, Zijiao and Bendahi, Abderrahim and Schubert, Alexander and Broderick, Chase and Liu, Chengzhi and Sun, Haoran and Li, Haoyang and Hua, Haoyue and Ding, Jiayuan and Huang, Jing and Gu, Shangding and Vinh, Tuan and Sun, Xiatao and Jiang, Yanqiong and Hu, Yiran and Xu, Zechen and Yang, Ziao and Wang, Chenguang and Spanos, Costas and Yang, Dongrong and Wang, Haocheng and Cheng, Jiayi and Wang, Jicheng and Peng, Junbo and Zhang, Kangqi and Kuang, Keying and Sun, Lichao and Yin, Ming and Xu, Orson and Siu, Vincent and Liao, Wanzhe and Shi, Wenqi and Deng, Wenxi and Zhang, Xiaopan and Zhu, Yangjunfeng and Shen, Yifan and Jiang, Yi and Li, Yinsheng and Pan, Youbang and Li, Yuan and Zhu, Yuchen and Zhang, Yunshi and Yang, Zhenning and Rao, Ziyan and Rafiei, Alireza and Glover, Andrew and Ren, Bixuan and Yuan, Boqin and Cai, Chonghao and Lei, Deren and Xing, Hanwen and Jin, Haoran and Gomez, Hector and Chandok, Ishaan Singh and Hu, Jiaming and Sun, Jianxin and Purnomo, Justin and Sun, Kunyang and Ding, Lei and Liao, Longtai and Kim, Lukas and Liu, Ming and Wuwu, Qingpo and Jin, Ran and Zhang, Ruiqi and Sun, Ting and Huang, Wenbo and Zhao, Xuandong and Chen, Peter and Jin, Chuanyang and Bao, Qiming and Shen, Xuhuan and Zhang, Yifan and Li, Yi and Wang, Yingze and Wang, Yixin and Yang, Yushi and Dong, Zhen and Yan, Zhiling and Xu, Zhongxing and Zhao, Zibo and Jiang, Zixin and Emami, Ali and Liu, Andy Zeyi and Wu, Angela and Caliskan, Aylin and Huang, Baihe and Huang, Baixiang and Yang, Benjamin and Zhang, Benran and Zhao, Brian and Bosio, Carlo and Guo, Charlie and Liu, Che and Du, Chenchen and Dai, Chen and Zhang, Chenlong and Li, Chenyu and Zhu, Chenyu and Zhang, Chi and Park, Core Francisco and Li, Daofeng and Wu, Dehao and Wong, Dowling and Kasson, Erin and Wu, Fangyu and Xu, Fan and Shen, Fiona and Hu, Haimin and Li, Haitao and Zhou, Hang and Szlyk, Hannah and Zhang, Hanning and Zhang, Hanwen and Fan, Hao and Ye, Haoran and Shang, HaoYang and Bai, Haoyue and Fu, Hengyu and Li, Hongwei and Wang, Huiqi and Li, Jiachen and Wang, Jialu and Liu, Jiaming and Shen, Jiawei and Zuo, Jierui and Fan, Jingxuan and Li, Jingyan and Yang, Jirong and So, Jonah and Overwiening, Jorin and Cavanagh, Joseph and Lee, Joseph and Zhang, Junge and Liu, Kaiyuan and Dhole, Kaustubh and Lin, Kevin Qinghong and Zhang, Kunpeng and Montgomery, Kyle and Paraboschi, Layna and Liana and Liu, Lihui and Zhou, Liwei and Lu, Lu and Wu, Mengyuan "Millie" and Tolonen, Mikko and Yan, Mingxuan and Shi, Minqi and Niu, Murphy and Lu, Pan and Cavazos-Rehg, Patricia and Zhao, Qijian and Mang, Qiuyang and Jaiswal, Rimika and Xu, Ruiling and Fan, Run-Ze and Ning, Ruoxi and Antu, Santanu Bosu and Rankin, Scott and Yang, Sen and Gu, Shi and Qiu, Shi and Zhu, Sichen and Xu, Sihan and Zhu, Siqi and Zhang, Song and Sun, Sophia and Castro, Stefano De and Rajwal, Swati and Shi, Tianneng and Chen, Tianxing and Zhou, Tianyue and Wang, Ting and Shaw, Ventus and Susoy, Vladislav and Xiao, Weihang and Wang, Wei and Lv, Wenbo and Zhang, Wenjin and Zhang, Wenshuo and Liang, Wenyu and Yu, Xiaohu and Li, Xiao and Yang, Xiaoli and Lan, Xin and Wang, Xinyu and Liu, Xuan and Chen, Xuewen and Pan, Xu and Zhu, Yan and Lyu, Yicheng and Gu, Yifan and Zhao, Yimin and Zhang, Ying and Ren, Yin and Chen, Yipu and Rong, Yiren and Shao, Yi and Zhu, Yishu and Zhang, Yizhi and Xiong, Yongyi and Li, Yuangang and Wu, Yuchen and Zheng, Yuhaohua and Wu, Yuhao and Deng, Yunfu and Li, Yushan and Wu, Yu and Xu, Yuzheng and Nie, Yuzhou and Wang, Zehao and Du, Zhehang and Yu, Zhengming and Jiang, Zhihan and Li, Zhi and Li, Zhixu and Yi, Zhuoran and Sun, Tao and Wang, Zhun and Xu, Justin and Bo, Shi and Casademunt, Helena and Chen, Wenjia and Cheng, Cheng and Deng, Nawen and Fu, Rao and Fu, Tianfu and Han, Yifan and He, Zhenyu and Jin, Qiao and Li, Langlang and Li, Yuetai and Liu, Sylvia and Mukherjee, Subhabrata and Ouyang, Yunqi and Ren, He and Shi, Dawei and Wu, Haoran and Wu, Zhiyue and Yao, Hannah and Yu, Jenny and Zhan, Rhea and Zhou, Luqing and Zhu, Blake and Zhu, Junfan and Das, Sanjiv and Gao, Yanjun and Guo, Wenbo and Schneider, Erika J. and Liu, Bing and Lu, Zhiyong and Ma, Yian and Mueller, Mark and Poovendran, Radha and Sojoudi, Somayeh and Yao, Huaxiu and Zhu, Yinglun and Zsolnai-Fehér, Károly and Denisov-Blanch, Yegor and Hu, Changran and Koc, Vincent and Yin, Hezheng and Xia, Naomi and Bai, Xinyi and Dilbagi, Arsh Shah and Gu, Jing and Liu, Yutong and Shi, Lin and Zhang, Qizheng and Hao, Yuexing and Li, Xiaoming and Li, Xiangyi and You, Bingran and Ke, Wenjun and Zhang, Daoan and Nemet, Ari and Zhang, Tianyi and Lyu, Xingda and Chawla, Pranit and Winata, Genta Indra and Tang, Maosen and Chiniya, Purva and Chen, Qiuyi and Dillmann, Steven and Xu, Wanghan and Yang, Wenqi and Wu, Xiaojun and Luo, Xin and Huang, Yue and Song, Dawn},
month = oct,
year = {2026},
note = {arXiv:2606.05405 [cs.AI]},
keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning},
}