Preprint · 2025

ARE: Scaling Up Agent Environments and Evaluations

Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, Thomas Scialom

Abstract

We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments, integration of synthetic or real applications, and execution of agentic orchestrations. ARE provides simple abstractions to build complex and diverse environments, each with their own rules, tools, content, and verifiers, helping to bridge the gap between model development and real-world deployment. We also propose Gaia2, a benchmark built in ARE and designed to measure general agent capabilities. Beyond search and execution, Gaia2 requires agents to handle ambiguities and noise, adapt to dynamic environments, collaborate with other agents, and operate under temporal constraints. Unlike prior benchmarks, Gaia2 runs asynchronously, surfacing new failure modes that are invisible in static settings. Our experiments show that no system dominates across the intelligence spectrum: stronger reasoning often comes at the cost of efficiency, and budget scaling curves plateau, highlighting the need for new architectures and adaptive compute strategies. Perhaps more importantly, ARE abstractions enable continuous extension of Gaia2 to other environments, empowering the community to rapidly create new benchmarks tailored to their domains. In AI's second half, progress increasingly depends on defining meaningful tasks and robust evaluations to drive frontier capabilities forward.

BibTeX Copy

@misc{froger_are_2025,
	title = {{ARE}: {Scaling} {Up} {Agent} {Environments} and {Evaluations}},
	shorttitle = {{ARE}},
	url = {http://arxiv.org/abs/2509.17158},
	doi = {10.48550/arXiv.2509.17158},
	abstract = {We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments, integration of synthetic or real applications, and execution of agentic orchestrations. ARE provides simple abstractions to build complex and diverse environments, each with their own rules, tools, content, and verifiers, helping to bridge the gap between model development and real-world deployment. We also propose Gaia2, a benchmark built in ARE and designed to measure general agent capabilities. Beyond search and execution, Gaia2 requires agents to handle ambiguities and noise, adapt to dynamic environments, collaborate with other agents, and operate under temporal constraints. Unlike prior benchmarks, Gaia2 runs asynchronously, surfacing new failure modes that are invisible in static settings. Our experiments show that no system dominates across the intelligence spectrum: stronger reasoning often comes at the cost of efficiency, and budget scaling curves plateau, highlighting the need for new architectures and adaptive compute strategies. Perhaps more importantly, ARE abstractions enable continuous extension of Gaia2 to other environments, empowering the community to rapidly create new benchmarks tailored to their domains. In AI's second half, progress increasingly depends on defining meaningful tasks and robust evaluations to drive frontier capabilities forward.},
	urldate = {2026-11-07},
	publisher = {arXiv},
	author = {Froger, Romain and Andrews, Pierre and Bettini, Matteo and Budhiraja, Amar and Cabral, Ricardo Silveira and Do, Virginie and Garreau, Emilien and Gaya, Jean-Baptiste and Laurençon, Hugo and Lecanu, Maxime and Malkan, Kunal and Mekala, Dheeraj and Ménard, Pierre and Bertran, Gerard Moreno-Torres and Piterbarg, Ulyana and Plekhanov, Mikhail and Rita, Mathieu and Rusakov, Andrey and Vorotilov, Vladislav and Wang, Mengjue and Yu, Ian and Benhalloum, Amine and Mialon, Grégoire and Scialom, Thomas},
	month = dec,
	year = {2025},
	note = {arXiv:2509.17158 [cs.AI]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
}