Preprint · 2024

$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan

Abstract

Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $τ$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.

BibTeX Copy

@misc{yao_-bench_2024,
	title = {\$τ\$-bench: {A} {Benchmark} for {Tool}-{Agent}-{User} {Interaction} in {Real}-{World} {Domains}},
	shorttitle = {\$τ\$-bench},
	url = {http://arxiv.org/abs/2406.12045},
	doi = {10.48550/arXiv.2406.12045},
	abstract = {Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose \$τ\$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass{\textasciicircum}k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on {\textless}50\% of the tasks, and are quite inconsistent (pass{\textasciicircum}8 {\textless}25\% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.},
	urldate = {2026-11-06},
	publisher = {arXiv},
	author = {Yao, Shunyu and Shinn, Noah and Razavi, Pedram and Narasimhan, Karthik},
	month = jun,
	year = {2024},
	note = {arXiv:2406.12045 [cs.AI]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
}