Artificial Analysis Arena Reveals Voice AI Trade-Offs

Artificial Analysis has introduced the Speech Agent Arena, a blind preference benchmark designed to evaluate native audio models across 35 real-world scenarios. These scenarios include 15 agentic tasks requiring tool calls, such as ordering pizzas under a $45 budget, and 20 non-agentic tasks focused on basic information exchange. The results have exposed a stark divergence between conversational charm and actual utility. The benchmark now replaces Conversational Dynamics in the Speech to Speech Index, holding a 25 percent weight alongside Speech Reasoning, tau-Voice agentic performance, and Task Success Rate.

The data shows that Gemini 3.1 Flash Live Preview - Minimal leads human preference with a 1046 Elo rating, yet it achieves only a 74.6 percent task success rate. Conversely, Grok Voice Think Fast 2.0 High tops task completion at 94.7 percent but ranks ninth in preference with a 908 Elo. Other tested models include GPT-Realtime-1.5, which serves as the anchor at 1000 Elo with an 85.1 percent success rate, and GPT-Realtime-2 High, which scored 914 Elo and an 89.8 percent success rate. GPT-Realtime-2.1 High reached an 892 Elo and 91.5 percent success rate, while the cascaded ElevenLabs Agents achieved a 937 Elo and 90.5 percent success rate.

Human preference closely tracks responsiveness, measured by Time to First Audio. Gemini 3.1 Flash Live Preview - Minimal responds in 0.96 seconds, GPT-Realtime-2 High takes 1.14 seconds, and the lower-ranked Qwen Audio 3.0 Realtime Plus lags at 1.54 seconds, landing at 699 Elo. Operating costs also vary widely, spanning from $1.50 per hour of input audio for the Gemini model to $4.80 for Grok and a steep $10.75 for GPT-Realtime-2.1 High.

For developers, these findings prove that automated benchmarks using synthetic users fail to capture real human experience. Because a model can sound polite while failing to execute a backend command, practitioners must treat task success as the primary metric for high-stakes applications like bookings or payments. Furthermore, the strong performance of cascaded systems like ElevenLabs shows that native audio models do not yet dominate the market by default.