Vals AI finds agent teams waste tokens for minimal gains
New evaluations from Vals AI show that deploying multi-agent AI teams multiplies token expenditure without delivering meaningful quality improvements over single models.
Evaluation firm Vals AI recently put frontier models through their paces on the Vibe Code Bench to compare individual systems against collaborative agent teams. Testing GPT-6 Sol and Claude Opus 5.5 across medium and maximum reasoning effort settings, the researchers discovered that multi-agent architectures cost between 1.8x and 5.1x more than single-agent setups. Out of four comparisons between teams and solo agents, only one showed a statistically significant improvement: GPT-6 Sol running at medium reasoning, where team coordination added 7.3 points. At maximum reasoning capacity, multi-agent setups provided no real edge for either Sol or Opus 5.5 over standalone implementations.
These findings align with observations across the industry. In separate evaluations using ProgramBench and internal tests with Opus 5.5, Anthropic observed diminishing performance returns when scaling up agent counts. While larger teams reached specific performance targets faster, scaling an agent pool from 10 to 100 only slightly nudged scores higher after 24 hours while consuming far more tokens. Similarly, model Fable 5.1 achieved stronger quality gains on a Lean theorem proving task when expanding past 10 agents, yet still scored below Opus 5.5 overall. On a separate knowledge base task, Fable 5.1 saw its score dip slightly when scaling from 30 to 100 agents.
Speaking on the Dwarkesh Podcast, OpenAI researcher Noam Brown clarified that multi-agent configurations primarily buy execution speed rather than superior reasoning quality. Deploying four agents allowed systems to complete tasks twice as fast at twice the cost, but scaling to 16 agents introduced further inefficiencies. Brown explained that while web research and math parallelize effectively, creative tasks like novel writing do not, noting that throwing 10,000 agents at a draft is as pointless as throwing 10,000 people at it. OpenAI developer Eric Provencher also cautioned against agent swarms due to what he termed a "coordination tax" that wastes money. For software practitioners, these results demonstrate that expanding agent teams is rarely a cost-effective strategy for improving task accuracy.
This is our own summary of reporting by The Decoder
