Latest update
'Even the world's best AI fails nine of ten tasks': SemiAnalysis's new GDPx benchmark has Opus 5.5 on top at just 12%
First seen on X 8 hours ago@dylan522p ♥ 1,1888 views
SemiAnalysis has released GDPx, a benchmark of 206 real-work tasks for AI agents, and even the winner, Anthropic's Opus 5.5, scored only 12.0%.
According to SemiAnalysis's newsletter dated Nov. 2, GDPx covers 11 domains including chip design, cybersecurity, biology, finance, law, quant trading and software engineering. Tasks built by 14 specialist data partners run up to eight hours each, using each lab's native harness such as Claude Code or Codex where possible. Scores are pass@1 over three attempts. OfficeChai reported Google's Gemini 4 Carbon second at 10.9% and OpenAI's GPT-6 Astra third at 9.6%. Cost per task was $51.41 for Opus and $53.99 for Astra, with Carbon at about a third of Opus.
Opus led quant trading and chip and hardware design, Carbon led biology, and Astra led materials science. In law, no model produced a usable redline: Opus averaged 44% partial credit, Astra 25% and Gemini 4 Argon 2%. In finance, none of 25 attempts at the public task succeeded. "The low scores are the point," SemiAnalysis said, per OfficeChai.
Dylan Patel, a co-author, posted on X: "Official apology to Google from me, they cooked with Carbon," calling it spiky across domains but "really good in a few areas." The post drew about 1,100 likes, and Google DeepMind CEO Demis Hassabis replied that new rigorous benchmarks are always welcome.
There are caveats. Samples are small, with three attempts per task and 20 to 25 rollouts in some domains, and safety classifiers dragged down Opus and Astra on cyber, biology and alignment. Without those three domains, Astra edges Opus by 0.1 point and Carbon falls to fifth. Google unveiled Carbon in late October; it is not yet generally available.
“Official apology to Google from me, they cooked with Carbon. It's spikier in terms of domains and capabilities, but it's really good in a few areas.”
