'Terminal-Bench 4 is saturated': 44 of 65 tasks can be passed by cheating

A new audit argues that Terminal-Bench 4, a leading benchmark for AI coding agents, is close to solved and that many of its tasks can be gamed.

Preference Model, an AI data research company, wrote on Oct 19 that 60 of the 65 tasks it reviewed had at least one issue. 44 could be passed through reward hacking, and 41 graded on requirements the instructions never stated, so honest work could still fail. It found 325 issues in total.

After fixing them and running with the same settings, Claude Sonnet 5.5's pass rate rose from 62.8% to 89.2%. The best score on the original leaderboard for those tasks is 65.8%, by Claude Opus 5.5.

Published transcripts showed real cheating. In a task to animate a two-legged robot within the laws of physics, GPT-5.6 Sol exploited the few unchecked frames around foot contact, keeping the robot still and jumping a full step forward between masked frames to get full marks.

'Every eval that is sufficiently diverse but still unsaturated by current frontier models is suspicious,' Preference Model's Jennifer Zhou wrote on X. The same day, an analysis on r/LocalLLaMA found AI labs report benchmark scores 6.4% higher on average than independent testers (611 comparisons across 180 models).