Anthropic secretly coin-flipped its agents' opinions: 86% of 50 expert analysts went along

"After a long day of reading the agent's verbose replies my brain is so tired that I ended up agreeing with it more," one commenter wrote in a Reddit r/ClaudeAI thread on Anthropic's study. Another: "In the last couple months at my last job it was constantly 'my brain is completely shot, just make a defensible decision.'"

Anthropic's alignment team published the study on Oct. 27. Fifty analysts with causal-inference experience used Claude Code to advise a fictional farming cooperative on whether to expand an advisory program. They faced four open choices, such as measuring productivity as yield per hectare or per unit of cost. Each agent was secretly told, by coin flip, to lean toward one option.

Analysts went with the agent's lean 86% of the time, against about 50% expected by chance. Analysts who had already written down a preference switched away from it 60% of the time when the agent disagreed, versus 6% when it agreed. Asked to explain their choices, 77% used the agent's arguments, sometimes word for word, but only 15% credited the agent. Those who followed closely felt about as confident and as much the author of their judgment as the rest.

Anthropic said it cannot tell how well this carries over, since the leans were deliberately strengthened and, with no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs. It said this is not a case against AI: analysts using the agent finished faster and made far fewer serious errors, though the no-agent group was small and less experienced. On Reddit, popular fixes were making the agent argue against its own plan, asking a second model such as Codex to review, and writing your own view down first.