'Neural networks are already clean': ex-OpenAI researcher's upbeat interpretability essay draws 10,000 likes
First seen on X 7 hours ago@nickcammarata ♥ 10,1401 views
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, says most neurons in models up to trillions of parameters can now be understood.
In a 65-minute essay, "Neural Networks Are Already Clean," posted Oct 15 on a site run by interpretability company Goodfire, he writes that OpenAI's first large automated interpretability project could read only about 2% of neurons, but modern agentic tools show a very different picture. He argues the famous evidence for superposition, where neurons pack unrelated concepts, was mostly misread.
His post on X passed 10,000 likes and 1.1 million views. OpenAI researcher roon (@tszzl) said it "may be remembered as a major moment in scientific history," while researcher Lucas Beyer complained he had read for ten minutes without reaching the point.
“interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building.”