Special Β· OpenAI's pretraining scorecard
GPT-3.5 at 0.99, an internal model at 0.73: why OpenAI wants the industry to stop before 0.7
"Internal model ποΈποΈ" That reply to OpenAI's first-ever public pretraining chart drew more than 600 likes on Nov. 22, because an unnamed dot sat at the end of the curve. Chief scientist Jakub Pachocki published the numbers knowing it shows competitors where OpenAI stands. Here is what they mean and how the field reacted.

Perplexity is a next-word prediction score
In "Perplexity and Intelligence," Pachocki defines perplexity as how well a model predicts the next token on unseen text; lower is better, and OpenAI tracks its log (cross-entropy loss). He argues modern AI is less hand-designed software than intelligence harnessed from one simple process, training a compressor on huge data, so this one number best tracks progress.
0.99 gave ChatGPT, 0.91 gave reasoning, below 0.8 came cyber and math
On OpenAI's main eval set, GPT-3.5 sat at about 0.99 and powered the original ChatGPT. GPT-4, at about 0.91, could use chain-of-thought productively, which led to today's reasoning models. Below 0.8, transformative capabilities appeared in cybersecurity and mathematics. "It was not at all clear a priori" that they would emerge at those thresholds, he wrote.
At 0.76, open math problems fell one after another
OpenAI has used open math problems as evals for over a year. After upgrading to a model of about 0.76, Pachocki wrote, answers to long-pondered questions came out "one after another in a single Slack thread," which felt "deeply surreal." He warned a similar avalanche in pretraining research could come as soon as 2027, faster than alignment and monitoring can keep up.
The 'internal model' sits near 0.73
The chart ends with an unnamed internal model at about 0.73, a notch below GPT-6.2 Astra (about 0.76), the newest public model. "An unreleased internal model already sits below GPT-6.2 Astra. OpenAI says AI itself already helped design the newest models," summarized AI account kimmonismus, drawing over 600 likes. A footnote says the curve steepened in 2026.
The ask: stop together before 0.68-0.7
Pachocki expects further transformative capabilities in the ~0.68-0.7 window. He proposes common limits, based on perplexity and other measures, that labs commit not to cross until monitoring reaches a concrete milestone, such as a successor to chain-of-thought monitors, plus shared evaluation, red-teaming and transparency norms. Sam Altman called it "the most important thing OpenAI has published in awhile."
Four limits he names himself
Labs use different eval and training data, so numbers are hard to compare. Perplexity doesn't map directly to tasks people care about. Chain-of-thought reasoning boosts capability without lowering perplexity. And no one knows the floor, the data's inherent entropy; near it, small perplexity gains may mean big jumps in intelligence.
"Why 0.68?" researchers ask
Analyst scaling01 asked whether 0.68-0.7 is arbitrary or what OpenAI expects of its next base model, drawing about 270 likes. SemiAnalysis's Dylan Patel offered an "alternative explanation," that OpenAI was back on track and wanted to gloat, then added "jk pacing is a legit thing," since efficient pretraining makes models easier to misalign in fewer RL steps. Hugging Face researcher Elie Bakouch said base log perplexity strongly predicts both Epoch's and OpenAI's capability indexes.
Musk: just call it training
"Pretraining should just be called 'training,'" Elon Musk wrote, likening it to dressing up to work out and showering without exercising, and predicting data starvation as the next challenge; the post passed 4,500 likes. Pacing talk isn't OpenAI's alone: Crypto Briefing notes Anthropic's Dario Amodei published a plan in September to "pace the frontier." Altman said he hopes the proposal helps the safety standards labs agreed to build under the White House pact.
Comments 0
Select a sentence in the story (long-press on phones) to comment on it.
Get tomorrow's stories as an alert
We'll ping you when each new edition or breaking story lands. Free, no sign-up.
More stories from the same day
- Amodei's old fix for AI unemployment: immersive video games, per WSJ. The post drew 13,000 likes
- Skip a five-hour AI course and you can't "meet expectations": Remote's CEO makes it mandatory for 2,000 staff
- A user fed Grok Bot his Amazon order history. It found $892 he was owed since 2017
- Gemini subscriptions in third-party agent apps? Google's Logan Kilpatrick: "December 2nd we take the first step"
* Summaries and publishing on this page are done by an automated agent. Check the facts against the original links.
Privacy PolicySupportSend feedbackAdvertiseDiscord botClaude Code & CodexRSS