Codex flooded Git with commits until it buckled: OpenAI built its own Git backend, 'Pando,' and cut push latency about 80%

OpenAI said it built a new Git backend, Pando, after its existing Git servers could not keep up with the work Codex generates, and now runs all development in its monorepo on it.

In a Nov. 2 blog post, "Scaling Git at OpenAI," the company said the source-control system had to be revamped for the level of concurrent work Codex makes possible. Since around the end of last year fetches and pushes had slowed and merges failed, with reliability at times below 90%. Since Pando, median push latency has fallen nearly 5x and it serves close to a petabyte of Git data a day.

According to The New Stack, quarterly commits to the monorepo will be up more than 2,400% by the end of this year versus Q1 2024. Pando drops global ordering of every branch and coordinates each branch on its own, so unrelated pushes do not queue. It also reuses stored packfiles instead of building one per fetch, served from regional caches with a hit rate above 99%. The trade-off, OpenAI said, is that a reader looking at several branches may see them at different points in time.

Interesting Engineering reported OpenAI built it with Codex in 10 weeks, with one engineer designing the core and teams of agents implementing it; it went into production in April and merge reliability is 99.99%.

OpenAI's Tibo (@thsottiaux) wrote "We know some Git-fu" (2,800+ likes). Greg Brockman, OpenAI co-founder, wrote that OpenAI is "considering providing this to others" and asked interested parties to email him (1,090+ likes). One reply noted a service named gitpando.com already exists.

The New Stack noted GitHub spent much of its recent Universe keynote on infrastructure, and that Git-based efforts like Entire and Cursor's Origin are emerging while Anthropic is starting from scratch with a new client and architecture.