Habitat serves more than 70 million requests a second. OpenAI reports no provisioning or noisy neighbour incidents in the months since it moved back to one shared pool, and it is building its own store.
OpenAI went back to one shared Cosmos DB container, and it is building its own store for its bulkiest data. Its application database now handles more than 70 million requests a second. According to OpenAI's Part I post, it has grown more than ten times a year for three years.
At that growth rate, having spare storage capacity on launch day is difficult. A shared pool supplies it more cheaply than dedicated containers. Part II of the engineering series, published on 23 October, describes OpenAI's return to one shared Cosmos DB container. Throttling and priority tiers keep it orderly.
The same post says OpenAI is also building its own exabyte scale store for its bulkiest data types. Microsoft supplies the shared pool today. OpenAI says the new store is for that bulky data.
Habitat is the online storage platform behind OpenAI's logins, settings and conversations. It supports products used by over 1 billion people each week across almost 40 regions and serves more than 500 petabytes.
Habitat's main database is Azure Cosmos DB, Microsoft's cloud database. Data sits in containers, and each container is reserved a fixed amount of throughput, counted in request units a second. The first design let many workloads share one container.
Why the first two designs failed
Sharing one container caused two problems. Heavy users of niche product features disrupted core login, which OpenAI calls noisy neighbours. As data grew, partitions split and each new partition received half the throughput. Storage growth alone cut the capacity left for live traffic.
The second design gave each new critical service its own container, while legacy workloads stayed in the shared one. Isolation mostly worked.
Provisioning then became the weak point. Teams had to size capacity before launch, and OpenAI says they often shipped and forgot. Engineers sat in incident rooms on launch day, scaling up containers at the last minute.
OpenAI's Figure 03 shows why pooling wins on launch day. A shared container has 100 million request units a second of capacity, 80% used, which leaves 20 million spare.
A product goes from a 1,000 unit load test to a 1 million unit launch. That is 1,000 times bigger, yet it takes only 5% of the spare room.
A dedicated container of 10,000 units faces the same launch at 100 times its size. Sizing it for the launch means reserving 1 million units up front. OpenAI names that over provisioning as the cost of single tenancy.
In OpenAI's worked scenario in Figure 03, the 20 million units of spare room in a shared container cover all of a 1 million request unit launch, while a dedicated container of 10,000 units covers 1 percent of it.
View data
| Design | Share of launch absorbed |
|---|---|
| Shared container, 20 million units spare | 100 |
| Dedicated container, 10,000 units | 1 |
A load test says little about launch day in a dedicated container. In the shared one, a new use case was unlikely to move total usage much. In a dedicated one it could be 1,000 times the baseline. OpenAI says the dedicated model failed to scale organisationally, because owning teams paid very different amounts of diligence to provisioning.
How the shared pool works
The third design returned to one shared container, using controls to isolate workloads. Hot key throttling slows the busiest keys first. Request tiering, which OpenAI calls Quality of Service, ranks bulk inserts and data extracts below live traffic.
Microsoft's Cosmos DB team also built a serverless mode for OpenAI. Each physical partition, a block of data with its own throughput limit, scales up to 10,000 units a second, and partitions split as usage grows.
OpenAI says the shared container removed the need for upfront capacity planning while keeping workloads isolated. It reports no provisioning or noisy neighbour incidents in the months since.
At this scale, OpenAI works directly with its database vendor’s engineers. Microsoft built a new mode for OpenAI. The two teams monitor every layer jointly and can page each other directly without support tiers. OpenAI calls that joint engineering critical to its Cosmos DB stability in the last year.
Two late 2024 outages show why OpenAI wants that closeness. A power outage hit the primary Azure region of one of its largest Cosmos DB accounts. Those accounts would take several hours to fail over. A month later a connection stampede to the original region overwhelmed its Cosmos DB gateway. OpenAI concludes that at scale, managed services become less managed.
OpenAI says it knows of no other company running Cosmos DB at its scale. Its largest account holds tens of petabytes and serves more than 20 million queries a second.
Why OpenAI is building another store
Replicating data is the next hard problem, and it is why OpenAI is building its own store. In its closing section, the company says scaling Habitat for the next billion users means solving that difficulty. It is building a custom exabyte scale store, optimised for Habitat's bulky data types.
The new store is built to need less flash, meaning fast solid state storage. In exchange it accepts a lower peak of queries per second, and it keeps Cosmos DB's document model.
Using flash has physical costs. In regions with limited data centre capacity, power or cooling, OpenAI says it must pick between nearby storage and serving compute.
Copying data everywhere is rare. Some accounts replicate to all 40 Azure regions, but only for a few hand selected datasets. Less flash per region should matter most where power and cooling cap capacity.
Read performance and dependence on Microsoft
Distance also limits performance, especially for reads. Reads are over 90% of Habitat requests. OpenAI cut the global median read time from roughly 120 milliseconds to roughly 10.
The 12 times improvement came from nearby replication, in region caching and a Rust rewrite. The slowest requests still reach distant regions.
OpenAI's answer to launch day risk was to stop sizing for each launch. It uses one shared container and throttles the hottest keys. Bulk jobs rank below live traffic, and a Microsoft built mode splits partitions as usage grows.
Pooling makes OpenAI more dependent on Microsoft’s engineering support. The dedicated model failed because planning sat with each owning team. The shared one works because Microsoft built a serverless mode for OpenAI and the two teams can page each other directly.
OpenAI's own store is still in development with no date, so the shared Cosmos DB container is the live system behind more than 1 billion weekly users. Microsoft, which supplies it, reports its fiscal first quarter on Wednesday 28 October at 2:30 pm Pacific.