Every time you ask an AI model a long question, something invisible and expensive happens inside the data center: a structure called the key-value cache bloats with information, consuming memory at a rate that quietly inflates the cost of every response, limits how long a model's "attention span" can be, and constrains how many users a single GPU can serve at once. It is, in the language of infrastructure, a bottleneck with a billion-dollar shadow.
On Monday, two Google Research scientists — Amir Zandieh and Vahab Mirrokni, a VP and Google Fellow — published details of a new compression algorithm called TurboQuant, which they claim can shrink that cache to as little as 3 bits per value, achieving up to 6x memory reduction, with effectively zero loss in model accuracy. The paper is slated for presentation at ICLR 2026.
If the results hold under independent scrutiny, the implications ripple far beyond a single benchmark.
The Problem With Being Precise
Modern large language models think in vectors — vast arrays of decimal numbers encoding meaning, context, and relationships. Storing those vectors at full 32-bit precision is memory-intensive by design. The key-value cache is where the model parks recently computed context so it doesn't have to recompute it constantly — a speed optimization that has itself become a memory liability as context windows stretch into the hundreds of thousands of tokens.
Prior compression techniques tried to address this, but most carried a hidden cost: they required storing extra "quantization constants" for every small block of compressed data, adding 1 to 2 bits of overhead that partially erased the gains.
TurboQuant attacks that overhead directly, using two interlocking algorithms the team also developed: PolarQuant and the Quantized Johnson-Lindenstrauss transform (QJL).
An Elegant Geometric Trick
PolarQuant's innovation is conceptual before it is computational. Traditional quantization treats vectors like a city grid — "3 blocks east, 4 blocks north" — which requires constantly recalibrating the grid's boundaries as data changes. PolarQuant reframes the same vector in polar coordinates: one radius, one angle. Because the angular distribution of a randomly rotated high-dimensional vector is mathematically predictable, the expensive normalization step disappears entirely. The boundaries are already known.
QJL handles the residual error left behind. Using the Johnson-Lindenstrauss transform — a classical result in dimensionality reduction — it collapses remaining error into a single sign bit (+1 or -1), adding exactly zero memory overhead while correcting a subtle but consequential bias. Without this correction, even a good quantizer produces systematically skewed inner-product estimates; in attention mechanisms, that skew compounds across millions of operations.
TurboQuant chains both methods: PolarQuant handles the bulk of compression, QJL scrubs the residual bias with one bit.
What the Benchmarks Show
Tested on Llama-3.1-8B-Instruct and Ministral-7B-Instruct across LongBench, Needle-in-a-Haystack, and several other standard long-context evaluations, TurboQuant at 3.5 bits matched full-precision baseline scores exactly on LongBench (50.06 vs. 50.06). On needle-in-a-haystack tasks — designed to find one fact buried in a massive document — it scored 0.997 against a full-precision baseline of 1.0. On H100 GPUs, 4-bit TurboQuant computed attention logits up to 8 times faster than unquantized 32-bit keys.
For vector search — the retrieval backbone of modern AI products — TurboQuant outperformed both PQ and RabitQ on recall across benchmarks, without requiring the offline codebook training those methods depend on. Indexing time: milliseconds, versus hundreds to thousands of seconds for baselines.
The Caveats Are Real
Independent reviewers examining the work note that the theoretical guarantees rest on high-dimensional asymptotic behavior — the assumption that random rotation sufficiently decouples coordinates. The lower-bound comparison is information-theoretic and worst-case, not a direct operational comparison against all possible competitors. The benchmark suite, while strong, does not yet cover the full range of model families, deployment configurations, or production latency scenarios that enterprise adoption would demand. And critically: this is still an arXiv preprint, awaiting peer review.
The Commercial Stakes
For AI inference providers, KV cache is a front-line cost. A method delivering 4.5x to 5x compression with negligible quality degradation — requiring no retraining, no calibration, no dataset-specific tuning — could meaningfully shift the economics of long-context serving: larger batch sizes, longer supported contexts per GPU, lower cost per query. For vector database companies and retrieval-augmented generation systems, the appeal is different but equally direct: smaller indices, faster ingestion, no retraining when embeddings change.
The researchers position TurboQuant not as an engineering patch but as a fundamental algorithmic result, provably near the theoretical limit of what compression can achieve. That claim awaits the judgment of the field. But the early read, from both the benchmarks and the math, suggests it may be worth taking seriously.
Sources: https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/