Google unveils TurboQuant — a candidate "terminator" for memory inflation in large models

Big claim, simple pitch

Google today introduced TurboQuant, an extreme compression algorithm it says can slash the working memory needed for large language model inference while keeping quality intact — and in some cases speeding things up. Could this finally change the calculus that has driven server memory and SSD demand higher? According to Google’s research blog and accompanying paper, TurboQuant compresses the key-value (KV) cache that transformer decoders use to “remember” prior tokens, reducing the in-memory footprint to only a few bits per channel with near-zero preprocessing.

Two-step math: PolarQuant and QJL

Google describes TurboQuant as a two-stage approach. The first stage, PolarQuant, re-parameterizes high‑dimensional vectors in a polar-like coordinate system after a random rotation, enabling a precomputed codebook to compactly represent most of the signal online. The second stage, QJL (a quantized Johnson‑Lindenstrauss–style residual correction), captures the leftover error with an extremely small 1‑bit residual that produces an unbiased inner‑product estimate when combined with the high‑precision query. Google reports results such as KV cache storage down to ~3 bits/channel, parity with full‑precision on long‑context benchmarks at 3.5 bits, only slight loss at 2.5 bits, and an 8× speedup on key attention kernels on an H100 GPU for its 4‑bit configuration.

Practical impact and market reaction

The paper frames TurboQuant as immediately relevant to long‑context inference, vector databases, real‑time indexing and on‑device reasoning for phones and embedded systems — areas where memory bandwidth and capacity are the critical bottlenecks. It has been reported that U.S. storage stocks, including makers of DRAM and SSDs, dipped on the news, reflecting investor concerns that widespread adoption of extreme quantization could alter future memory-capacity demand curves and server procurement plans.

Where this sits in geopolitics and the research calendar

The timing matters. Memory and storage supply chains have been politically charged recently: advanced semiconductor export controls and supply‑chain diversification are reshaping where and how cloud and AI hardware are bought and built. TurboQuant could reduce pressure for very large high‑bandwidth memory pools in data centers, but it does not erase strategic concerns about chip fabs, access to accelerators, or national technology policy. Google says the work will appear at ICLR 2026 and AISTATS 2026; the company’s blog and arXiv preprint contain the technical details and benchmark claims for reviewers and industry watchers to vet.