There is a small identity at the heart of every transformer that, once seen, makes a great deal of modern infrastructure inevitable. It is the observation that when an autoregressive model generates the next token, the keys and values it computed for every previous token are exactly the same as they were a moment ago. The model is, in a real sense, asking the past nothing new — it is only asking the past whether this token attends to it. The keys haven’t moved. The values haven’t moved. The work has already been done.
This is why the KV cache exists. It is also, eventually, why the KV cache is the single most expensive thing in modern inference.
Begin in the middle. A transformer generating token t must compute attention against tokens 1 through t-1. Without a cache, each new token costs O(t·d) work in attention alone — a quadratic curve when summed across a generation. With a cache, the keys and values from earlier tokens are simply read off, and the marginal cost per new token becomes linear in t. This is not an optimization; this is what makes real-time generation possible at all. Without it, a thousand-token completion would take orders of magnitude longer than it does, and the shape of every product built on language models would be different.
So we cache. And the moment we cache, the geometry of the system shifts. Compute is no longer the binding constraint of inference; memory bandwidth is. The cache lives on the accelerator. It must be read every generation step. It scales as O(L · H · d) per token, where L is the number of layers, H the number of heads, and d the per-head dimension — and then multiplied by context length. For a 70B-parameter model with a 100k-token context, the KV cache can easily reach tens of gigabytes per request. Twenty users in flight on a single GPU and the cache, not the weights, owns the device.
Everything you have heard about the “long-context era” is, mechanically, about this. Grouped-query attention reduces H in the cache by a factor of 4 or 8 — entire heads share keys and values. Multi-query attention takes the same idea to its extreme limit, with one K and V per layer. PagedAttention, the trick behind vLLM, treats the cache like virtual memory with page tables, so that small KV pages can be shared across requests and fragmentation is bounded. Sliding-window attention bounds the cache by discarding old tokens entirely. Mamba and state-space models propose, more radically, that the cache itself was the wrong shape — a fixed-size recurrent state should replace the linearly growing one.
What is interesting about this whole line of work is how thoroughly it is governed by a single physical fact: the GPU can move data slower than it can multiply numbers. Once that asymmetry exists, every architectural decision in the long-context regime is shaped by it. The math of attention has not changed since 2017. The economics of attention have changed entirely, and the architectures we read about now are downstream of that economics.
There is, in the end, something fitting about the modern transformer being shaped less by its mathematics than by the boring practicalities of where bytes happen to live. The model is the same. The bottleneck moved. And in moving, it reshaped the field around itself — which is the most reliable thing one can say about any technology that is more than five years old. The interesting work is no longer in the cache. It is in trying to abolish it.