At the context lengths shipped in 2026, the key-value cache is frequently larger than the model weights. Weight quantisation solved one memory problem and revealed another, and cache compression is now where most serving-side memory work happens.
Why the Cache Got So Large
Every token processed leaves key and value tensors that must be retained so subsequent tokens can attend to them. Cache size scales linearly with context length, and context lengths went from 8,000 tokens to a million in roughly three years. For a 70B-class model the cache adds roughly 1-2GB at 8K tokens and 150-250GB at 1M, against roughly 35GB for the weights at Q4.
That is the practical reason million-token context windows are widely advertised and rarely used at full length outside data centres. See per-model memory requirements.
The Techniques
| Technique | Approach | Quality cost |
|---|---|---|
| Cache quantisation | Store keys and values at lower precision | Low at 8-bit, noticeable below |
| Token eviction | Drop cache entries judged unimportant | Depends entirely on the heuristic |
| Attention sinks | Retain the first few tokens plus a sliding window | Loses genuine long-range recall |
| Grouped and multi-query attention | Share key-value heads across query heads | Architectural, set at training time |
| Latent attention | Compress keys and values into a lower-dimensional space | Architectural, substantial savings |
| Sparse attention selection | Attend only to selected cache blocks | Architectural, see MiniMax MSA |
The first three are serving-side and can be applied to an existing model. The last three are architectural and must be designed in at training time, which is why recent open-weight releases increasingly advertise their attention design. See sparse attention architectures.
The Eviction Problem
Eviction is the most tempting and most dangerous family. Dropping cache entries frees memory immediately, and heuristics based on attention scores usually work well on benchmarks. The failure mode is that "usually" hides the case you care about: a detail mentioned once early in a long document is exactly what a low attention score would evict and exactly what a user asking about that document wants retrieved.
This produces a characteristic failure where a model handles long context well on aggregate benchmarks and fails on specific needle-retrieval questions. Any long-context claim is worth testing against your own retrieval patterns rather than trusting a benchmark average.
Brand Visibility Implications
Cache compression decides how much retrieved material an assistant can actually hold while answering. When cache is expensive, systems retrieve fewer sources and truncate more aggressively, which favours whichever source ranks first. When it is cheap, more sources survive into the answer and citation counts rise. There is also a specific risk worth knowing: aggressive eviction can drop a brand mentioned once early in a long retrieved document, so being mentioned is not the same as surviving to the answer. Position and prominence within a source matter more under compression than they do without it.
Methodology
Compiled from provider documentation, published benchmark suites, and independent measurement through July 2026. Inference performance figures depend heavily on model, quantisation, batch size, context length, and region, so ranges are given rather than point values and cross-provider comparison carries real uncertainty. Re-verify before quoting. Updated quarterly.
How Presenc AI Helps
Presenc AI measures whether a brand survives from retrieved source into the final answer, not only whether the source was retrieved.