Standard attention costs quadratic time and memory in sequence length. At a million tokens that is untenable, so every long-context model shipping in 2026 replaces or modifies it. The designs differ in what they give up.
The Approaches
| Approach | Mechanism | Trade-off |
|---|---|---|
| Sliding window | Attend only within a fixed local window | Loses long-range dependency entirely |
| Sliding window plus sinks | Local window plus retained initial tokens | Recovers stability, not full long-range recall |
| Block-sparse selection | Select which cache blocks to attend to per query | Selection quality becomes a failure mode |
| Linear attention | Reformulate to avoid the quadratic term | Historically weaker exact recall |
| State-space hybrids | Interleave Mamba-style layers with attention | Complexity, harder to reason about |
| Latent compression | Compress keys and values to lower dimension | Lossy, tuned at training time |
MiniMax Sparse Attention
MSA, shipped in MiniMax M3, is a two-stage block-sparse design. A lightweight index branch selects which blocks of the key-value cache are relevant to a given input, then the main attention layer processes only those. MiniMax reports more than 9x faster prefill and more than 15x faster decode against MiniMax M2, at roughly one twentieth the per-token compute.
Those are large claims from a single lab's own reporting, and independent verification remains limited. If they hold and generalise, the cost of long-context inference falls enough to change what retrieval-heavy answering costs. See the MiniMax M3 brief.
Kimi Delta Attention
Kimi K3 introduced Kimi Delta Attention alongside Attention Residuals, which Moonshot credits for improved efficiency and reasoning quality at 2.8 trillion parameters and a million-token context. Architectural detail is thinner in public reporting than for MSA, and the two innovations are described together, which makes their individual contributions hard to separate. See the Kimi K3 brief.
The Recurring Trade
Every one of these designs buys efficiency by not attending to everything, and the question is always whether the skipped material mattered. Benchmarks tend to reward designs that handle typical long documents well. The failure mode is atypical retrieval: the specific fact in the specific place the selection heuristic decided to skip.
This is the same structural problem as cache eviction, arriving through architecture rather than serving policy. See KV cache compression. A model can hold a strong long-context benchmark score and still miss the one detail your use case depends on, so test against your own retrieval pattern.
Brand Visibility Implications
Attention architecture determines what an assistant actually reads from a retrieved page. A brand mentioned once, deep inside a long document, is precisely the case sparse selection is most likely to skip. Under these architectures, being present in a source is weaker than being prominent in it: early placement, heading proximity, and repetition matter more than they did under dense attention. This is a mechanistic reason why the structural placement advice in GEO practice holds, rather than a stylistic preference.
Methodology
Compiled from provider documentation, published benchmark suites, and independent measurement through July 2026. Inference performance figures depend heavily on model, quantisation, batch size, context length, and region, so ranges are given rather than point values and cross-provider comparison carries real uncertainty. Re-verify before quoting. Updated quarterly.
How Presenc AI Helps
Presenc AI measures whether a brand mention in a retrieved source actually reaches the generated answer, and how placement within the source affects that.