Speculative decoding is the rare optimisation that makes inference faster without changing the output distribution. That property, exact equivalence rather than approximation, is why it moved from research to default in serving stacks faster than most efficiency techniques.
The Mechanism
A small fast draft model proposes several tokens ahead. The large target model then verifies all of them in a single forward pass, because verifying a sequence is parallel work while generating one is sequential. Accepted tokens are kept; at the first rejection, generation falls back to the target model's own choice and drafting restarts.
The output is mathematically identical to what the target model would have produced alone. You are not trading quality for speed, which is unusual and is the whole appeal.
Acceptance Rate Decides Everything
| Acceptance rate | Typical outcome | |
|---|---|---|
| High, easy or predictable text | Substantial speedup, often 2-3x | |
| Moderate | Modest speedup | |
| Low, hard or unusual content | Net slowdown from wasted draft compute |
Speedup depends on how often the draft model guesses right, which depends on content. Boilerplate code, formulaic prose, and structured output draft extremely well because the next token is often nearly determined. Novel reasoning and unusual domain content draft poorly. Reported average speedups therefore hide wide variance, and a figure measured on code completion will not hold on open-ended reasoning.
The draft model also has to be genuinely cheap. If drafting costs a meaningful fraction of verification, the arithmetic stops working even at good acceptance rates.
Variants
Three families. Separate draft models use a smaller model from the same family, which is simple but requires a suitable small sibling. Self-speculation methods such as Medusa and EAGLE add lightweight prediction heads to the target model itself, removing the need for a separate model. And n-gram or prompt-lookup drafting predicts from text already in the context, which is remarkably effective for tasks with heavy repetition such as editing a document or summarising a provided passage.
Prompt-lookup drafting is underrated. For any workload where the output substantially echoes the input, it is nearly free.
Adoption
Speculative decoding is supported across the major open serving stacks and is used internally by large providers, though most do not disclose where. Since it does not change output, providers have no obligation to announce it, which makes deployment hard to track from outside. Treat it as widely deployed rather than precisely measurable. See the serving stack comparison.
Brand Visibility Implications
The indirect effect is what matters. Cheaper inference means systems can afford more steps within the same budget: more retrieval, more comparison, more verification. Techniques like this are part of why agentic, multi-step answering became economically viable, and multi-step answering cites more sources than single-pass answering does. See the inference and training split.
Methodology
Compiled from provider documentation, published benchmark suites, and independent measurement through July 2026. Inference performance figures depend heavily on model, quantisation, batch size, context length, and region, so ranges are given rather than point values and cross-provider comparison carries real uncertainty. Re-verify before quoting. Updated quarterly.
How Presenc AI Helps
Presenc AI measures how many sources reach an answer across platforms, which is where inference-efficiency gains ultimately show up for brands.