Tokens per second is the number everyone quotes and the wrong one to optimise for agentic work. Time to first token, governed by prefill throughput, is what determines whether a local setup feels usable when the prompt is long. This page measures it.
Prefill and Decode Are Different Problems
Decode generates one token at a time and is bandwidth-bound: the model must read its weights from memory for every token produced. Prefill processes the entire input prompt in parallel and is compute-bound. A device can be excellent at one and mediocre at the other, and the two failure modes feel completely different to use.
| Device | Prefill, 7B to 8B model (tokens/sec) | Decode (tokens/sec) | Ratio |
|---|---|---|---|
| RTX 5090 (Llama 2 7B, llama.cpp scoreboard) | 14,000 to 15,000 | 290 to 300 | ~48x |
| DGX Spark (Llama 3.1 8B, Ollama) | 7,614 | 38.0 | ~200x |
| Mac Studio M5 Max (Llama 2 7B, llama.cpp scoreboard) | 3,220 | 119.9 | ~27x |
These are published measurements, revised on October 2, 2026 from earlier estimates. The RTX 5090 is about 2.5x the M5 Max on decode and about 4.5x on prefill. DGX Spark shows the widest split: fast prompt processing and slow generation. Any benchmark quoting only tokens per second hides that.
What This Costs in Wall-Clock Time
| Prompt length | RTX 5090 | DGX Spark | M5 Max |
|---|---|---|---|
| 2,000 tokens | ~0.1 s | ~0.3 s | ~0.6 s |
| 8,000 tokens | ~0.6 s | ~1.1 s | ~2.5 s |
| 40,000 tokens | ~2.8 s | ~5.3 s | ~12 s |
| 128,000 tokens | ~9 s | ~17 s | ~40 s |
These times are arithmetic from the small-model prefill rates above, and larger models are several times slower on every device. A coding agent re-sending a 40,000-token context on every turn waits about 3 seconds per turn on a 5090 and about 12 seconds on an M5 Max with a 7B model. Across a working session that gap adds up.
The M5 Neural Accelerator Path
Apple's M5 generation narrows this specifically. Neural Accelerators in the GPU cores give MLX a reported 4.06x time-to-first-token advantage over llama.cpp's Metal backend. Measured on an M5 Max with a 35B-class sparse model in NVFP4, prefill moved from 1,154 to 1,810 tokens per second. That does not close the gap to a 5090, but it materially changes the experience, and it only applies if you are using MLX. See MLX versus llama.cpp.
Mitigations
Prompt caching is the most effective, reusing the computed key-value cache across turns so only new tokens are processed. Most serious runtimes support it and it converts the repeated full-prefill cost into an incremental one. Beyond that: keep system prompts short, avoid re-sending unchanged documents, prefer retrieval over stuffing the full corpus into context, and pick a smaller model when the prompt is long, because prefill cost scales with model size as well as prompt length.
Brand Visibility Implications
Prefill economics shape how much source material an AI system will pull in before answering. When prompt processing is expensive, systems retrieve less and lean harder on parametric recall, which favours brands with strong training-data presence. When it is cheap, retrieval-heavy answering dominates and well-structured, easily-retrieved documentation wins. Hardware improvements at this layer therefore shift the balance between the two levers brands have. See RAG fetchability benchmarks.
Methodology
Vendor specifications come from NVIDIA, Apple, and model-card publications. Throughput figures aggregate community benchmark reporting from the llama.cpp discussions, the MLX repository, and published independent test suites. Single-stream decode unless stated otherwise. Ranges rather than point values are used wherever independent runs disagree, which is most of the time: quantisation format, prompt length, thermal state, and runtime version each move these numbers by more than the differences being measured. Treat every figure as an order-of-magnitude guide, not a specification. Updated quarterly.
How Presenc AI Helps
Presenc AI measures both parametric brand recall and retrieval-driven citation, so teams can see which lever is carrying their visibility on each platform.