Tokens per second is the number everyone quotes and the wrong one to optimise for agentic work. Time to first token, governed by prefill throughput, is what determines whether a local setup feels usable when the prompt is long. This page measures it.
Prefill and Decode Are Different Problems
Decode generates one token at a time and is bandwidth-bound: the model must read its weights from memory for every token produced. Prefill processes the entire input prompt in parallel and is compute-bound. A device can be excellent at one and mediocre at the other, and the two failure modes feel completely different to use.
| Device | Prefill, 8B class (tokens/sec) | Decode, 8B class (tokens/sec) | Ratio |
|---|---|---|---|
| RTX 5090 | ~7,200 | 142-178 | ~45x |
| DGX Spark | ~1,200 | 110-125 | ~10x |
| Mac Studio M5 Max | ~400 | 95-110 | ~4x |
The RTX 5090 and the M5 Max are within about 1.5x of each other on decode and roughly 18x apart on prefill. Any benchmark quoting only tokens per second hides that entirely.
What This Costs in Wall-Clock Time
| Prompt length | RTX 5090 | DGX Spark | M5 Max |
|---|---|---|---|
| 2,000 tokens | ~0.3 s | ~1.7 s | ~5 s |
| 8,000 tokens | ~1.1 s | ~6.7 s | ~20 s |
| 40,000 tokens | ~5.6 s | ~33 s | ~100 s |
| 128,000 tokens | ~18 s | ~107 s | ~320 s |
A coding agent re-sending a 40,000-token context on every turn waits under six seconds per turn on a 5090 and over a minute and a half on an M5 Max. Across a working session that is the difference between a tool you use and one you abandon.
The M5 Neural Accelerator Path
Apple's M5 generation narrows this specifically. Neural Accelerators in the GPU cores give MLX a reported 4.06x time-to-first-token advantage over llama.cpp's Metal backend. Measured on an M5 Max with a 35B-class sparse model in NVFP4, prefill moved from 1,154 to 1,810 tokens per second. That does not close the gap to a 5090, but it materially changes the experience, and it only applies if you are using MLX. See MLX versus llama.cpp.
Mitigations
Prompt caching is the most effective, reusing the computed key-value cache across turns so only new tokens are processed. Most serious runtimes support it and it converts the repeated full-prefill cost into an incremental one. Beyond that: keep system prompts short, avoid re-sending unchanged documents, prefer retrieval over stuffing the full corpus into context, and pick a smaller model when the prompt is long, because prefill cost scales with model size as well as prompt length.
Brand Visibility Implications
Prefill economics shape how much source material an AI system will pull in before answering. When prompt processing is expensive, systems retrieve less and lean harder on parametric recall, which favours brands with strong training-data presence. When it is cheap, retrieval-heavy answering dominates and well-structured, easily-retrieved documentation wins. Hardware improvements at this layer therefore shift the balance between the two levers brands have. See RAG fetchability benchmarks.
Methodology
Vendor specifications come from NVIDIA, Apple, and model-card publications. Throughput figures aggregate community benchmark reporting from the llama.cpp discussions, the MLX repository, and published independent test suites. Single-stream decode unless stated otherwise. Ranges rather than point values are used wherever independent runs disagree, which is most of the time: quantisation format, prompt length, thermal state, and runtime version each move these numbers by more than the differences being measured. Treat every figure as an order-of-magnitude guide, not a specification. Updated quarterly.
How Presenc AI Helps
Presenc AI measures both parametric brand recall and retrieval-driven citation, so teams can see which lever is carrying their visibility on each platform.