Latency is the most misreported number in LLM serving. Providers publish averages, users experience tails, and the two can differ by an order of magnitude. This page covers what to measure and why the published figures rarely match production.
Three Different Numbers
| Metric | What it measures | Dominated by |
|---|---|---|
| Time to first token (TTFT) | Delay before output starts | Prefill compute, queueing, network |
| Inter-token latency | Gap between successive tokens | Memory bandwidth, batch pressure |
| Total completion time | End to end for the full response | TTFT plus output length |
These optimise against each other. Aggressive batching improves throughput per dollar and inter-token latency under load while worsening TTFT, because requests wait to be grouped. A provider tuned for cheap bulk serving will feel slow interactively even at good headline throughput.
The Tail Is the Product
Median latency describes a typical request. Users perceive the slow ones. For interactive applications, p95 and p99 determine whether the product feels reliable, and the gap between median and p99 varies enormously across providers, more than median latency itself varies.
Tail latency is also where shared capacity shows up. A provider running high utilisation on shared hardware delivers good medians and poor tails, because your request occasionally queues behind someone else's large batch. Dedicated capacity mostly solves this and costs multiples more.
What Published Benchmarks Miss
Five things. Region: cross-continent adds real round-trip time before any compute happens. Time of day: shared capacity degrades under regional peak load. Prompt length: TTFT scales with prefill, so a benchmark on short prompts says nothing about long-context behaviour. Output length: total time is dominated by how much the model generates, and verbose models lose on total time even at good per-token speed. And cold starts: less popular models may not be resident, adding seconds to the first request.
The output-length point interacts with token efficiency. A model using 17 percent fewer output tokens finishes meaningfully sooner at identical per-token speed. See the Intelligence Index, which reports token consumption alongside capability.
Measuring Your Own
Test with your real prompts at your real lengths, from your real region, at your real time of day, and report p50, p95, and p99 rather than a mean. Run for at least a full daily cycle. A single afternoon benchmark from a well-connected data centre will look excellent and predict nothing.
Brand Visibility Implications
Latency budgets constrain retrieval depth. An assistant working against a tight response-time target retrieves fewer sources and cites fewer of them; one with headroom can gather and compare more. Platform latency characteristics therefore partly determine how many brands make it into a given answer, which is a structural reason citation counts differ across platforms independent of content quality.
Methodology
Compiled from provider documentation, published benchmark suites, and independent measurement through July 2026. Inference performance figures depend heavily on model, quantisation, batch size, context length, and region, so ranges are given rather than point values and cross-provider comparison carries real uncertainty. Re-verify before quoting. Updated quarterly.
How Presenc AI Helps
Presenc AI measures how many sources reach an answer on each platform and which brands survive that filtering.