Your dashboard shows a P95 latency of 4.2 seconds. The benchmark site says your model does 800 tokens per second — “fast” — and your users are still closing the tab before the first token lands.
That gap is the whole story of LLM latency: the number everyone benchmarks and the number users actually feel are different numbers, and most teams optimize the wrong one.
Latency is the most data-rich, prescription-poor corner of the LLM stack: benchmark sites publish TTFT and tokens-per-second tables by the hundreds — Artificial Analysis’s provider pages and AI API Cost’s live speed leaderboard are the reference points — and almost none of them tell you what to do about your number. Vendor docs explain their own stack, not yours. And the field’s favorite metric — tokens per second — is regularly mistaken for the thing users actually feel.
This guide is the prescription layer: what latency actually means (four different numbers, only one of which is TPS), why it’s a product metric now, how to measure it with a budget you can defend, how to fix each layer — streaming, caching, model tiering, regional routing — and the mistakes that turn optimization into superstition.
What “Latency” Actually Means for LLM APIs
Takeaway: there are four latency numbers, and optimizing the wrong one is how teams ship “fast” models that feel slow.
- TTFT — time to first token. What users feel first: the gap between “send” and the first streamed token. The single most important number for interactive products.
- Inter-token latency — the inverse of TPS. How fast the rest of the response streams. Matters for long outputs and agent loops; irrelevant for a one-line answer.
- Total request time. The sum, minus streaming perception. What batch and background work cares about; not what users feel.
- Perceived latency. What streaming turns into: TTFT plus the pacing of the stream. A system with a good TTFT and steady pacing feels fast even when total time is long.
The field’s fixation on TPS is the trap: an 800 tokens/sec model with a 1.5-second TTFT loses the first impression to a 300 tokens/sec model with a 300ms TTFT. Measure all four; optimize by use case.
Why Latency Is a Product Metric
Takeaway: agents multiply latency, and users feel the product — the numbers moved from the benchmark sheet to the churn chart.
Two structural reasons latency is now a product decision:
-
Agent loops multiply every wait. A single conversation makes N sequential model calls; a 500ms-per-call advantage becomes a 5-second advantage on a 10-call loop. The same compounding logic that drives the 800ms rule for interactive systems applies here: sequential calls multiply, so per-call latency is a product decision, not a performance nicety.
-
Perceived speed is retention. Streaming UI studies consistently show first-token time drives perceived quality; the fastest benchmark model loses if its TTFT is slow. The product question isn’t “how fast is the model” — it’s “how fast is my user’s first token.”
How to Measure: The Latency Budget
Takeaway: measurement is a budget, not a benchmark — decompose, measure at P50 and P95, in your region, with your prompt sizes.
The budget decomposition:
| Stage | What it includes | Typical range |
|---|---|---|
| Network | DNS, TLS, connection, regional distance | 20-200ms |
| Queueing | provider-side, rate-limit proximity | 0-500ms+ |
| TTFT | model prefill + first token | 200-1500ms |
| Inter-token | generation pacing | 1-5ms/token |
| Client | parsing, rendering, streaming plumbing | 10-100ms |
Rules that make the budget honest:
- Measure in production regions. A US-east measurement of a Singapore-bound product is a different number entirely — regional latency can exceed model latency.
- P50 hides the story; P95 tells it. The median hides the timeouts; the tail is what users remember.
- Same prompt sizes, same concurrency. Benchmarks that use tiny prompts flatter TTFT and punish nothing. Your prompt mix or the measurement is marketing.
The measurement script, minimal but honest:
import time
from openai import OpenAI
client = OpenAI() # your production endpoint and region
PROMPT = "..." # a representative production prompt
N, STREAM = 50, True # 50 runs, streaming on
ttfts, ipss = [], []
for _ in range(N):
t0 = time.perf_counter()
first = True
stream = client.chat.completions.create(model="gpt-4o-mini",
messages=[{"role": "user", "content": PROMPT}],
stream=STREAM)
for chunk in stream:
if first:
ttfts.append((time.perf_counter() - t0) * 1000) # TTFT in ms
first = False
t_last = time.perf_counter()
else:
ipss.append((time.perf_counter() - t_last) * 1000) # inter-token ms
t_last = time.perf_counter()
def pct(xs, p):
xs = sorted(xs); return xs[int(len(xs) * p)]
print(f"TTFT P50={pct(ttfts, .5):.0f}ms P95={pct(ttfts, .95):.0f}ms")
print(f"Inter-token P50={pct(ipss, .5):.1f}ms P95={pct(ipss, .95):.1f}ms")Run it from the regions your users are actually in, with your real prompt sizes, and record the results as the baseline your CI suite compares against.
The measurement habit belongs in CI: a latency regression suite that fails on TTFT drift is the only thing that keeps “the model got better” honest — the same discipline good observability practice builds for the rest of your stack.
How to Optimize: Streaming, Caching, Routing
Takeaway: three levers, in order of implementation — stream everything, cache the repeated, route by tier — and all three are configs, not projects.
Layer 1 — Streaming. The first lever and the cheapest: stream responses and render tokens as they arrive. The interactive decision is SSE versus WebSocket — SSE for request-response streams (simpler, HTTP-native, works through most proxies), WebSocket for bidirectional flows (agents, realtime audio). The production details that break naive streaming: proxy buffering (intermediaries that hold the response until it’s complete defeat the purpose), disconnect handling (the client abandons; the stream must abort), and backpressure. Streaming doesn’t make the model faster — it makes the user’s wait disappear into the stream’s pacing.
Layer 2 — Caching. Two distinct wins: prompt caching (repeated prefixes skip prefill, cutting TTFT on the second identical call) and response caching (identical requests served without model contact). The prompt caching guide covers the economics; the latency angle is the same lever: stable prefixes make the second call faster and cheaper. Watch the miss rate — a cache that misses 90% of the time adds overhead without benefit.
Layer 3 — Model tiering and routing. The interactive path doesn’t need the frontier model for every turn: route UX-critical calls to the fast tier (including the specialized-chip providers covered in our fast-inference comparison) and background work to the cost tier. Custom routing makes the per-request decision mechanical, and fallbacks keep the fast tier’s occasional unavailability from becoming your latency story.
How to Fix Global Latency: Regional Routing
Takeaway: for a global user base, region choice beats model choice — the same model, the right region, is the difference between 300ms and 900ms.
The numbers don’t lie: a model served from the US to a European user carries 100-200ms of extra network latency per hop versus a regional endpoint, and the difference compounds across agent loops. The fix is architectural:
- Nearest-region routing. Serve each user from the closest region with the model available — the load-balancing layer does this at the endpoint level.
- Auto-failover across regions. When a provider or region degrades, fail over to the next region before the user’s TTFT budget is spent — the auto-failover docs cover the mechanics.
- Edge calls, briefly. For extremely latency-sensitive paths, edge functions can front the LLM call — reducing the network hop and handling connection reuse. The honest note: edge adds its own cold-start cost, and for most workloads the regional endpoint wins; test before adopting (this is the one-edge-case we’ll call out rather than over-promise).
The measurement rule from earlier applies here with extra force: regional routing decisions need regional measurements — a US benchmark of a global routing change is not a measurement.
Common Mistakes
Takeaway: four failure modes — each one turns a latency initiative into theater.
- Optimizing one metric. TPS-fixation with TTFT ignored; TTFT-fixation with streaming ignored. The four-number model in this guide is the antidote.
- Ignoring the network layer. All model-side optimization, zero regional routing — for a global product, that’s optimizing the wrong half of the budget.
- Cache with no hit-rate discipline. Caching “because it’s fast” with a 90% miss rate — the caching economics only work when the prefix is stable and the hit rate is measured.
- No baseline before optimizing. Changed the stack, shipped the change, never measured the before. Without the CI latency suite, “optimization” is a hope with a dashboard.
FAQ
Which latency metric should I optimize first?
TTFT for interactive products — it’s what users feel first. TPS for agent loops and long outputs. Optimize the metric your use case feels, then measure the others so they don’t regress silently.
Is streaming faster, or does it just feel faster?
Both, in different senses: total time is usually unchanged, but perceived latency collapses because the first token arrives early and the stream paces the rest. For interactive products, perceived latency is the product metric — stream everything.
How much does region choice affect latency?
Often more than model choice: cross-continental network hops add hundreds of milliseconds per call, compounding across agent loops. Nearest-region routing is the highest-leverage latency change most global products never make.
SSE or WebSocket for streaming?
SSE for request-response streams — simpler, HTTP-native, proxy-friendly. WebSocket for bidirectional flows like realtime agents. Choosing by the flow shape, not by hype, is the whole answer.
Does caching really reduce latency?
Prompt caching cuts TTFT on repeated prefixes (prefill is the expensive part), and response caching eliminates model latency entirely for identical requests — but only when hit rates are real. Measure the hit rate; the miss rate is the tax.
Why is my P95 so much worse than my P50?
Tail latency in LLM APIs comes from provider queueing under load, rate-limit proximity, and regional network variance. If the tail matters (it does), the fix is a mix of headroom, fallbacks, and regional routing — not a faster model.
Summary
LLM API latency optimization is a four-number problem: measure TTFT, inter-token, total, and perceived latency at P50 and P95 in your production regions; then fix each layer — streaming for perception, caching for repeats, tiering for cost-speed balance, and regional routing for the network half of the budget. The benchmark tables tell you what’s possible; your budget tells you what’s yours. Measure first, optimize second, and the fastest model in the world becomes the one your users actually feel.
You can’t fix a latency budget you’ve never measured. Get your TokSpan API key — $5 in free credits to measure with — and run the same prompts from a few regions.