AI response latency: what actually matters for customer UX
AI responses are slower than human-typed responses. Here is how we close the perceived gap: streaming, typing indicators, optimistic UI, and when to skip AI entirely.
The expectation mismatch
A human support agent typing in a ticketing system might take 2–5 minutes to respond. A WhatsApp customer expects a reply in under 30 seconds. An AI response that takes 4 seconds feels slow — even though it is 30x faster than a human.
The latency problem is partly technical and partly perceptual.
Technical latency
AI response time is dominated by: 1. Retrieval — vector search over the knowledge base (50–200ms) 2. Reranking — cross-encoder scoring (30–80ms) 3. Generation — time-to-first-token from the model (200–800ms), then streaming 4. Post-processing — faithfulness check, response formatting (100–300ms)
For a transactional query ("what is my order status?"), the total pipeline runs in 800ms–1.5s. For a complex advisory query, it can be 3–5s. Model choice is the biggest lever: switching from Opus to Haiku cuts generation time by 3x for queries that do not need the heavyweight model.
Perceived latency
The perceptual fix is streaming. Rather than waiting for the full response, we stream tokens as they are generated. The customer sees the first word in 200ms and the full response arrives progressively. Perceived completion time is not latency — it is time-to-first-token.
A typing indicator in the widget shows "AI is composing a reply…" immediately after the customer sends a message. This anchors the expectation: something is happening.
Optimistic UI
For confirmations (order status lookups, booking confirmations), we can optimistically render a partial response before generation completes. "Found your order, fetching details…" appears in 100ms; the actual tracking information streams in 800ms. The customer feels an immediate acknowledgment.
When to skip AI
Some queries have a deterministic answer that does not need LLM generation — order status from an API call, store hours from a config, account balance from a database query. We route these directly to a structured response template, bypassing the language model entirely. Response time: under 100ms. This is the highest-leverage latency optimisation: not faster AI, but less AI.
The practical rule: measure time-to-first-token (not end-to-end latency), use streaming for all LLM responses, and identify the high-volume deterministic queries to route around the model.