Let VR

There is no typing indicator in a room

Conversational latency that a text interface absorbs invisibly becomes a performance problem the moment the speaker has a body.

10 min read

The delay did not change; the contract did

Put a language model behind a chat box and a two-second wait is unremarkable. The interface has an established way of saying "working on it" — a spinner, three dots, a streaming cursor — and users have spent years learning to read it. Nothing about the wait needs explaining.

Put the same model behind a character standing two metres from you in a headset and the two seconds become a problem that no amount of backend work fixes, because the problem is not the duration. A person who is present and silent is communicating something. Depending on how long the silence runs and what preceded it, they are thinking, hesitating, declining to answer, or broken. The user will pick one of those readings, and if your character has no way to indicate which, the reading they pick is usually the last one.

What human conversation actually runs at

The timing reference here is unusually solid. Work on turn-taking across ten unrelated languages found the gap between one speaker finishing and the next starting clusters tightly around 200 milliseconds, with remarkably little cultural variation. That is faster than it is possible to plan a response, which is the point: listeners begin preparing their turn while the other person is still talking, and launch it on a predicted ending.

This matters for what a delay means rather than for what a delay costs. Because the baseline is that tight, silences above it are not neutral. A gap of half a second is already long enough to be noticed. Around a second, listeners start reading it as reluctance or disagreement — this is well documented in how people interpret delayed responses to invitations and requests. Beyond that the interpretation depends entirely on what the speaker is visibly doing.

A model pipeline that answers in 800 milliseconds is doing well by engineering standards and is sitting squarely in the range where a human listener expects the pause to mean something. You are not going to win this on latency alone. The question is what the character is doing with the time.

The pieces the delay is made of

Worth separating, because they have different fixes and only some of them are on your side of the network.

Endpointing
Deciding the user has finished speaking. Aggressive settings cut people off mid-sentence; conservative ones add a fixed tax to every single turn. This is frequently the largest single contributor and the one teams look at last, because it is not where the interesting technology is.
Transcription
Streaming models produce partial results as the user talks, which means most of this can be overlapped with speech rather than paid afterwards. A batch transcription that starts when the user stops is paying for the whole utterance twice.
Time to first token
Depends on model size, deployment, and how long your prompt is. Prompt length is the part under your control and the part that quietly grows: a conversation history that accumulates unbounded makes every turn slower than the last, which users experience as the character getting tired.
Speech synthesis
A streaming voice can start speaking on the first clause rather than the last token, which decouples "first sound" from "full answer". This is the single largest perceived win available, because the user is responding to when the character starts, not when it stops.
Animation dispatch
Lip sync and gesture have to be driven from audio that is still arriving. Systems that wait for a complete audio buffer before animating give back everything the streaming voice just saved.

Variance hurts more than the mean

A pipeline that reliably answers in 900 milliseconds is more comfortable to talk to than one averaging 600 while ranging between 200 and 2,000. This is not intuitive from a metrics dashboard, where the second system looks better.

The reason is that a consistent delay can be absorbed into character. Someone who always takes a beat before answering is thoughtful, or careful, or elderly, and after three exchanges the user has adjusted their own timing to match and stopped noticing. A delay that varies cannot be read as a trait, because traits are stable. It gets read as the thing it is: a system whose response time depends on something the user cannot see.

The design consequence is that clamping is sometimes worth more than optimising. Holding a fast response back to a floor of around 700 milliseconds, so that the fast and slow cases look the same, costs you nothing a user values and removes the jitter they were reading as unreliability. Engineers tend to resist this, because it means deliberately making the good case worse, and it is usually the right call anyway.

The performance that fills the gap

Since the silence will be interpreted, the practical move is to give it something to be. A character who breaks eye contact and looks up and away is visibly thinking, and that reading holds for a surprisingly long time — considerably longer than a character who simply stands still, which reads as frozen within about a second.

The trap is that any single filler animation is a tell the moment it repeats. Users identify a looping thinking pose faster than designers expect, and once identified it stops being a thinking pose and becomes the loading spinner it always was. A small library sampled without immediate repetition holds up; one gesture does not survive a ten-minute conversation.

An immediate non-verbal acknowledgement is worth more than a better filler. A nod, a small lean, a shift of weight, dispatched the instant endpointing fires and before the model has produced anything, tells the user they were heard. That is the actual anxiety in the gap, and it can be answered locally in under a hundred milliseconds while the real response is still being generated somewhere else.

Some of the cheapest wins are in the writing. A character with a reason to be slow — reading something, working with their hands, thinking out loud — buys seconds that a character standing at attention does not. This is also the least portable advice in this piece, because it constrains what your character can be, and that is a product decision rather than an engineering one.

Interruption is the part that is actually hard

Everything above is about making a delay tolerable. The harder problem is that real conversation is interruptible, and most deployed pipelines are not.

Human speakers stop mid-word when the other person starts, and the interruption itself carries meaning: it signals disagreement, impatience, or that the point has landed and elaboration is unwanted. A system that finishes its sentence while the user is talking over it is not merely slow. It is communicating that it is not listening, which is a much worse thing to communicate than a pause.

Handling it means being able to cancel synthesis mid-utterance, stop the animation without it snapping, discard the remainder of the generated response, and re-enter listening with the partial utterance retained as context. Each piece is tractable and the combination is rarely built, because it is invisible when it works and only shows up in user testing when someone talks the way they talk to a person rather than the way they talk to a machine.

This is also where being in VR raises the stakes. A user talking to a chat window knows they are operating software and adjusts. A user standing in a room with an embodied character does not adjust in the same way, because presence is doing its job, and the system gets held to a standard the interface itself invited.

Related pages