Low-Latency TTS for Voice Agents

A voice agent can pick the perfect words and still feel slow. The caller does not hear the whole reply at once. They hear the gap before the voice starts. That gap is where responsiveness lives or dies. This post is the latency deep-dive that the broader guide on choosing a text-to-speech engine defers to.

We will separate the metric that matters from the one that misleads. We will cover time to first audio, streaming versus batch synthesis, how to measure at the tail, adaptable targets, and the trade-off against naturalness. The goal is a number you can gate on.

What TTS latency actually means

Text-to-speech turns a string of text into an audio waveform. In a voice agent, that step sits at the end of the turn. The agent decides what to say. Then it must say it out loud. TTS latency is how long that saying takes.

But "how long" hides two very different numbers. The first is time to first audio: how long until the caller hears the opening sound. The second is total synthesis time: how long until the full reply is rendered. These are not the same. Confusing them is the most common latency mistake.

Total synthesis time scales with reply length. A long sentence takes longer to render in full. Time to first audio does not. With the right setup, the first chunk of audio can arrive in a fraction of the total. That distinction is the whole game for responsiveness.

Why time to first audio governs responsiveness

Callers judge responsiveness by the silence after they stop talking. If the voice starts quickly, the agent feels alert. If the silence stretches, the agent feels broken. What matters is when speech begins, not when it ends.

This is a property of human conversation. In natural dialogue, turn-taking gaps are short. A long pause reads as hesitation or a dropped line. So the agent must begin speaking inside that comfortable window. Time to first audio is the metric that maps to it.

Total synthesis time barely registers here. Once the caller hears the voice, the rest of the audio streams while they listen. The reply can still be rendering as they hear its first words. What they felt was the gap before it started. That gap is TTFA.

TTS latency is only one slice of the full delay, though. The complete gap also includes speech recognition and the model's thinking time. If you want the whole picture, the end-to-end latency guide walks through every stage. This post zooms in on the TTS slice.

Streaming versus non-streaming TTS

The single biggest lever on TTFA is whether the engine streams. Non-streaming, or batch, synthesis renders the entire reply first. Only then does audio start playing. Streaming synthesis emits audio in chunks as it generates. The first chunk plays while later ones are still forming.

The difference is stark for anything longer than a few words. Batch synthesis ties TTFA to total synthesis time. A long reply means a long wait. Streaming decouples them. TTFA stays low even as the reply grows. For a live agent, streaming is almost always the right default.

Some engines go further and accept streamed text input too. The agent sends words as they are decided, not after the full sentence. This shaves more time off the start. The table below sums up the trade-offs.

Streaming is not free. It adds complexity and a new failure mode. A chunk can lag mid-reply and cause a stall the caller hears as a stutter. So measuring streaming latency needs more than a single start number. You watch the gaps between chunks too.

How to measure TTS latency for voice agents

You cannot improve a number you do not measure honestly. Measure the TTS stage in isolation, on realistic text, at the tail.

1. Isolate the TTS stage — Time from when text is sent to the engine until the first audio byte returns. Exclude recognition and model time so you measure TTS alone.

2. Measure time to first audio, not total time — Record TTFA as the primary number. Track total synthesis time separately, as a secondary signal for long replies.

3. Use realistic reply text — Feed the actual sentences your agent produces, with real names, numbers, and lengths. Short clean samples flatter the engine.

4. Read the tail, not the average — Report TTFA at the 90th and 95th percentiles. An average hides the slow calls that callers actually feel.

5. Measure under load — Re-run at your expected peak concurrency. Latency and its tail both stretch when the engine is busy.

6. Watch inter-chunk gaps for streaming — Track the pauses between audio chunks, not just the start. A late chunk mid-reply is a stall the caller hears.

7. Test across accents, pace, and networks — Vary voice, speaking speed, and connection quality. The tail you find on easy inputs is not the tail production sees.

The same discipline that separates TTFA from response time overall is covered in the latency-versus-response-time guide. Keep the two metrics apart and both stay meaningful.

Target ranges you can adapt

Numbers help, but treat them as starting points, not law. The right target depends on your use case, network, and caller expectations. Latency is an engineering property) you tune, not a fixed constant.

For a live conversational agent, a common target is TTFA under roughly 300 milliseconds at p90. Under about 150 milliseconds feels crisp. Above 500 milliseconds, replies start to feel sluggish. These sit inside the wider turn budget the human ear tolerates. The telecom standard ITU-T G.114 puts the comfortable one-way mouth-to-ear limit near 150 milliseconds, which frames how tight the whole chain must be.

Adapt from there. A reception or sales agent needs the tightest TTFA, since a laggy opener loses the call. An outbound reminder or IVR-style flow can tolerate more. A regulated intake agent may accept slightly more latency in exchange for more careful, natural speech. Set the number to the cost of a slow start in your context.

Whatever target you pick, gate on it. A TTFA budget is only useful as a pass or fail line in your production-readiness bar. Cleared on easy inputs at one call at a time, it means little. Cleared at the tail under load, it means the agent will feel responsive to real callers.

The naturalness-versus-latency trade-off

Faster is not always better. The engines that start speaking soonest sometimes sound flatter. The ones that sound warmest sometimes take longer to begin. There is a real trade-off between latency and quality, and both matter to the call.

Quality is usually scored with a mean opinion score, a rating of how natural and clear the speech sounds. A voice that starts fast but sounds robotic can still lose a caller. A voice that sounds human but starts late feels sluggish. The best setup wins on both, but you often have to balance them.

The balance is use-case specific. A high-volume support line may favor a hair more latency for warmth that keeps callers calm. A quick confirmation flow may favor speed over polish. There is no universal answer. The point is to measure both numbers, on your own traffic, and decide the trade-off deliberately rather than by accident.

Do not judge the trade-off from a vendor demo. Demos use short, clean, cherry-picked text. Your agent produces long, messy, real replies. Score TTFA and naturalness together, on your own sentences, at the tail. That is the only comparison that predicts the live call.

Common mistakes

The errors repeat across teams. Reporting total synthesis time as if it were the responsiveness metric. Measuring on short clean samples instead of real replies. Reading the average and ignoring a slow tail. Testing one call at a time, then being surprised under load. Using batch synthesis for a live agent and eating a long opening silence. Choosing the fastest voice on paper without scoring how natural it sounds. Each one ships an agent that felt fine in testing and lags in production.

Measuring TTS latency with Evalgent

Evalgent is an independent platform for testing and evaluating voice agents. It measures the TTS slice the way callers experience it. Metrics express your TTFA budget as an explicit pass or fail, read at p90 and p95, so the result is a clear go or no-go rather than a gut call. Scenarios run your real reply text, including long and awkward sentences, not clean samples.

Profiles vary accent, speaking pace, and network conditions, so the tail you measure matches the tail production sees. Evaluations run the suite at concurrency, so latency is tested under load, not one call at a time. Reviews let your team replay any turn behind a slow number and hear the gap the caller heard. This same rigor underpins broader TTS evaluation for voice agents and how you evaluate voice agent vendors overall.

The result is a latency claim you can defend: the agent started speaking fast enough, at the tail, under load, on your own text. To measure your TTS latency this way, book a demo.

The bottom line

Time to first audio, not total synthesis time, governs how responsive a voice agent feels. Stream the audio, measure TTFA at the tail under load, set a target for your use case, and gate on it.

Frequently asked questions

What is time to first audio in TTS?

Time to first audio, or TTFA, is how long a text-to-speech engine takes to produce the first playable sound after receiving text. For voice agents it is the metric that governs perceived responsiveness. It differs from total synthesis time, which measures how long the full reply takes to render. Callers feel TTFA, not total time.

Why does TTFA matter more than total synthesis time?

Because callers judge an agent by the silence after they stop talking. If the voice starts quickly, the agent feels alert. Total synthesis time scales with reply length, but the later audio streams while the caller listens. What they felt was the gap before speech began. That gap is TTFA, so it is the number to optimize.

What is a good TTS latency target for voice agents?

A common starting target is TTFA under roughly 300 milliseconds at p90, with under 150 milliseconds feeling crisp and over 500 feeling sluggish. Treat these as adaptable defaults, not fixed rules. A reception or sales agent needs the tightest number, while an outbound reminder flow can tolerate more. Set the target to the cost of a slow start in your use case.

What is the difference between streaming and non-streaming TTS?

Non-streaming, or batch, synthesis renders the entire reply before any audio plays, so time to first audio scales with reply length. Streaming synthesis emits audio in chunks as it generates, so the first chunk plays while later ones form. Streaming keeps TTFA low and roughly constant. For live conversational agents, streaming is almost always the right default.

How do you measure TTS latency at percentiles?

Isolate the TTS stage, feed realistic reply text, and record time to first audio for many calls. Report the 90th and 95th percentiles rather than the average, because averages hide the slow calls callers actually feel. Re-run under peak concurrency, since the tail stretches under load. For streaming, also track the gaps between audio chunks to catch mid-reply stalls.

Does lower TTS latency reduce speech quality?

Sometimes. Engines that start speaking soonest can sound flatter, while warmer voices may take longer to begin. There is a real trade-off between latency and naturalness, often scored with a mean opinion score. The right balance is use-case specific. Measure both numbers on your own traffic and decide the trade-off deliberately, rather than assuming faster always wins.

Is TTS latency the same as end-to-end voice agent latency?

No. TTS latency is one slice of the full delay a caller feels. The complete gap also includes speech recognition and the model's thinking time before the reply is chosen. TTS is only the final saying-it-out-loud step. Measure it in isolation to tune the engine, but track the whole chain to understand the total silence callers experience.

Why measure TTS latency under load?

Because latency and its tail both stretch when the engine is busy. A number that looks fine at one call at a time can degrade sharply at peak concurrency. Testing under expected load exposes the slow tail real callers will hit. A TTFA target cleared on easy, single-call runs tells you little about how the agent will feel in production.