Deepgram STT: latency, diarization, and long-call stability

Three questions come up again and again before a team commits to Deepgram for speech-to-text: how fast is it, how well does it tell speakers apart, and does it hold up on long calls. These are the right questions, because latency, diarization, and stability are exactly where a real-time voice agent lives or dies. This guide answers each with what Deepgram documents, and then covers the part that matters most — how to verify the answers on your own audio, which is what Evalgent is built for. It is a companion to our broader Deepgram STT testing guide, focused on these three specifics rather than the full method.

How fast is Deepgram STT?

Latency is the first question for anything in a voice pipeline, and Deepgram's streaming speech-to-text is designed for it. Per Deepgram, the Nova-3 model delivers sub-300-millisecond streaming latency, which is well inside the range a voice agent needs to feel responsive.

The number to understand is not a single figure but a sum. As the measuring streaming latency docs explain, total transcript latency is network transit time plus transcription latency plus your own client processing. That matters because two of those three are yours to control: where you run relative to Deepgram's endpoint, and how efficiently your client handles results. Streaming also emits interim results before final ones, so your agent can start reacting to a partial transcript rather than waiting for the finalized text. For a voice agent, speech-to-text latency is one link in a longer chain, which our reduce latency guide puts in context alongside the LLM and text-to-speech.

How many speakers can Deepgram distinguish?

The second question is about diarization — telling who said what. Deepgram's diarization recognizes speaker changes and assigns a speaker label to each word in the transcript automatically, with no manual tagging and no need to declare the number of speakers up front.

That "no fixed count" design is the practical answer: you do not tell Deepgram to expect two speakers or five, it labels them as it hears distinct voices. For a typical voice agent call — one caller and the agent, or a caller and a human on a warm transfer — this is straightforward. Accuracy is a separate matter from capability, though. Diarization gets harder with overlapping speech, very similar voices, and narrowband telephony audio that strips the acoustic detail speaker separation relies on. So the capability is broad, but the accuracy on your calls depends on your conditions, which is why measuring it beats assuming it.

How stable is Deepgram on long calls?

The third question is stability over long-duration streaming. A voice agent call can run for many minutes, and a streaming connection has to survive silences, pauses, and network hiccups without dropping.

Deepgram streams over a WebSocket, and the key detail for long calls is keeping that connection healthy. Per Deepgram, you can send periodic KeepAlive messages to hold the connection open during stretches of silence — though KeepAlive alone will not keep it open forever; the connection needs real audio too. Just as important is handling the failures that do happen: the connection error and timeout recovery docs cover reconnecting cleanly when a stream drops mid-call. Stability, in other words, is partly the service and partly your client's handling of silence and reconnection. A long call that survives a network blip without losing the transcript is an engineering property you build and then verify, not a spec you can read off a page.

STT alone vs the Voice Agent API

One clarification worth making: these questions apply whether you use Deepgram's speech-to-text on its own or inside its Voice Agent API. If you run STT standalone — feeding transcripts to your own LLM and text-to-speech — latency, diarization, and stability are properties of that speech-to-text stream directly. If you use the Voice Agent API, the same speech-to-text sits inside the orchestrated loop, so its latency contributes to the end-to-end turn and its stability is part of the whole connection's stability.

Either way, the verification approach is the same: measure the behavior on your own audio, under your conditions. The difference is only where in your stack the number lands, not whether you should test it.

The three answers are not independent

It is tempting to treat latency, diarization, and stability as separate checkboxes, but on a real call they pull on each other. Enabling extra processing like diarization or advanced formatting can add to transcription latency, so the fast number you saw with a bare configuration is not the number you get with everything turned on. A long, multi-speaker call stresses all three at once: latency has to stay low while diarization tracks who is speaking across many minutes and the connection stays healthy through pauses.

That interaction is exactly why a single-metric spec check is misleading. A configuration that is fast but drops diarization accuracy, or one that diarizes well but adds latency, is a real trade-off you only see when you measure the three together on the same calls. Decide which features you actually need, enable exactly those, and measure the resulting latency, accuracy, and stability as one system — because that combined behavior, not any single documented number, is what your callers experience.

Why you should verify these on your own audio

Here is the thread running through all three answers: the documented numbers are real, but they describe capability, not your calls. Latency depends on where you run and how noisy your audio is. Diarization accuracy depends on your speakers and your line quality. Stability depends on your network and your reconnection logic. A vendor spec is a starting point; your production audio is the test.

This is the same principle as our Deepgram STT testing guide: a benchmark measured on clean, read speech tells you little about a noisy phone call with an accented speaker. The responsible move is to measure latency, diarization accuracy, and long-call stability under your own conditions before you rely on them, the same discipline our STT evaluation guide applies to accuracy.

What to measure

To verify Deepgram's performance for your use case, measure these under realistic conditions.

Measure the tail, not just the average. A median latency inside 300 milliseconds with a bad p95 still means a slice of calls feel slow. And measure per condition, because a clean-audio number hides the noisy, accented, or telephony calls where latency, diarization, and word accuracy all degrade together — a point our word error rate guide makes for accuracy specifically.

Verifying Deepgram STT with Evalgent

Evalgent measures these properties the way your callers will experience them. Scenarios drive realistic calls — including long ones, multi-speaker moments, and the noisy, accented audio that stresses diarization and accuracy. Profiles vary caller voice, accent, and line quality, so latency and diarization are reported per condition rather than on clean audio alone. Metrics record word emission latency at percentiles, diarization accuracy, and long-call stability with thresholds you set. Evaluations run the checks as automated batches before release, so you know the numbers hold before production does. Reviews let you replay a long or noisy call and see exactly where latency spiked or a speaker was mislabeled.

The result is confidence grounded in your own audio: not "Deepgram is fast and stable in general," but "Deepgram is fast, accurate, and stable enough on the calls we actually get." For the full method, see the Deepgram STT testing guide and the AI voice agent testing pillar.

Conclusion

Deepgram's streaming speech-to-text is built for real-time voice agents: sub-300-millisecond latency, automatic per-word diarization, and long-call stability with KeepAlive and reconnection. Those are capabilities you can rely on, once you have confirmed them on your own audio.

Read the spec, then run the test. The numbers that matter are not the ones on the docs page but the ones your production calls produce, measured per condition and at the tail. A capability confirmed on your own audio is one you can build on; a capability read off a page is only a hypothesis until you do.

Frequently asked questions

What is the typical latency of Deepgram's real-time speech recognition?

Per Deepgram, the Nova-3 model delivers sub-300-millisecond streaming latency, which is well within what a real-time voice agent needs. Total transcript latency is network transit plus transcription plus your client processing, so two of the three factors are yours to control. Streaming also emits interim results before finals, letting your agent react to partial transcripts sooner.

How many speakers can Deepgram's STT distinguish?

Deepgram's diarization assigns a speaker label to each word automatically and recognizes speaker changes without you declaring a speaker count up front. For a typical caller-and-agent call it is straightforward. Accuracy, as opposed to capability, depends on your conditions — overlapping speech, very similar voices, and narrowband telephony audio all make speaker separation harder, so measure it on your own calls.

How stable is Deepgram on long-duration voice calls?

Deepgram streams over a WebSocket that can run for long calls, but stability is partly yours. Send periodic KeepAlive messages to hold the connection open during silence — though KeepAlive alone will not keep it open without real audio — and add reconnection so a dropped stream resumes cleanly. A long call surviving a blip without losing the transcript is something you build, then verify.

Does Deepgram support speaker diarization?

Yes. Deepgram's diarization recognizes when the speaker changes and labels each word with a speaker automatically, so you do not need to tag speakers manually or specify how many there are. It is well suited to voice agent calls. As with any diarization, accuracy varies with overlap, voice similarity, and audio quality, so verify it on audio that resembles your real traffic.

How do you keep a Deepgram connection alive?

Send periodic KeepAlive messages to keep the WebSocket open during stretches of silence, which effectively pauses the connection without closing it. Note that KeepAlive alone will not prevent closure indefinitely — the connection also needs real audio. For long calls, pair KeepAlive with reconnection handling so that if the stream does drop, your client re-establishes it and continues transcription without losing the call.

How do you measure Deepgram STT latency?

Measure word emission latency — the delay from a word being spoken to its transcript appearing — at percentiles like p50 and p95, not just the average. Remember total latency is network transit plus transcription plus client processing, so test from the region you deploy in, under realistic load and audio. The gap between p50 and p95 tells you how consistent the latency is under real conditions.

Is Deepgram fast enough for voice agents?

For most voice agents, yes. Sub-300-millisecond streaming latency leaves comfortable room within the sub-second end-to-end budget a natural conversation needs, once you add the LLM and text-to-speech. The caveat is that speech-to-text is one link in the chain, and your network, audio quality, and client handling affect the real number, so confirm the end-to-end latency on your own calls.

How accurate is Deepgram diarization?

Deepgram diarizes automatically and labels speakers per word, but accuracy depends on the audio. Clean, separated speech diarizes well; overlapping speakers, very similar voices, and narrowband telephony audio make it harder, because speaker separation relies on acoustic detail that noise and compression strip away. Because accuracy is condition-dependent, the honest way to know is to measure it on your own calls.