How to Evaluate a Voice Model's Turn-Taking

# How to evaluate a voice model's turn-taking

Quick answer

> Quick answer: Voice model turn-taking evaluation measures how well a realtime model manages the flow of a spoken conversation. You score response latency after the caller stops, false interruptions, missed barge-in, awkward gaps, and double-talk. Judge it on real, messy call audio, not scripts, using concrete metrics like turn latency and interruption precision.

A realtime voice model can say the right words and still feel broken. The words are fine. The timing is wrong. It talks over the caller, or it leaves a long silence, or it keeps going after the caller has clearly cut in. Callers hang up on that agent even when every sentence is correct.

Turn-taking is the part of quality that transcripts never show. This post is about measuring it. Not defining barge-in, and not evaluating a whole voice API. Our barge-in versus turn-taking guide defines the concepts, and our post on evaluating a realtime voice API covers the full surface. Here the focus is narrow: the failure modes, the metrics, and the method for scoring one realtime model's conversational timing.

Evalgent is an independent evaluator. We score turn-taking on your own call audio, by ear and by measured timing, without favoring any provider.

What turn-taking means for a voice model

> Turn-taking: the system by which speakers exchange the conversational floor, deciding who talks, when, and for how long. A voice model does this by predicting when the caller has finished and when it should speak, listen, or yield.

Human conversation runs on tight timing. The study of it, conversation analysis, shows that people swap turns with gaps often under a fifth of a second. The classic account of how this works is Sacks, Schegloff, and Jefferson's model of turn-taking organization. People rarely collide, and when they do, they repair it fast.

A voice model has to fake that skill in real time. It has to detect when the caller stops, decide whether the caller is truly done or just pausing, and start speaking without stepping on them. It also has to stop when interrupted. Get any of this wrong and the call feels off, no matter how good the answer is.

Why turn-taking must be judged on audio and messy calls

Turn-taking lives in the audio. It does not live in the transcript. A transcript of a bad call and a good call can read identically. One agent waited two seconds and cut the caller off twice; the other flowed. The words are the same. Only the audio tells them apart. This is the core case we make for audio-based evaluation over transcript-only scoring.

Scripted demos hide the problem. In a demo, the tester speaks in clean, complete sentences and waits politely. Real callers do not. They pause mid-sentence to think. They say "um" and "hold on." They talk over the agent. They give short backchannels like "mm-hmm" while the agent is still talking. A model tuned on tidy speech falls apart on any of that.

So you have to test on messy input. Use real calls, or synthetic calls built to mimic real ones. Include mid-sentence pauses, filler words, hesitations, interruptions, and overlapping speech. The point is to trigger the failures a demo never surfaces.

The turn-taking failure modes callers actually notice

Turn-taking breaks in a handful of recognizable ways. Each one sounds specific, and each one is measurable. The table below maps every failure mode to what the caller hears and how you put a number on it.

Read each row as a separate score. A model can be fast yet interrupt constantly. Another can never interrupt yet leave dead air on every turn. You want the full profile before you trust it live.

The metrics that turn a vibe into a score

"Feels natural" is not a metric. To hold a provider accountable, you need numbers anyone can reproduce. These are the ones that matter most.

Turn latency. This is the gap between the caller finishing and the agent starting to speak. Measure it at the audio boundary, from the caller's end-of-speech to the agent's speech onset. Latency here is what the caller experiences as dead air. Report the median and a high percentile, because a good median with a bad tail still frustrates callers. Measure it under real network conditions, since a fast demo connection hides the lag on a mobile line.

Interruption precision and recall. Treat interruptions as a detection problem. Precision is the share of the agent's interruptions that were actually warranted. Recall, for barge-in, is the share of genuine caller interruptions the agent correctly yielded to. These map cleanly to the standard definitions of precision and recall. A model that never interrupts has perfect precision and useless recall. You want both high.

Endpointing accuracy. Endpointing is the decision that the caller has finished a turn. Cut too early and you talk over a thinking pause. Wait too long and every reply feels sluggish. Our endpointing guide covers the mechanics. Score both the false-endpoint rate and the late-endpoint delay, since the two trade off against each other.

Overlap and double-talk. Some overlap is normal and even good, like a quick backchannel. Sustained overlap where neither party yields is the failure. Measure total seconds of sustained simultaneous speech per call. This is where voice activity detection on both channels earns its keep, since you need to know who was speaking when.

Backchannel handling. A backchannel is a short listener signal like "mm-hmm" or "right" that means "keep going," not "stop." A good model ignores it and holds the floor. A bad one stops or restarts. Score the false-yield rate: turns where a backchannel wrongly made the agent stop.

How to run a turn-taking evaluation

Here is a repeatable method. It works for any realtime or speech-to-speech model, whichever provider built it.

1. Define the conversations that matter. List the call types your callers actually make. Note where interruptions, pauses, and quick exchanges are common, since those stress turn-taking most.

2. Build a messy test set. Assemble real or synthetic calls with mid-sentence pauses, filler words, backchannels, interruptions, and overlapping speech. A clean script will pass a broken model.

3. Run the calls and capture both channels. Place the calls into the model and record caller audio and agent audio separately, with aligned timestamps. You cannot measure overlap from a single mixed track.

4. Mark the event boundaries. For each turn, mark caller end-of-speech, agent speech onset, and any interruption or barge-in. Automate this with voice activity detection, then spot-check by ear.

5. Compute turn latency. For every turn, measure the gap from caller end-of-speech to agent onset. Report median and a high percentile per call type.

6. Score interruptions. Label each agent interruption as warranted or not, and each caller barge-in as yielded or not. Compute interruption precision and barge-in recall.

7. Measure stop latency on barge-in. When the caller cuts in, time how long until the agent goes silent. A slow stop is still a failure even if it eventually stops.

8. Count gaps and overlap. Flag every silent gap over your threshold with no acknowledgment. Sum sustained double-talk seconds per call.

9. Check backchannel handling. Inject backchannels like "mm-hmm" while the agent speaks. Confirm it holds the floor and does not falsely yield.

10. Score per cohort and repeat. Group results by call type and caller profile. Rerun the whole set after any model or prompt change to catch regressions.

Run this on your own data, not a vendor benchmark. Turn-taking is sensitive to accent, pacing, and call type, so a number from someone else's calls will not predict yours. That is the case we make for benchmarking on your own data.

Building a turn-taking test set

Your test set is the whole game. A weak set makes a weak model look fine.

Start from real calls if you have them. Pull recordings where callers hesitated, interrupted, or spoke over the agent. These are the hard cases, and they are where turn-taking earns or loses trust. Anonymize as your policy requires, then label the timing events.

Where you lack real audio, build synthetic calls that behave like real ones. Script a caller who pauses mid-sentence, says "um, hold on," and cuts the agent off partway through a sentence. Add backchannels during the agent's turns. Vary accent, pace, and line quality. The goal is not clean coverage; it is realistic stress.

Cover the specific patterns that break models. Include a caller who thinks out loud with long pauses, to probe premature endpointing. Include a caller who interrupts to correct the agent, to probe barge-in recall. Include a caller who says "mm-hmm" while listening, to probe backchannel confusion. Include a fast back-and-forth, to probe double-talk. Each pattern targets one row of the failure table.

Keep the set fixed so results are comparable over time. When you add cases, version the set. A turn-taking score only means something if the same calls produced it.

Where Evalgent fits

Most teams cannot run this in-house at any scale. Capturing dual-channel audio, marking event boundaries, and scoring interruption precision across hundreds of calls is real work. Doing it the same way every time, so the numbers stay comparable, is harder still.

Evalgent runs it as an independent third party. We take your call types, build a messy test set that mirrors your real traffic, and score every turn-taking dimension on audio. You get turn latency distributions, interruption precision and recall, endpointing accuracy, and overlap time, broken down by cohort. Because we favor no provider, the score is one you can put in front of a vendor or a regulator. Our voice agent evaluation pillar shows how turn-taking fits the wider picture, and our independent evaluation post explains why third-party scoring matters for high-stakes calls.

When you compare providers, run the same test set through each one. Score them side by side, on your data, with one rubric. That is the only fair comparison of conversational timing.

Frequently asked questions

What is voice model turn-taking evaluation?

Voice model turn-taking evaluation measures how well a realtime model manages conversational flow. It scores how fast the model responds after a caller stops, whether it interrupts, whether it yields when interrupted, and how it handles pauses, backchannels, and overlapping speech. The evaluation runs on real call audio, because timing failures never appear in a transcript.

How do you measure a voice model's response latency?

Measure turn latency at the audio boundary. Record the caller and agent on separate channels with aligned timestamps, then measure the gap from the caller's end-of-speech to the agent's speech onset. Report the median and a high percentile, since a good median with a bad tail still frustrates callers. Always test under realistic network conditions, not a fast demo line.

What is a false interruption in a voice agent?

A false interruption is when the agent starts talking while the caller is still mid-sentence, usually because it decided the turn had ended too early. It sounds like the agent talking over the caller. You measure it with interruption precision, the share of the agent's interruptions that were actually warranted, and with the false-endpoint rate.

How do you test barge-in on a realtime voice model?

Have the caller cut in while the agent is speaking, across many turns, and check two things. First, does the agent stop at all: measure barge-in recall, the share of genuine interruptions it yields to. Second, how fast does it stop: measure stop latency from the caller's barge-in to the agent going silent. A slow stop is still a failure.

Why can't you evaluate turn-taking from a transcript alone?

Turn-taking lives in timing, and a transcript strips timing out. A smooth call and a call full of talk-over and dead air can produce identical text. Only the audio reveals a two-second delay, an interruption, or an awkward gap. That is why turn-taking must be scored on the recorded audio of both speakers, not on the words alone.

How does a voice model handle backchannels like "mm-hmm"?

A backchannel is a short listener signal such as "mm-hmm" or "right" that means "keep going," not "stop talking." A good model recognizes it and holds the floor. A poorly tuned model treats it as an interruption and stops or restarts. Test this by injecting backchannels during the agent's turns and scoring the false-yield rate.

What metrics matter most for turn-taking quality?

Turn latency matters most, since dead air is what callers feel first. After that, score interruption precision and barge-in recall, endpointing accuracy, sustained overlap or double-talk time, and backchannel false-yield rate. Track each separately, because a model can be fast yet interrupt constantly, or polite yet slow. No single number captures conversational timing.

Should you evaluate turn-taking on scripted or real calls?

On real, messy calls. Scripted callers speak in clean sentences and wait politely, which hides the failures that break live conversations. Real callers pause, hesitate, interrupt, and speak over the agent. Build a test set with those patterns, or use anonymized real recordings, and run the same set through every model you compare on your own data.

The bottom line

A voice model's turn-taking must be scored on real call audio, using concrete metrics like turn latency, interruption precision, and barge-in recall. Transcripts and scripted demos hide the timing failures that make callers hang up.

Want to see how your voice model handles real, messy calls? Book a demo and get an independent turn-taking evaluation on your own audio.