Voice Model Turn-Taking Evaluation: Metrics, Test Matrix, and FTO Code

A voice agent can say every word correctly and still feel broken. It talks over a caller who paused to find a card number. It waits a second and a half after "yes." It stops dead on "mm-hm," then restarts the sentence. Transcripts show none of this.

Most guides stop at "measure latency and interruptions." That is not enough to decide anything. You need precise definitions, a human reference point, a test set that triggers the failures, and a way to read the speed vs. cut-in trade-off. This post gives you all four, with numbers from research papers and open benchmarks.

What turn-taking evaluation measures

Turn-taking is the system speakers use to decide who talks, when, and for how long. A voice agent faces four decisions over and over. It can take the floor, hold it, yield it, or give a short backchannel without taking it. Every turn-taking failure is one of those decisions made at the wrong moment.

Research models frame it the same way. The Voice Activity Projection work from KTH labels mutual silences as SHIFT (the other person speaks next) or HOLD (the same person continues), and separates short backchannels from real turn starts. Full-Duplex-Bench, a 2025 benchmark for full-duplex spoken dialogue models, tests four behaviors: pause handling, backchanneling, smooth turn-taking, and user interruption. If your evaluation does not cover each of those, it has a gap.

A turn-taking score asks whether the agent spoke and stopped at the right time. It does not ask whether the answer was correct. Keep those scores separate, or a fast, rude agent and a slow, accurate one average out to the same number. For the broader picture, see our guide to evaluating a realtime voice API, and for why timing needs audio rather than text, see transcript vs audio evaluation.

What the research says about human timing

You need a reference point before you can call a gap "slow." Three findings matter most.

The 200 ms gap is close to universal

Stivers et al. (2009, PNAS) measured response timing to questions in 10 languages, from major world languages to small indigenous communities. The mean response came just over 200 ms after the question ended. Japanese was fastest at about 7 ms on average and Danish slowest at about 469 ms. Every language showed a single-peaked distribution with its mode between 0 and 200 ms.

Two things follow. First, no cascaded agent hits a 200 ms gap by reacting to silence. Second, the spread between languages is real. One endpointing threshold tuned on English callers will feel rushed to some callers and sluggish to others. Report results per language.

People plan their reply while the other person is still talking

Levinson and Torreira (2015) point out the puzzle: gaps are about 200 ms, but producing even a single word takes over 600 ms. People can only hit 200 ms by predicting the end of the other person's turn and preparing the answer in overlap. That is exactly what modern turn detectors and speculative generation try to copy.

The same paper includes a new analysis of 348 Switchboard telephone conversations, about 38 hours of speech. The numbers are useful benchmarks for an evaluator:

• 77% of the signal was one speaker alone, 19.2% silence, and only 3.8% simultaneous speech.
• 30.1% of floor transfers started in overlap (a negative FTO), not after a gap.
• Those overlaps were short. The modal overlap was 96 ms, the median 205 ms, and 75% were under 374 ms.
• Speech was segmented into inter-pausal units (IPUs) using silences of 180 ms or more. That is a sensible bridging value for your own segmentation.

The paper also cites Heldner (2011), who found that gaps or overlaps under about 120 ms are not perceived as gaps or overlaps at all.

The practical lesson: "zero overlap" is the wrong target. A short overlap at a turn boundary is normal human behavior. What callers notice is long overlap where the agent runs over a sentence that is still going.

Pauses inside a turn are longer than gaps between turns

Skantze's 2021 review of turn-taking in conversational systems makes the point that breaks silence-based endpointing: pauses within a turn are, on average, longer than gaps between turns. A system that waits for silence cannot tell "I'm thinking" from "I'm done." It must either wait long, which feels sluggish, or fire early, which cuts people off. The review notes that classic dialogue systems used silence thresholds around 700 to 1,000 ms for this reason, against a human gap of about 200 ms.

Why silence thresholds fail: inside an endpointer

To evaluate turn-taking, you need to know which component made each decision. Here is the path every caller turn takes in a cascaded stack.

1. Audio frames arrive. Telephony usually delivers G.711 mu-law at 8 kHz in 20 ms packets. Your stack may resample to 16 kHz for the VAD and turn model.

2. VAD marks speech and silence. It reports a speech end only after a minimum silence. Silero VAD's default `min_silence_duration` in LiveKit is 0.55 s.

3. A turn model scores the pause. It decides whether the caller is done, often from audio alone.

4. An endpointing policy commits. It combines the model score with a minimum wait and a maximum timeout.

5. The LLM, TTS, and transport run. Only then does the caller hear a reply.

Each production framework exposes these knobs differently, and the defaults matter more than most teams realize.

Sources: LiveKit turn handling options, LiveKit turn detector, Pipecat Smart Turn, Deepgram Flux end-of-turn parameters.

The last column is the one nobody benchmarks. When the turn model misses a true end of turn, the caller waits until the timeout. A 2% miss rate with a 5-second backstop produces a handful of five-second silences per hundred calls. Those are the calls that end with "Hello? Are you there?" Your evaluation should count missed turns separately and report how long each one waited.

What the research models look at

Two research lines shaped today's production detectors.

TurnGPT (Ekstedt and Skantze, Findings of EMNLP 2020) fine-tuned GPT-2 to predict, after each word, the probability that a turn shift comes next. It captures syntactic and pragmatic completeness from text. Its weakness is the same as any text-only detector: it cannot hear a rising intonation or a drawn-out "and..." LiveKit's own docs now mark its text-based turn detector as deprecated in favor of an audio model. The text model's published accuracy table shows why: for English it caught 99.3% of true turn ends but only 87% of the pauses where the caller would continue.

Voice Activity Projection (Ekstedt and Skantze, Interspeech 2022) predicts future voice activity for both speakers over a 2-second window. The window has four bins per speaker of 200, 400, 600, and 800 ms, which gives 256 possible future states. It is trained on Switchboard without labels. Its SHIFT vs HOLD weighted F1 was 0.899, against 0.843 for always guessing the majority class. The F1 for the SHIFT label alone was only 0.510. Even a strong research model finds "the other person is about to speak" much harder than "the same person will continue." Expect your production detector to be weakest in the same place.

A worked latency budget

Here is an illustrative budget for one floor transfer in a cascaded agent. Only the turn-model figure is sourced; the rest are assumptions to show the arithmetic.

The endpointing wait is the largest single block you control. Speculative generation overlaps the LLM with that wait. Deepgram's docs say eager end of turn can trim 100 to 200 ms of end-to-end latency, at the cost of 50 to 70% more LLM calls (Flux eager end of turn). LiveKit's `preemptive_generation` does something similar and, by default, runs only the LLM early, not the TTS. Either way, measure the result at the audio, not from framework timers. For hop-level measurement, see our post on time to first audio.

The metrics, defined precisely

Vague metric names cause most arguments about turn-taking. "Latency" can mean five different things. Use these definitions, all computed from a stereo recording with caller and agent on separate channels.

Inter-pausal unit (IPU). A stretch of speech on one channel, with silences shorter than 180 ms bridged. Everything else is built from IPUs.

Floor transfer offset (FTO). For each transfer from caller to agent: FTO = agent IPU onset minus the end of the caller's last IPU. Positive values are gaps. Negative values are overlaps. Report the distribution (p10, p50, p90), not a mean. A mean hides the long tail that callers remember.

Overlap share. The fraction of transfers with negative FTO. Compare it with the 30.1% Switchboard figure. A share near zero is not automatically good. It can mean the agent always waits too long.

False cut-in rate. False cut-ins divided by all agent floor-takes. A false cut-in is either (a) the agent starts while the caller's IPU still has more than 300 ms left, or (b) the agent starts in a caller pause and the caller resumes a real utterance within 1 s. Type (b) is the more common failure and the one silence thresholds cause. Full-Duplex-Bench calls the pause version the Takeover Rate (TOR) on pause-handling items.

Missed-turn rate. True caller turn ends where the agent did not start speaking within a deadline, divided by all true turn ends. Use 2.5 s as a starting deadline. Also report the wait on each miss, because it is set by your timeout.

Barge-in stop time. For each caller interruption longer than 1 s: agent IPU offset minus caller IPU onset, measured on the caller's side of the call. Report p50 and p90, plus the missed barge-in rate (agent still talking 2 s after the caller started).

Backchannel false-yield rate. Caller backchannels during agent speech that made the agent stop, divided by all such backchannels. Full-Duplex-Bench treats a segment as a backchannel when it is under 1 s and under two words. That is a usable operational rule. Also report stall time: how long the agent stayed silent before resuming.

Interruption recovery. After a barge-in, does the model's context contain only what the caller heard? Score it with a probe question ("What was the last thing you told me?") and a binary pass or fail.

Starting thresholds

These are judgment calls for a cascaded phone agent, not research results. Treat them as a starting bar and adjust for your use case.

What Full-Duplex-Bench shows about speed vs. politeness

Full-Duplex-Bench (arXiv 2503.04721) fed scripted and real dialogue audio to four systems and scored them automatically. Its setup is worth copying. Pause-handling items use turns with an internal pause of 0.4 to 1 s. Smooth turn-taking items come from the Candor corpus, with 5 s of silence appended so the model has time to respond. Interruption items play the interrupting speech 7 s after the preceding user utterance. The results, from the paper's Table III:

Read it as a trade-off, not a ranking. The two end-to-end models answered in 0.27 to 0.35 s, close to human timing. They also took over during 93% to 99% of the caller's mid-turn pauses. Gemini Live was the most patient during pauses but took about 1.3 s to answer. It also failed to take the turn in about 35% of real dialogue transitions, which is a missed-turn problem. Freeze-Omni, a cascaded design with an explicit listening and speaking state predictor, gave the best replies after interruptions but responded slowly.

Two lessons carry over. Fast and polite pull against each other, so a single latency number is incomplete. And an explicit turn-state module handled pauses far better than implicit turn-taking. These are early-2025 model versions, so treat the results as evidence about design patterns, not as a ranking of current products.

The latency vs. false cut-in trade-off curve

Every endpointer has a dial. Turn it toward speed and false cut-ins rise. Turn it toward certainty and dead air rises. The useful question is what latency you pay for a given cut-in rate.

LiveKit's open eot-bench harness answers that directly. It annotates every silence of at least 100 ms in real human-to-agent turns. The last silence is the true end of the turn and every earlier one is a hold. It then sweeps three policy knobs: confidence threshold, minimum action delay, and timeout. The README reports these English operating points:

LiveKit publishes this benchmark, and its own model leads it, so run the harness on your data before you treat the ranking as settled. Three findings hold whatever the ranking:

• Silence alone is not viable at conversational speed. Tuned to answer in 300 ms, the VAD baseline cut off more than half of the turns with mid-turn pauses. It needed 1.6 s of waiting to get to 5%.
• Learned detectors move the whole curve, not one point. That is why you should compare stacks at a fixed cut-off budget, not at their default settings.
• "Latency" in this chart is policy wait, not compute. A model that infers in 10 ms but waits 600 ms to be sure still costs 600 ms of dead air.

You can build the same curve for your own stack. Run your test set at three or four settings of the main dial (LiveKit `min_delay` and `unlikely_threshold`, Pipecat VAD `stop_secs`, Flux `eot_threshold`) and plot false cut-in rate against FTO p50. Pick the operating point from the curve, not from a default. Our endpointing comparison covers the stack-level choice.

Measuring FTO from stereo recordings

All of the metrics above come from one input: a two-channel recording, caller on one channel and agent on the other, time-aligned. Get the recording right first.

Record at the edge closest to the caller. Twilio stores call recordings as dual-channel by default, one leg per channel, and the `RecordingChannels` parameter controls it (Twilio Recordings resource). Do not build the agent channel from the TTS output on your server. Server-side audio does not include the audio still queued at the telephony edge, so it makes barge-in stop time look better than what the caller hears.

Verify channel mapping on one test call. Speak first, stay silent, and check which channel lights up. Getting this wrong swaps every metric's sign.

Never mix and re-split. Once the channels are mixed, overlap is unrecoverable.

The script below is simplified but runnable. It uses py-webrtcvad to find speech, bridges 180 ms silences into IPUs, and classifies each event with the rules from the metrics section.

"""fto.py: turn-taking events from a stereo call recording (simplified).
Left channel = caller, right channel = agent. Sample rate must be 8, 16, 32 or 48 kHz.
pip install webrtcvad soundfile numpy
"""
import sys
import numpy as np
import soundfile as sf
import webrtcvad

FRAME_MS = 20          # webrtcvad accepts 10, 20 or 30 ms frames
BRIDGE_MS = 180        # silences shorter than this stay inside one IPU
MIN_IPU_MS = 100       # drop clicks and blips
BACKCHANNEL_MS = 1000  # caller IPU shorter than this = backchannel candidate
OVERLAP_OK_MS = 300    # agent may start this early before caller's IPU ends
RESUME_MS = 1000       # caller resumes this soon after agent onset = premature take
STOP_LIMIT_MS = 2000   # agent still talking this long after barge-in = missed
YIELD_MS = 600         # agent went quiet this soon after a backchannel = wrongly yielded

def ipus(x, sr, aggressiveness=2):
    vad = webrtcvad.Vad(aggressiveness)
    n = sr * FRAME_MS // 1000
    pcm = (np.clip(x, -1.0, 1.0) * 32767).astype(np.int16)
    flags = [vad.is_speech(pcm[i:i + n].tobytes(), sr)
             for i in range(0, len(pcm) - n + 1, n)]
    segs, start = [], None
    for i, f in enumerate(flags + [False]):
        t = i * FRAME_MS
        if f and start is None:
            start = t
        elif not f and start is not None:
            segs.append([start, t]); start = None
    merged = []
    for s, e in segs:
        if merged and s - merged[-1][1] < BRIDGE_MS:
            merged[-1][1] = e
        else:
            merged.append([s, e])
    return [(s, e) for s, e in merged if e - s >= MIN_IPU_MS]

def events(caller, agent):
    out = []
    for a_on, a_off in agent:
        live = [c for c in caller if c[0] < a_on < c[1]]
        if live:  # agent started while caller was talking
            c_off = live[0][1]
            if c_off - a_on <= OVERLAP_OK_MS:
                out.append(("fto", a_on, a_on - c_off))      # negative FTO
            else:
                out.append(("cut_in", a_on, c_off - a_on))   # ms of caller speech run over
            continue
        prev_c = max((c[1] for c in caller if c[1] <= a_on), default=None)
        prev_a = max((a[1] for a in agent if a[1] <= a_on), default=-1)
        if prev_c is None or prev_a > prev_c:
            continue  # agent continuing its own turn, not a floor transfer
        resumed = [c for c in caller
                   if a_on < c[0] < min(a_off, a_on + RESUME_MS)
                   and c[1] - c[0] >= BACKCHANNEL_MS]
        kind = "premature" if resumed else "fto"
        out.append((kind, a_on, a_on - prev_c))              # positive gap
    for c_on, c_off in caller:  # caller starts while agent is talking
        talking = [a for a in agent if a[0] < c_on < a[1]]
        if not talking:
            continue
        a_off = talking[0][1]
        if c_off - c_on < BACKCHANNEL_MS:
            kind = "bc_yield" if a_off - c_on < YIELD_MS else "bc_held"
        else:
            kind = "barge_missed" if a_off - c_on > STOP_LIMIT_MS else "barge_stop"
        out.append((kind, c_on, a_off - c_on))
    return sorted(out, key=lambda r: r[1])

def summarize(evts):
    fto = np.array([v for k, _, v in evts if k == "fto"])
    stops = np.array([v for k, _, v in evts if k == "barge_stop"])
    takes = sum(k in ("fto", "cut_in", "premature") for k, _, _ in evts)
    count = lambda k: sum(e[0] == k for e in evts)
    pct = lambda a, q: float(np.percentile(a, q)) if len(a) else float("nan")
    return {
        "floor_transfers": takes,
        "fto_p50_ms": pct(fto, 50), "fto_p90_ms": pct(fto, 90),
        "overlap_share": float((fto < 0).mean()) if len(fto) else float("nan"),
        "false_cut_in_rate": (count("cut_in") + count("premature")) / max(takes, 1),
        "barge_stop_p90_ms": pct(stops, 90),
        "missed_barge_ins": count("barge_missed"),
        "backchannel_yields": count("bc_yield"),
    }

if __name__ == "__main__":
    audio, sr = sf.read(sys.argv[1], dtype="float32")
    caller, agent = ipus(audio[:, 0], sr), ipus(audio[:, 1], sr)
    evts = events(caller, agent)
    for e in evts:
        print(*e, sep="\t")
    print(summarize(evts))

Run it as `python fto.py call.wav`. It prints one line per event and a summary dictionary per call. Aggregate across calls before you compute percentiles.

How to keep the numbers honest

• Echo bleed corrupts the caller channel. On speakerphones, the agent's voice leaks into the caller's microphone. The VAD then sees "caller speech" during every agent turn and reports fake barge-ins. Check a sample of flagged barge-ins by ear, and test echo as its own condition.
• WebRTC VAD is coarse. Noisy channels produce false speech. Swap in a stronger VAD for noisy conditions, and keep the same VAD for every system you compare.
• Use labeled pauses where you can. The "premature" heuristic cannot tell a premature agent from a caller who chose to interrupt. With scripted stimuli you know where the hold pauses are, so score cut-ins against those labels.
• Use the same rules for every vendor. The constants above are arbitrary but explicit. Changing `BRIDGE_MS` from 180 to 300 changes your FTO distribution. Freeze them per evaluation.

The test matrix: utterance types by conditions

A turn-taking test set must make the agent decide at hard moments. Random real calls contain too few of them. Build the set from utterance types that each target one decision, then run each type under several audio conditions.

Utterance types

Digit strings (U3) deserve special weight. Callers read card, phone, and account numbers in chunks with long pauses, and each pause looks like a turn end. The Smart Turn project lists text conditioning, so the model "knows" a caller is reading a number, as a future goal. That tells you current audio-only detectors do not have that context. Test it explicitly. U5 is the counterweight. An agent tuned to survive U3 often gets slow on "yes."

For U9, remember the Stivers spread: mean gaps differ by hundreds of milliseconds across languages. LiveKit's audio turn detector uses per-language calibrated thresholds and supports 14 languages; Smart Turn v3.2 lists 23. Check which languages your stack covers before you test, and see our multilingual testing guide for building the stimuli.

Conditions

C1 matters because most turn models take 16 kHz input. Smart Turn's repo says it takes 16 kHz mono PCM. Phone audio upsampled from 8 kHz carries nothing above 4 kHz, so test the detector on the telephony path you ship, not on browser audio. C5 matters because Deepgram's own Flux docs warn that echo and background noise can cause false `StartOfTurn` events. For noise generation, see our guide to testing STT under background noise, and for VAD misfires specifically, testing VAD misfires.

You do not need all 54 cells at full depth. Run every utterance type under C0, C1, and C3 at 10 dB. Run U1, U3, U6, and U7 under every condition, because those are the types where audio quality changes the decision. That is 27 required cells plus 12 extended cells.

How many samples per cell

The unit of measurement is the decision point, such as a hold pause or a barge-in, not the call. To estimate a false cut-in rate near 5% within plus or minus 2 percentage points at 95% confidence:

n = 1.96² × 0.05 × 0.95 / 0.02² ≈ 457 hold pauses

To detect an improvement from 8% to 4% between two configurations at 80% power and a two-sided 5% significance level, the standard two-proportion formula gives about 553 hold pauses per configuration. Pauses within the same call are correlated, so spread them across many calls and callers rather than packing 20 into one. If you see zero cut-ins in 300 pauses, the rule of three still allows a true rate up to about 1% (3 / 300). Zero is not proof of zero.

This is why regressions slip through. A 20-call smoke test has maybe 60 hold pauses and cannot tell 4% from 8%. Our post on synthetic callers covers generating volume, and disfluency robustness covers U4-style stimuli in depth.

Barge-in and interruption recovery: where stop time hides

Barge-in looks simple: the caller talks, the agent stops. In practice the stop passes through several stages, each adding time.

The minimum-speech gate. Frameworks wait for enough speech before treating it as an interruption, so a cough does not stop the agent. LiveKit's `interruption.min_duration` defaults to 0.5 s, and `min_words` can require transcribed words too. That gate is a floor under your stop time. You cannot measure a p50 below it, and you should not try to. The question is whether the gate is right for your callers.

Server cancellation. The framework cancels LLM generation and TTS. In Deepgram's Voice Agent API, the server stops the agent's turn and emits a `UserStartedSpeaking` event.

The edge buffer. This is where most stop time hides. The TTS produces audio faster than real time, so seconds of speech may already sit in the telephony provider's buffer. In Deepgram's Twilio guide, your bridge must send a Twilio `clear` message on `UserStartedSpeaking`, or the queued audio keeps playing (Deepgram Twilio Voice Agent guide). Server logs will show a fast stop. The caller will hear the agent keep talking. This is why stop time must come from the caller-side recording.

False-interruption recovery. Some stacks pause on any detected speech and resume if no transcript follows. LiveKit's `false_interruption_timeout` defaults to 2.0 s with `resume_false_interruption` on. A backchannel that triggers this path produces a stall of up to two seconds, then the agent resumes. Count those stalls in the backchannel metric. LiveKit's `adaptive` interruption mode, a context-aware barge-in model, is designed to reduce them. Measure whether it does on your U6 stimuli.

Context truncation is a correctness test, not a timing test

After a barge-in, the model's conversation history should hold only what the caller heard. OpenAI's Realtime API documents this directly. The client sends `conversation.item.truncate` with `audio_end_ms` so the server drops the unheard part of the transcript (OpenAI Realtime conversations guide). Developers have reported a trap in public bug threads. If `audio_end_ms` comes from wall-clock time since playback started rather than from audio actually played, it overshoots after any playback stall. The model then believes it said things the caller never heard.

You can test this without reading code. Have the agent read a two-part answer, such as a balance and a due date. Interrupt after the first part, then ask, "What was the last thing you told me?" A correct agent repeats only the balance. A broken one claims it gave the due date. One scripted call per condition catches a bug class that timing metrics miss. If you hire anyone, Evalgent included, to audit turn-taking, ask for this probe by name.

For deeper coverage of interruption scoring, see our interruption detection evaluation guide and the full-duplex voice agents explainer.

Production gotchas that change turn-taking mid-call

These are documented behaviors that make turn-taking differ between your test and your production traffic.

• LiveKit falls back to a smaller model and stays there. If the full `v1` turn detector times out or cannot reach LiveKit Inference, the session logs one warning, emits a default probability so the turn is not blocked, and switches to `v1-mini` for the rest of the session. In eot-bench, `v1-mini` shows 27.8% false cut-offs at 300 ms against 9.9% for `v1`. A session can silently degrade halfway through. Log the fallback and slice your metrics by detector version.
• Burstable instances cause timeouts. LiveKit recommends compute-optimized instances over burstable ones such as AWS t3 for `v1-mini`, to avoid inference timeouts from CPU credit limits.
• No STT means English thresholds. LiveKit's audio detector picks per-language thresholds from the language the STT reports. With a realtime model and no STT, it defaults to English thresholds. A Spanish call on that setup is endpointed with English calibration.
• Delays can stack. In LiveKit's STT turn-detection mode, `min_delay` is applied after the STT provider's own endpointing delay, not instead of it. Two "300 ms" settings can become 600 ms.
• Dynamic endpointing learns per session. LiveKit's `dynamic` mode adapts the delay with an exponential moving average of the session's pauses (`alpha` defaults to 0.9). Early turns behave differently from late turns. Report metrics by turn index, or a short test call will not match a long real one.
• Two turn detectors fight. With LiveKit's detector and a realtime model, disable the model's built-in turn detection, as the docs instruct.
• Speculative generation needs cancellation tests. Flux `TurnResumed` and LiveKit preemptive generation both cancel drafts when the caller keeps talking. A missed cancellation answers half a sentence. U1 and U2 catch it.
• Smart Turn sees 8 seconds. It analyzes up to the most recent 8 s of the caller's turn. Long, rambling turns are truncated from the start, so test long U4 turns as well as short ones.

Log these per call. Our list of what to log on every voice agent call covers the turn events and config fields you need.

How to run a voice model turn-taking evaluation

1. Freeze the configuration. Record the stack version, turn detector and version, every endpointing and interruption parameter, VAD settings, and the LLM and TTS models. A score without a config is not reproducible.

2. Set up caller-side stereo recording. Use dual-channel recording at the telephony edge. Verify channel mapping on one test call.

3. Build the stimulus set from the matrix. Script U1 to U9 with known pause positions and lengths. Generate them with varied voices, or record real people for U4, U6, and U9, where synthetic speech is least natural.

4. Apply the conditions. Render each stimulus through C0 to C5 with fixed seeds for noise and packet loss, so reruns are comparable.

5. Size each cell. Aim for about 450 decision points per cell where you need a tight estimate. Spread them across many calls and caller voices.

6. Run the calls through the real path. Use the same telephony, codec, and region as production. Browser tests miss C1 and C4.

7. Segment and classify. Run `fto.py` or your own segmenter with frozen constants. Score cut-ins against the known pause labels wherever stimuli are scripted.

8. Run the truncation probe. Interrupt a two-part answer and ask what was said last. Record pass or fail per condition.

9. Sweep one dial to draw your curve. Repeat U1, U3, and U5 at three or four endpointing settings. Plot false cut-in rate against FTO p50 and choose the operating point.

10. Report by cohort, then rerun on every change. Break results down by utterance type, condition, language, and turn index. Rerun the full set after any model, prompt, framework, or telephony change, since turn-taking regresses silently.

Run this on your own audio. A benchmark built on someone else's callers will not predict how your callers pause, which is the core argument in our post on benchmarking on your own data.

Where independent evaluation fits

Framework dashboards measure from the server. That is fine for debugging and wrong for comparing vendors. The numbers that matter are FTO, cut-ins, and stop time at the caller's ear, computed the same way for every stack.

Evalgent runs this protocol as an independent third party: pre-launch audits, bake-offs where several vendors run the same stimuli through the same telephony path, and regression runs after upgrades. We report the trade-off curve and per-cohort results, not one blended score. If you are planning a vendor comparison, our voice agent POC bake-off guide shows how to structure it. Stack-specific test plans are in the LiveKit testing guide and the Pipecat testing guide, and pause handling evaluation goes deeper on hold decisions.

Frequently asked questions

What is voice model turn-taking evaluation?

It measures when a voice agent takes, holds, and yields the conversational floor. You record caller and agent on separate channels, then compute floor transfer offset, false cut-in rate, missed-turn rate, barge-in stop time, backchannel false-yield rate, and whether context was truncated correctly after interruptions. It scores timing, separately from whether the answer was right.

What is floor transfer offset (FTO)?

FTO is the time from the end of one speaker's speech to the start of the next speaker's speech at a turn change. Positive values are gaps; negative values are overlaps. Human conversation centers around 200 ms across languages, and about 30% of telephone transfers start in a brief overlap. Report FTO as a distribution with p50 and p90.

How do you test VAD, interruption handling, and barge-in for a voice agent?

Script stimuli with known pauses, backchannels, and interruptions, then run them through the production telephony path with dual-channel recording. Measure false cut-ins on hold pauses, backchannel false-yields, and barge-in stop time from the caller-side channel. Add echo and noise conditions, since both trigger false speech detection. Finish with a context truncation probe after each interruption.

What is a good false cut-in rate?

A reasonable starting bar is 5% or less of agent floor-takes, which matches the budget used by the open eot-bench harness. Measure it on hard stimuli such as mid-sentence pauses, lists, and digit strings, not on clean complete sentences. Always report it alongside FTO, because any system can lower cut-ins by waiting longer.

How does Deepgram's Voice Agent API handle barge-in?

When the caller starts speaking over the agent, Deepgram stops the agent's turn and sends a `UserStartedSpeaking` event. Your telephony bridge must then clear any audio still buffered at the edge. In Deepgram's Twilio guide, that means sending a Twilio `clear` message. Without it, queued agent audio keeps playing to the caller.

How do you test voice agents with interruptions, background noise, slow speakers, and accents?

Build a matrix. Rows are utterance types such as mid-sentence pauses, digit strings, fillers, backchannels, and barge-in corrections. Columns are conditions such as 8 kHz telephony, noise at 20, 10, and 5 dB SNR, packet loss, and echo. Include slow speakers and multiple accents and languages in the voices, then score each cell separately.

Why is latency alone a misleading turn-taking metric?

Because latency and false cut-ins trade off. Any endpointer can respond faster by accepting more interruptions, or interrupt less by waiting longer. In Full-Duplex-Bench, models that answered in about 0.27 s also took over during most mid-turn pauses. Compare systems at a fixed cut-in budget, or plot the full latency versus cut-in curve.

How many test calls do you need for turn-taking evaluation?

Count decision points, not calls. Estimating a 5% false cut-in rate within plus or minus 2 points takes about 457 hold pauses. Detecting a change from 8% to 4% at 80% power takes about 553 per configuration. Spread them across many calls and voices, since pauses within one call are correlated.

The bottom line

Turn-taking quality is a trade-off between speed and interruptions, so evaluate it with caller-side stereo recordings, precise metrics like FTO and false cut-in rate, and a test matrix that forces hard decisions. Pick your operating point from a measured curve on your own audio, and rerun the matrix after every change, because turn-taking regresses without warning.