How to Evaluate a Speech-to-Speech Model: Benchmarks, Scorecard, and a Fair Bake-Off

Choosing a speech-to-speech model is a model-selection problem, not an agent-testing problem. You are deciding which audio-in, audio-out model to build on: GPT-Live, Grok Voice, Gemini Live, Nova Sonic, Qwen Omni, or an open model like Moshi. Leaderboards make that look easy. They are not.

The same model can rank first on one benchmark and near the bottom on another. A model that scores 97% on spoken reasoning can complete fewer than half of realistic customer-service tasks. This guide maps what each research benchmark actually measures, where the published numbers come from, what none of them test, and how to run a bake-off you can defend. Evalgent is an independent evaluator with no stake in which model you pick.

If you already picked a model and need to test the agent built on it, read our companion guide on testing speech-to-speech voice agents. If you are still deciding between architectures, start with cascading vs speech-to-speech.

What does it mean to evaluate a speech-to-speech model?

Speech-to-speech model evaluation: measuring an end-to-end audio model's capabilities, with prompt, tools, and audio path held constant, so you can compare models and choose one. It differs from agent testing, which checks one configured agent against its own requirements.

The distinction matters because it changes what you hold fixed. In agent testing, the model is a given and you vary scenarios. In model evaluation, you hold the scenarios, prompt, tools, audio codec, and turn-detection settings constant, and vary only the model. Anything else you vary contaminates the comparison.

It also changes the question. Agent testing asks "does this agent pass?" Model evaluation asks "how much capability does this model keep once it has to listen and speak, and where does it break?" That second question is what the research literature has studied, and it has some uncomfortable answers.

Inside the box: how speech-to-speech models work and what that predicts

You cannot inspect a speech-to-speech model's internal transcript, but you can reason from its architecture. Each design has predictable weak points. Knowing them tells you which probes to write first.

Codec tokenizers: what the model actually hears

Speech-to-speech models do not hear waveforms. A neural codec first turns audio into discrete tokens. Kyutai's Moshi paper documents its codec, Mimi, in detail, and the numbers explain a lot.

Mimi takes a 24 kHz waveform and compresses it to 12.5 frames per second. That is one frame every 80 ms. Each frame is quantized with 8 codebooks of 2,048 entries, for about 1.1 kbps. The first codebook is distilled from WavLM to carry meaning. The rest carry acoustic detail like timbre and prosody.

Do the math on sequence length. At 12.5 Hz with 8 codebooks, a model would need 100 tokens per second of audio if it flattened them. English text runs about 3 to 4 tokens per second. So an audio model reasons over a sequence roughly 25 to 30 times longer than the equivalent text. Moshi solves this with a hierarchical RQ-Transformer, but the cost shows up as reduced language capacity.

Now add a phone call. Telephony carries G.711 μ-law at 8 kHz, which holds no content above 4 kHz. Upsampling that to 24 kHz for the codec creates no new information. The tokenizer sees narrowband audio it may have seen little of in training. That is why phone audio belongs in every evaluation condition, not just a "realistic" add-on.

Multi-stream full-duplex models (Moshi and its descendants)

Moshi models two audio streams in parallel: its own speech and the user's. Our full-duplex voice agents guide covers the conversational side of this design. There are no explicit turns. On top, it predicts time-aligned text tokens before its audio tokens, a step the authors call Inner Monologue. The paper reports a theoretical latency of 160 ms and about 200 ms in practice, below the roughly 230 ms average gap in natural conversation.

The architecture predicts three failure modes:

• Weaker reasoning and knowledge. Audio training eats capacity. Moshi's MMLU fell to 49.7 from 54.3 for its text backbone, Helium, per the same paper. On Big Bench Audio, the Artificial Analysis leaderboard lists Moshi at 4%.
• Talking into pauses. With no turn controller, the model emits a frame of its own stream every 80 ms, silence included, so it decides whether to speak at that rate. Full-Duplex-Bench found end-to-end models like dGSLM and Moshi take over the turn during user pauses more often than systems with an explicit turn-state module.
• Collapse to silence. The Moshi authors note that weak models in their ablations tended to collapse to silence, and used transcript length as a quality proxy. For a buyer, long silences mid-call are a symptom worth counting.

Interleaved text-audio decoders

A second family runs one sequence that alternates text and speech tokens. GLM-4-Voice uses a 12.5 Hz single-codebook speech tokenizer at 175 bps and alternates 13 text tokens with 26 speech tokens per step. At 12.5 Hz, 26 speech tokens cover about 2.1 seconds of audio. The text plan for each chunk is committed before its audio.

This predicts different failures. Reasoning inherits more from the text backbone, so knowledge holds up better. But speech tuning can still erode text skills: VoiceBench found LLaMA-Omni dropped more than 11 points on text tasks compared with its own base model. Half-duplex designs in this family typically rely on an external voice activity detector, so turn-taking quality depends on endpointing settings as much as on the model.

Closed realtime APIs

GPT-Live, GPT-Realtime, Gemini Live, Grok Voice, and Nova Sonic do not publish their architectures. The Full-Duplex-Bench authors say as much. Treat them as black boxes and infer behavior from probes.

One documented detail changes how you score them. OpenAI's out-of-band transcription cookbook explains that input transcription runs on a separate ASR model, so the transcript can diverge from what the speech model understood. OpenAI says to treat it "as a rough guide." xAI's voice agent docs say its input transcript is cumulative and may revise earlier text. Developers also report output transcripts that do not match the spoken audio on the OpenAI community forum.

The practical rule: never grade a speech-to-speech model from the vendor's transcript. Transcribe the output audio yourself with a fixed ASR, and grade outcomes from tool calls and system state. Our guide to audio versus transcript evaluation covers the method.

The research benchmark landscape: what each one actually measures

There are now more than a dozen public benchmarks for spoken language models. They measure very different things. Reading a leaderboard without knowing the benchmark behind it is how teams pick the wrong model.

Headline findings a buyer can use

• Pipelines beat open end-to-end models on spoken instructions. VoiceBench found simple Whisper-plus-LLM pipelines outperformed all open-source end-to-end models by more than 20 points. One end-to-end model, VITA, scored more than 35 points lower on spoken instructions than on the same instructions in text.
• Speaking rate breaks models. VoiceBench reports the end-to-end models it tested degrade significantly below 0.5x or above 1.5x speed, while the pipeline held steady from 0.25x to 2.0x. Fast talkers and slow, elderly callers are both in your traffic.
• Fast models interrupt more. Full-Duplex-Bench found dGSLM and Moshi respond in about 0.3 seconds but take over the turn often. Freeze-Omni, with an explicit turn-state classifier, interrupted less but responded more slowly. Gemini Live (gemini-2.0-flash-live-001 at the time) took the turn only 65.5% of the time when it should have on real dialogue data.
• Instruction following erodes. Speech-IFEval found most speech-aware models struggle with basic constraints that text models handle easily, and are highly sensitive to prompt wording. URO-Bench reports open models lag their backbone LLMs on instruction following.
• Memory is weaker in voice. ContextDialog found speech-based models recall past utterances worse than text models, especially facts the user said aloud, even with retrieval.

The speech-mode tax: the same model in text and voice

The most useful number you can compute is the gap between a model's text capability and its voice capability on the same tasks. Two studies measured it directly.

Artificial Analysis introduced Big Bench Audio with a four-way test of GPT-4o. In text-to-text mode it scored 92%. With text in and speech out it scored 74%. Fully speech-to-speech, it scored 66%. Both the listening and the speaking side cost accuracy. A Whisper, GPT-4o, and TTS pipeline showed minimal degradation from text.

The tau-Voice paper from Sierra and Princeton ran 278 grounded customer-service tasks. GPT-5 in text completed 85%. Voice agents from Google, OpenAI, and xAI completed 31 to 51% in clean conditions and 26 to 38% under realistic noise, accents, and turn-taking. That is 30 to 45% of text capability retained. The models tested were gpt-realtime-1.5, gemini-live-2.5-flash-native-audio, and grok-voice-agent, so newer releases will differ. The method is what transfers.

The ablation on the retail domain is where it gets practical:

Source: tau-Voice, Table 7. Three lessons follow. First, the worst factor differs by provider: accents cost xAI 18 points while Google lost 1, and turn-taking cost Google 11. Second, factors do not add up neatly, so you cannot test them one at a time and sum. Third, the highest clean score had the largest relative drop under realistic conditions, 37% for OpenAI. A strong clean demo predicts little.

The failure analysis is just as useful. Of the failures reviewed, 79 to 90% came from the agent, not the simulator. Authentication was the dominant bottleneck: agents failed to capture names and emails even when callers spelled them letter by letter. Agents also claimed to have updated an address without making the tool call.

So build your evaluation around a control. Run every scenario through the same vendor's best text model with the same prompt and tools. The difference is the speech-mode tax for your workload. If a model's voice score is 40 points below its own text score, the problem is the audio path, not your prompt.

Why speech-to-speech leaderboards disagree

Artificial Analysis now publishes four components per model: Big Bench Audio reasoning, a Full-Duplex-Bench subset for conversational dynamics, tau-Voice for agentic performance, and a human-preference arena. Put them side by side and the rankings scatter.

As of October 2, 2026, Gemini 3.8 Live scores 92% on spoken reasoning and 96.1% on conversational dynamics, but 30.1% on tau-Voice. Qwen Audio 3.0 Realtime Plus scores 99% on reasoning and 54.6% on tau-Voice. Grok Voice Think Fast 2.0 High scores 97% and 56.5%. GPT-Live-1 (Sol, low) scores 89% on reasoning, lower than several rivals, yet 59.3% on tau-Voice, the highest of the five.

Two task-success numbers can disagree too. Gemini 3.8 Live shows a 93.2% task success rate in the arena, where people complete short scenarios such as booking a dental appointment, against 30.1% on tau-Voice's multi-step, policy-bound tasks. The model did not change. The task distribution did. Your calls look like one of those distributions, and you need to know which.

Three more reasons to read leaderboards with care:

• Saturation. Top models score 97 to 100% on Big Bench Audio. A benchmark where everyone scores near the ceiling cannot separate them.
• Noise. A pass rate's 95% confidence interval is about 1.96 × √(p(1−p)/n). At p = 0.5 and n = 278 tasks, that is ±5.9 points. Artificial Analysis notes some tau-Voice results, including GPT-Live-1 (Sol, low), come from a single trial. A 2.8-point gap between GPT-Live-1 (59.3%) and Grok Voice 2.0 (56.5%) is inside the noise.
• Configuration. Grok Voice's `reasoning.effort` defaults to `"high"`, and leaderboards list GPT-Realtime-2 at several effort levels with Big Bench Audio scores from 72% to 97%. A model name without its settings is not a result.

What benchmarks miss: the dimensions a buyer must test

Public benchmarks are a screening tool. These are the dimensions they skip or underweight, each with the evidence that it matters.

Telephony audio and codec path

Only tau-Voice applies G.711 μ-law at 8 kHz, and it does so in every condition. Most benchmarks use wideband synthetic audio. If your calls arrive over SIP or PSTN, every test must too. Check what the API accepts natively: xAI's API takes `audio/pcmu` and `audio/pcma` at 8 kHz directly, which avoids a resampling step in your own code. Our telephony guide covers the leg itself.

Domain entities and spelled input

tau-Voice's authentication failures generalize. Policy numbers, member IDs, email addresses, drug names, and street names are where speech models fail and where errors cost the most. Score exact match on every entity, not word error rate. Our entity accuracy guide explains why WER hides these.

Tool calling under speech

The model must hear the right value, put it in the right argument, and call the tool before claiming success. Count three outcomes separately: wrong tool, wrong argument, and hallucinated completion, where the model says it acted but no call exists. A gotcha from xAI's docs: tool calls arrive with `response.done` after the audio deltas, and sending `response.create` before playback finishes causes overlapping audio. Your harness has to respect that or it will log false barge-in failures. See tool call accuracy.

Long-session instruction decay

Benchmarks are mostly single-turn or short. Real calls run 20 to 60 turns. ContextDialog and Speech-IFEval both point to weaker memory and fragile instruction following in speech mode. Measure rule adherence in turns 1 to 5 and again after turn 20, and report the delta.

Interruption handling and selectivity

tau-Voice measured four interaction scores under realistic conditions. OpenAI's agent (gpt-realtime-1.5) yielded and responded quickly, but its selectivity was 6%: it reacted to nearly every backchannel, cough, and "hold on." xAI's grok-voice-agent had the best selectivity, 57%, but interrupted callers at a rate of 84%, close to once per turn. No provider led on both. Your scorecard needs separate metrics for yielding, not interrupting, and ignoring noise. Our turn-taking evaluation guide defines them.

Voice and persona consistency

Codec models generate timbre from acoustic tokens, so voice identity can drift across a long call or after an interruption. Benchmarks do not measure this. Compare speaker embeddings of the agent's output across turns against a reference clip, and flag any turn below your similarity floor. Also check for accent or language switches after the caller code-switches.

Safety in audio

Text red-teaming does not cover audio. Best-of-N Jailbreaking applied random speed, pitch, noise, volume, and music augmentations to spoken harmful requests. With 7,200 samples it reached 71% attack success on GPT-4o Realtime, 71% on Gemini Flash, and 59% on Gemini Pro. On Gemini Pro, audio inputs gave about 10% higher attack success than text. Attack success grew as a power law in the number of attempts, so a patient attacker on a phone line matters. Our red-team audit guide covers how to scope it.

A speech-to-speech model evaluation scorecard

This scorecard turns the evidence above into weights and thresholds. The thresholds are suggested starting points, not industry standards. Tighten them for regulated or high-value calls.

Score each dimension 0 to 100 against its threshold, multiply by the weight, and sum. A model that fails any gate is out, whatever its total. Weighted totals hide catastrophic failures, and gates stop that.

Two rules keep it honest. Report confidence intervals with every rate. And report the text-mode control next to each voice score so reviewers can see how much capability the audio path costs.

How to run a fair speech-to-speech model bake-off

A bake-off is fair when the only thing that changes between runs is the model. Everything below exists to enforce that. Our voice agent POC and bake-off guide covers the commercial side; this is the technical protocol.

1. Pin every model version. Use versioned names, never aliases. xAI's docs note that `grok-voice-latest` points to the newest model and recommend pinning a versioned name such as `grok-voice-think-fast-2.0` in production. Record reasoning effort and voice for each run.

2. Fix the audio path. Send every model the same caller audio in the same codec and rate. For phone traffic, that means μ-law at 8 kHz end to end. Stream in real time, in about 100 ms chunks, because server-side voice activity detection depends on timing.

3. Equalize turn detection. Set VAD threshold, silence duration, and prefix padding explicitly for every vendor. Defaults differ: xAI's threshold defaults to 0.85 and prefix padding to 333 ms. Unequal endpointing will look like a latency or interruption difference between models.

4. Use one prompt and one tool schema. Port the prompt once, review it with each vendor's prompting guide, then freeze it. Keep a text-mode control run with the same prompt and tools.

5. Build a test matrix. Cross N scenarios with M audio conditions. A workable start is 60 scenarios across happy paths, entity-heavy flows, policy edge cases, and adversarial calls, times 4 conditions: clean phone, noise at 10 dB SNR, accented speakers, and interruption-heavy callers. That is 240 calls per model per trial.

6. Run at least three trials. Speech models are stochastic. Report pass@1 and pass^3, the share of scenarios that pass all three times, a metric tau-bench introduced to measure reliability.

7. Grade from ground truth. Score tasks by system end state and tool logs. Transcribe output audio with one fixed ASR for every model. Use human review on a sample to calibrate any LLM judge.

8. Compute statistics and cost. Calculate confidence intervals, check whether differences are significant, and compute cost per resolved call before declaring a winner.

A minimal harness for one scenario (simplified)

This sketch runs one pre-recorded caller turn against xAI's realtime endpoint over μ-law audio, measures turn gap at the client, flags audio that starts before the caller finishes, and records tool calls. Event and field names follow xAI's voice agent documentation. The same loop works against OpenAI-compatible realtime endpoints with a different URL, model name, and session fields, so verify each vendor's docs.

# Simplified, illustrative harness. Prepare audio first:
#   ffmpeg -i caller.wav -ar 8000 -ac 1 -f mulaw caller.ulaw
import asyncio, base64, json, os, time
import websockets

MODEL = "grok-voice-think-fast-2.0"   # pin a version, never "latest"
URL = f"wss://api.x.ai/v1/realtime?model={MODEL}"
CHUNK = 800                            # 100 ms of 8 kHz mu-law (1 byte/sample)
SILENCE = b"\xff" * CHUNK              # mu-law zero level

SESSION = {
    "type": "session.update",
    "session": {
        "instructions": open("prompt.txt").read(),
        "voice": "eve",
        "reasoning": {"effort": "high"},
        "audio": {
            "input": {"format": {"type": "audio/pcmu"}},
            "output": {"format": {"type": "audio/pcmu"}},
        },
        "turn_detection": {           # set explicitly, same for every vendor
            "type": "server_vad",
            "threshold": 0.6,
            "silence_duration_ms": 500,
            "prefix_padding_ms": 300,
        },
        "tools": json.load(open("tools.json")),
    },
}

async def send_audio(ws, data):
    await ws.send(json.dumps({"type": "input_audio_buffer.append",
                              "audio": base64.b64encode(data).decode()}))

async def sender(ws, audio, marks):
    for i in range(0, len(audio), CHUNK):     # pace in real time
        await send_audio(ws, audio[i:i + CHUNK])
        await asyncio.sleep(0.1)
    marks["caller_done"] = time.monotonic()
    for _ in range(30):                        # 3 s trailing silence for VAD
        await send_audio(ws, SILENCE)
        await asyncio.sleep(0.1)

async def receiver(ws, result, marks):
    async for raw in ws:
        msg = json.loads(raw)
        kind = msg["type"]
        if kind == "response.output_audio.delta" and result["turn_gap_s"] is None:
            if "caller_done" not in marks:
                result["talked_over_caller"] = True
            else:
                result["turn_gap_s"] = time.monotonic() - marks["caller_done"]
        elif kind == "response.function_call_arguments.done":
            result["tool_calls"].append(
                {"name": msg["name"], "args": json.loads(msg["arguments"])})
        elif kind == "response.done" and "caller_done" in marks:
            return

async def run_turn(path):
    audio = open(path, "rb").read()
    result = {"tool_calls": [], "turn_gap_s": None, "talked_over_caller": False}
    marks = {}
    headers = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps(SESSION))
        await asyncio.wait_for(asyncio.gather(
            sender(ws, audio, marks), receiver(ws, result, marks)), timeout=60)
    return result

print(asyncio.run(run_turn("caller.ulaw")))

Measured this way, the turn gap includes the server's silence window, which is what the caller experiences. Subtract nothing. A production harness also saves output audio, drives multi-turn scripts, injects interruptions, and executes tool calls against a sandbox. Our guide to synthetic callers covers that layer.

Sample size: how many calls you need

To detect a difference between two pass rates with independent samples, a standard approximation is:

n per model = (z₁₋α/₂ + z₁₋β)² × (p₁(1−p₁) + p₂(1−p₂)) / (p₁ − p₂)²

For 95% confidence (1.96) and 80% power (0.84), detecting 70% versus 60% needs (2.80)² × (0.21 + 0.24) / 0.01 = 7.84 × 0.45 / 0.01 ≈ 353 scenario-runs per model. Detecting a 5-point gap takes about four times as many.

Two ways to need fewer. Run the same scenarios on every model and use a paired test such as McNemar's, which only counts scenarios where the models disagree. And spend your sample on the conditions that matter for your traffic, not on clean audio where every model passes.

Reliability math: pass^k

If a model passes a scenario 80% of the time and trials are independent, the chance it passes three runs in a row is 0.8³ = 0.51. At 95% per run, it is 0.86. Callers do not get to retry your agent three times, but your operations team lives with the variance. pass^3 exposes models that look fine on average and fail unpredictably.

Cost per resolved call

List price per minute is not the number that matters. Use cost per call including the fallback path:

cost per call = (price per minute × minutes) + (1 − task success) × cost of a human-handled call

Worked example with illustrative assumptions: a 4-minute call and a $6 fully loaded cost for a human-handled call. Model A lists at $0.05 per minute and Model B at $0.08 per minute, the current list prices of GPT-Live-1 and Grok Voice. Suppose your bake-off measured 70% success for A and 80% for B.

The more expensive model per minute is cheaper per call once failures are priced in. The model price is rarely the cost driver; resolution rate is. Our cost per resolution guide extends this to containment and repeat calls.

Latency budget at the caller's ear (illustrative)

Leaderboard time-to-first-audio is measured on Big Bench Audio question files, not live phone turns. Callers experience the full turn gap. With a 500 ms silence window, a model time-to-first-audio of 0.7 to 1.5 seconds (the range Artificial Analysis lists for the five models above), and an assumed 100 to 200 ms for the telephony leg and jitter buffer, the gap lands between about 1.3 and 2.2 seconds. Natural conversation averages about 230 ms. The silence window is the biggest lever you control, and shortening it raises interruptions. See time to first audio.

Gotchas that skew bake-offs

• Faster-than-real-time streaming. Sending a file in one burst breaks server VAD timing and inflates or hides latency.
• No trailing silence. If you stop sending audio after the caller finishes, server VAD may never see the end of speech.
• Sample-rate mislabels. Sending 8 kHz audio declared as 24 kHz produces fast, high-pitched speech the model will misunderstand. Check each session's format before you trust any score.
• Alias drift. A vendor updates "latest" mid-bake-off and half your runs use a different model.
• Grading from vendor transcripts. Separate ASR models can disagree with what the speech model heard.
• Session resets. Full-Duplex-Bench restarted the Gemini Live session after each reply. That is fine for a turn-taking probe and wrong for long-session adherence.
• Python 3.13 removed `audioop`. Old conversion scripts fail on import. Convert audio with ffmpeg ahead of time.

Reading the results and deciding

Start with the gates. Drop any model that fails one. Then compare weighted totals, but only treat differences larger than the confidence interval as real.

Next, look at the shape of failures, not just the score. A model that loses points on accents is a problem if 30% of your callers are non-native speakers and a non-issue if they are not. A model with poor selectivity hurts most in noisy environments like cars and warehouses. Map each failure cluster to your traffic mix.

Finally, decide whether speech-to-speech is the right architecture at all. If the best voice score sits 30 or more points below the same vendor's text control on your tasks, a cascaded pipeline may serve your use case better today. VoiceBench and Big Bench Audio both found pipelines lose little capability from text. Our real-time voice API evaluation guide covers the API-level trade-offs, and the Grok Voice agent review and GPT-Live testing guide cover two current options in detail.

Where independent evaluation helps

Vendors publish the benchmarks they win. Leaderboards use their own task mix and settings. Neither tells you how a model handles your callers, your entities, your tools, and your phone path.

Evalgent runs speech-to-speech bake-offs as an independent evaluator. We hold the prompt, tools, codec, and turn detection constant, run every model through the same scenario-by-condition matrix, and score outcomes from audio and system state with confidence intervals. After you choose, the same suite becomes your regression test for every model version the vendor ships, which is where speech models change the most. If you already run a golden dataset, we can start from it.

Frequently asked questions

How do you evaluate a speech-to-speech model?

Run it on your own scenarios over your real audio path, usually 8 kHz phone audio, with prompt, tools, and turn detection held constant. Score task success from system state, entity exact match, tool-call correctness, turn-taking, turn gap, long-session adherence, voice consistency, and audio safety. Compare each score with the same vendor's text model as a control.

Which benchmark is best for comparing speech-to-speech models?

No single one. Big Bench Audio measures spoken reasoning and is near saturation. Full-Duplex-Bench measures turn-taking. tau-Voice measures grounded tasks with tools under realistic audio, which is closest to production. Use them to shortlist, then run your own bake-off, because rankings change across benchmarks.

How much worse are voice models than text models?

On Big Bench Audio, GPT-4o fell from 92% in text to 66% speech-to-speech. On tau-Voice, voice agents completed 31 to 51% of tasks in clean audio and 26 to 38% in realistic conditions, against 85% for GPT-5 in text. The gap varies by model and task, so measure it on yours.

How does Grok's speech-to-speech model compare on benchmarks?

On the Artificial Analysis leaderboard as of October 2, 2026, Grok Voice Think Fast 2.0 High scores 97% on Big Bench Audio, 95.1% on conversational dynamics, and 56.5% on tau-Voice, with 0.70 seconds to first audio. In the tau-Voice paper, the earlier Grok voice agent lost 18 points with accented callers in the retail domain. Test accents explicitly.

Can I trust the realtime API's transcript to grade a model?

No. OpenAI documents that input transcription runs on a separate ASR model and can diverge from what the speech model understood. Developers also report output transcripts that differ from spoken audio. Transcribe the output audio yourself with one fixed ASR for all models and grade outcomes from tool calls and system state.

How many test calls do I need for a fair bake-off?

To detect a 10-point difference in task success, such as 70% versus 60%, at 95% confidence and 80% power, you need about 350 scenario-runs per model. A 5-point difference needs about four times that. Paired designs, where every model runs the same scenarios, reduce the count.

Why does the same model score differently on different leaderboards?

Each benchmark uses a different task distribution, audio, and scoring. Gemini 3.8 Live shows 93.2% task success on simple arena tasks but 30.1% on tau-Voice's multi-step policy tasks. Reasoning effort settings and single-trial runs add more variance. Treat a model name without its settings and confidence interval as incomplete.

Should I test speech-to-speech models for audio jailbreaks?

Yes. Best-of-N Jailbreaking reached 71% attack success on GPT-4o Realtime by adding random speed, pitch, noise, and music changes to spoken requests over 7,200 attempts. Gemini Pro was easier to jailbreak with audio than text. Run an augmented spoken attack suite as a hard gate before launch.

The bottom line

Public benchmarks tell you which speech-to-speech models deserve a shortlist, but they disagree with each other and skip phone audio, entities, tools, long calls, and audio safety. Pick the model by running a pinned, equalized bake-off on your own calls with a text-mode control, a gated scorecard, and confidence intervals.