How to Evaluate a Speech-to-Speech Model

# How to evaluate a speech-to-speech model

Quick answer

> Quick answer: To evaluate a speech-to-speech model, judge it on audio, not transcripts. Run your own realistic calls and score task success, turn-taking, latency, naturalness, and policy adherence by ear. Word error rate alone misses most of the quality, because the model never exposes a text layer to inspect.

Speech-to-speech models change how you have to evaluate a voice agent. The old playbook leaned on transcripts and word error rate. A speech-to-speech model gives you neither. Audio goes in, audio comes out, and the middle is a black box. So you have to score what the caller actually hears.

This guide covers how to evaluate a single speech-to-speech model well. It is not an architecture debate. Our cascading vs speech-to-speech comparison handles that, and our guide to testing speech-to-speech agents covers testing in general. Here the focus is narrow: the dimensions of quality, and the method for scoring one end-to-end model on your own calls.

Evalgent is an independent evaluator. We score a speech-to-speech model on your calls, by ear and on task success, without favoring any provider.

What is a speech-to-speech model?

> Speech-to-speech model: a single end-to-end model that takes audio in and emits audio out directly, with no separate transcription or synthesis stage in the middle.

A classic cascading agent chains three models. Speech-to-text turns audio into words. A language model reasons. Text-to-speech turns words back into audio. Each stage leaves a transcript you can inspect.

A speech-to-speech model collapses that pipeline. It is an end-to-end model, often trained with multimodal learning so that audio, meaning, and voice share one representation. There is no intermediate text. That is why these models feel fast and natural. It is also why they are hard to measure.

Why word error rate does not capture speech-to-speech quality

Word error rate compares a transcript to a reference. It is a good metric for a speech recognition stage. A speech-to-speech model has no such stage to score.

There is no internal transcript to grade. You can transcribe the audio with a separate model, but that is your transcript, not the model's. It tells you what a second system heard. It does not tell you what the speech-to-speech model understood.

Even a perfect transcript would miss most of the experience. Word error rate cannot hear a flat, robotic tone. It cannot tell that the agent talked over the caller. It cannot measure a two-second pause before every answer. Those failures are audible, not textual. This is the core reason we argue for audio-based evaluation over transcript-only scoring.

So word error rate becomes one weak signal among many. The real question is whether the agent did the right thing, sounded right, and got out of the caller's way.

The dimensions that decide speech-to-speech quality

Quality for a speech-to-speech model spreads across several dimensions. No single number captures it. The table below maps each dimension to how you measure it and what a good result looks like.

Read each dimension as a separate score. A model can be fast and natural yet drift off policy. Another can follow rules yet sound cold. You want the full picture before you trust it in production. Our voice agent evaluation pillar breaks these axes down further.

How to evaluate a speech-to-speech model

Here is a repeatable method. It works for any single end-to-end model, whichever provider built it.

1. Define the job. Write down what the agent must do. List the tasks, the policies, and the caller goals that count as success.

2. Build real scenarios. Use your own calls, not generic demos. Include easy paths, edge cases, and the hard names and numbers your callers actually say.

3. Run the calls on audio. Place live or synthetic calls into the model. Capture the caller audio and the agent audio for every turn.

4. Score task success first. For each call, mark whether the caller's goal was met. This is the outcome that matters most.

5. Rate the audio by ear. Judge naturalness, prosody, and clarity on a fixed scale. Note any tone that would annoy or confuse a real caller.

6. Measure timing. Record the gap between the caller stopping and the agent replying. Flag long pauses and any talk-over.

7. Test turn-taking and barge-in. Interrupt the agent mid-sentence. Check that it stops cleanly and picks up your new intent.

8. Probe policy adherence. Push the agent toward off-limits actions. Confirm it refuses, escalates, or redirects as your rules require.

9. Stress the hard cases. Feed accents, background noise, spelled names, and long digit strings. Confirm the agent handles them.

10. Score per cohort and repeat. Group results by scenario type and caller profile. Rerun after any model or prompt change to catch regressions.

Run this on your own data, not the vendor's benchmark. A number produced on someone else's calls does not predict your outcomes. That is the case we make for benchmarking on your own data.

Turn-taking, interruptions, and latency

Real-time behavior is where speech-to-speech models win or lose. A model can say the right words and still feel broken if the timing is off.

Good turn-taking means the agent knows when to speak and when to listen. It should not talk over the caller. It should not leave awkward gaps. When the caller interrupts, the agent should stop at once and adapt. Our guide to barge-in versus turn-taking explains the difference in detail.

Latency is the delay between the caller finishing and the agent starting. Speech-to-speech models often shine here because there are no stage hand-offs. But you still have to measure it on your calls, under real network conditions. A demo on a fast connection hides the dead air a caller feels on a mobile line. Our latency guide for voice agents covers how to time it.

Measure timing at the audio boundary. You do not need the model's internals to know it paused too long. The caller heard the pause, and so can you.

Naturalness and prosody: why you must judge on audio

Naturalness is the reason teams reach for speech-to-speech in the first place. It is also the reason a transcript-only test lies to you.

Prosody is the rhythm, stress, and intonation of speech. It carries emotion, emphasis, and intent. A speech-to-speech model can preserve prosody that a text pipeline flattens. It can also get it wrong in ways no transcript will show.

Score naturalness by ear on a fixed scale. The mean opinion score is a well-established way to rate perceived audio quality from 1 to 5. Use a consistent rubric so different raters agree. Note tone that sounds flat, rushed, or oddly cheerful for the context.

The point is simple. Only a human ear, or a rubric built to mimic one, can judge whether the voice sounds right. This is why we insist on listening to the audio, not just reading the words.

Instruction and policy adherence without a text layer

Policy adherence is harder to verify when there is no transcript to inspect. You cannot open the model and read what rule it applied. You can only observe what it did.

So test behavior directly. Write scenarios that push the agent toward the wrong action. Ask it to skip a required disclosure. Ask it to share something it should not. Then grade whether it held the line.

You can still use a strong model as a grader. Research on using a language model as a judge shows this can approximate human judgment on open-ended behavior. Feed the grader the call audio, or a transcript of it, plus your policy. Let it flag violations for a human to confirm.

The honest signal here is action, not intention. Grade what the agent said and did on the call. That is what the caller experienced, and what a regulator would review.

Robustness on names, numbers, and noise

Speech-to-speech models can stumble on the hard, boring details. Names, digits, dates, and spellings are where callers lose trust fast.

Build a battery of hard inputs. Include uncommon names, long account numbers, and dates spoken in different formats. Add background noise and a range of accents. Speak them the way real callers do, at real speed.

Then check the output against ground truth. Did the agent repeat the account number correctly? Did it capture the name it was spelled? Did noise cause it to mishear a key figure? These checks matter more in regulated and high-stakes calls, where one wrong digit is a failed call.

Observability: why speech-to-speech is harder to debug

Observability is the weakest point of a speech-to-speech model. When a cascading agent fails, you can read the stage transcripts and see where it broke. A speech-to-speech model gives you no such trail.

You cannot see why it did something. You can only see what it did. So build your evaluation to reconstruct as much as possible from the outside. Keep the full call audio. Keep the timing of every turn. Keep the caller input and the agent output side by side.

This external record is your debugging surface. It will not tell you the model's internal reasoning. It will tell you the pattern of failures, which cohorts fail, and which conditions trigger them. That is enough to decide whether to ship, and enough to hold a provider accountable.

When to run your own speech-to-speech evaluation

Not every team needs the same depth. Match the effort to the stakes.

Run a full evaluation when the calls carry money, health, or legal risk. A collections, healthcare, or claims line needs every dimension scored, per cohort, before launch. A low-stakes information line can start lighter and add rigor over time.

Run it again on every change. A new model version, a new prompt, or a new voice can shift behavior in ways a spot check misses. Independent, repeatable scoring is how you catch that. Our guide to evaluating voice agent vendors shows how to structure the process across providers.

If you are choosing between speech-to-speech providers, run the same scenarios through each one. Score them side by side, on your data, with one rubric. That is the only fair comparison, and it is what an independent evaluator like Evalgent exists to run.

Frequently asked questions

How do you evaluate a speech-to-speech model?

Evaluate a speech-to-speech model on audio, not transcripts. Run your own realistic calls, then score task success, turn-taking, latency, naturalness, and policy adherence. Rate the audio by ear on a fixed scale, measure timing at the boundary, and repeat per cohort after every model change. Judge what the caller experienced, not the model's internals.

Can you measure word error rate on a speech-to-speech model?

Not directly. A speech-to-speech model exposes no internal transcript to compare against a reference, so word error rate does not apply the same way. You can transcribe the audio with a separate model and reason about accuracy, but that is an external check. The honest signal is whether the agent acted correctly, not what a second system transcribed.

Why is a speech-to-speech model harder to evaluate than a cascading one?

A cascading agent produces a transcript at each stage, so you can see where a call broke. A speech-to-speech model removes the middle entirely. Audio goes in and audio comes out through one model, with no text to inspect. You lose per-stage metrics and must judge behavior from the outside, using outcomes, audio, and timing.

What metrics matter most for a speech-to-speech model?

Task success matters most: did the caller reach their goal. After that, score turn-taking and interruption handling, latency, naturalness and prosody, policy adherence, and robustness on names and numbers. Track each separately, because a model can be fast and natural yet drift off policy. No single number captures speech-to-speech quality on its own.

Do you need to judge a speech-to-speech model on audio?

Yes. Prosody, tone, pauses, and talk-over are audible failures that no transcript reveals. A word-perfect transcript can still hide a flat voice, a two-second delay, or an agent that interrupts the caller. Rate the audio by ear on a consistent rubric, such as a mean opinion score, so different raters agree on what good sounds like.

How do you test turn-taking in a speech-to-speech model?

Interrupt the agent mid-sentence during a call and watch how it reacts. It should stop cleanly, drop its current response, and pick up your new intent without confusion. Listen for overlap, false starts, and awkward gaps across many turns. Score barge-in handling and timing at the audio boundary, since you cannot see the model's internal turn logic.

How do you evaluate policy adherence without a transcript?

Test behavior directly with scenarios that push the agent toward the wrong action. Ask it to skip a disclosure or share restricted information, then grade whether it held the line. A strong model can act as a first-pass grader on the call audio, flagging likely violations. A human confirms them. Grade the action, not the model's unseen reasoning.

Should you evaluate a speech-to-speech model on your own calls?

Yes. A score produced on a vendor's benchmark does not predict your outcomes, because your callers, accents, and edge cases differ. Run your own scenarios through the model, capture the audio, and score per cohort. Rerun the same calls through each provider you are comparing. Independent scoring on your data is the only fair test.

The bottom line

A speech-to-speech model must be judged on audio and on task success, not on transcripts or word error rate. Run your own calls, score every quality dimension, and repeat after each change.

Ready to see how a speech-to-speech model scores on your calls? Book a demo and get an independent evaluation on your own data.