Transcript vs Audio Evaluation for Voice Agents: What Each One Misses (2026)

Every voice agent call leaves two records: the transcript, a tidy list of turns, and the audio that actually crossed the phone line. Most teams grade the transcript because it is small and easy to feed to an LLM. Then a caller complains about something the transcript never showed.

The usual advice is "transcripts miss tone, so listen to some calls." This guide goes further. It lists 15 failures a transcript cannot show and how to detect each. It explains why the transcript you grade is your own recognizer's output, which hides its own mistakes. It pulls numbers from recent research on audio-LLM judges, does the cost math with current pricing, and ends with a protocol and decision table you can copy.

This piece sits under our broader voice agent evaluation guide. It answers one narrower question: which artifact should each check run on?

What a transcript actually is

A voice agent transcript feels like a recording written down. It is not. It is stitched together from two different machine outputs, and neither one is the audio.

The caller's lines come from your speech-to-text (STT) model. They are the model's best guess at what was said, formed in real time, often from 8 kHz telephone audio that was resampled to 16 kHz on the way in. If the STT heard "fifty" when the caller said "fifteen," the transcript says "fifty." Nothing in the text marks it as a guess.

The agent's lines come from the LLM's text output. They are what the agent intended to say, not what the text-to-speech (TTS) engine produced or what the caller heard. If TTS mangled a drug name, the transcript still shows it spelled right. If the caller interrupted halfway through a sentence, the transcript may still show the whole sentence.

So a transcript has three gaps built in. It holds the STT's errors as if they were facts. It holds the agent's intent as if it were delivery. And it drops time entirely: who spoke when, for how long, and whether two voices overlapped. Every failure in the next section falls into one of those three gaps.

Frameworks handle the agent side differently, and it matters for evaluation. LiveKit's documentation says that when voice and transcription are both on, the agent transcript is synchronized with playback and truncated to the spoken output when the agent is interrupted. Turn that sync off with `sync_transcription=False` and text goes out as soon as it exists. OpenAI's Realtime API asks the client to send `conversation.item.truncate` with an `audio_end_ms` value on interruption, so the server drops the unplayed portion of the transcript. Forget to send it, or log the streaming transcript deltas instead of the final item, and your logs contain words the caller never heard. Know which one your logging pipeline captures before you trust it.

The 15 failures a transcript cannot show

This is the core taxonomy. Each row names the failure, why the transcript hides it, and the cheapest reliable way to detect it. "Signal" means a deterministic measurement on the waveform. "Re-transcribe" means running an independent STT over the recording. "Audio judge" means an audio-capable LLM listening to the clip.

Transcripts do show wrong facts, policy misses, wrong tool choice, loops, missed escalation and task outcome. Text judges and rules handle those better than any audio method.

Two patterns stand out. Most audio-only failures are timing or level problems, and a voice activity detector plus arithmetic catches them on every call. The rest split into words that came out wrong, which re-transcription finds, and delivery that came out wrong, which needs an audio judge or a human. That last group is small, and it shapes the protocol below.

Interruption handling deserves its own measurement plan. Our guide to interruption detection evaluation covers the turn-level metrics in depth.

The ASR laundering problem

Here is the failure that most transcript-based evaluation misses, and it is a design flaw rather than a gap.

When you grade a call from its transcript, the caller's side of that transcript came from your own STT. Your judge sees what the system thought was said. If the STT was wrong, the judge inherits the error as ground truth. The mistake gets laundered: it enters as a recognition error and comes out looking like a correct conversation.

Walk through a concrete case. The caller says "I need fifteen milligrams." The line is noisy and the STT returns "fifty milligrams." The agent, working correctly from what it received, confirms fifty. The transcript reads: caller asks for fifty, agent confirms fifty. A text judge checks that the agent confirmed the requested dose and passes the call. The call was a failure, and every text-based check says it was perfect.

Laundering also runs the other way. The STT drops a word, the agent asks the caller to repeat, and a text judge marks the agent down for "not listening." Now you are penalizing the LLM for an STT problem, and your prompt team chases a bug that lives in the recognizer.

Two pieces of research explain why this is not rare.

Real-conversation error is higher than benchmarks suggest. In WER we are and WER we think we are (Findings of EMNLP 2020), Szymański and colleagues compared three commercial ASR systems on real call-center conversations and the HUB'05 benchmark. They found error rates significantly higher than the best published results. Systems have improved since, but clean-benchmark WER still says little about your noisy calls.

STT can also invent speech. The Careless Whisper study (FAccT 2024) found that about 1% of Whisper transcriptions contained whole phrases or sentences that did not exist in the audio, and 38% of those hallucinations included explicit harms such as false authority or violence. Hallucinations were more common for speakers with longer non-vocal pauses. Phone calls are full of pauses, hold time and line noise, which is exactly where invented words appear. A transcript-only judge cannot tell an invented caller sentence from a real one.

The fix: never let the system under test grade itself. Re-transcribe the caller channel with an independent STT, ideally a different model family, and compare. Agreement proves nothing, but disagreement on entities like numbers, names and dates is a precise flag. Our guide to STT entity accuracy covers why entity error matters more than overall WER for this.

Measuring STT accuracy without ground truth

People often ask how to evaluate transcription accuracy in Deepgram, or any STT, on production calls where nobody has written a reference transcript. You have three tiers, from cheapest to most rigorous.

1. Cross-ASR disagreement rate. Run a second STT over the recorded caller channel. Compute word-level disagreement after normalizing case, punctuation and number formatting. This is not WER, because neither side is the reference, but a rising trend is a real signal.

2. Entity disagreement rate. Pull the key slots (amounts, dates, account fragments, names) from both transcripts and compare them exactly. Any mismatch on a slot the agent acted on gets routed to a human. This is the number that predicts business damage.

3. True WER on a labeled slice. Have humans transcribe a stratified sample, say 200 utterances per condition (quiet, noisy, accented, speakerphone). Compute WER = (S + D + I) / N, where S, D and I are substitutions, deletions and insertions and N is reference word count. Use it to calibrate how much the cross-ASR disagreement rate means.

Normalization is the main trap. Deepgram's `smart_format` turns "fifteen" into "15." Your live STT may not. Without normalizing both sides the same way, every number becomes a false disagreement and the flag drowns in noise. Our Deepgram STT testing guide and the guide on how to improve WER go deeper on the reference-transcript side.

What the research says about audio LLM judges

If transcripts hide so much, why not hand every recording to an audio-capable LLM and ask it to score the call? Because the research from 2025 and 2026 shows those judges are much weaker than text judges, and the way they fail matters for how you use them.

Pointwise scoring collapses toward chance

The most useful study for builders is Do Audio Language Models Use Paralinguistic Evidence? (arXiv 2608.06718). The authors built counterfactual audit items: the transcript stays fixed while the affect, prosody, or timing of an emotional shift changes. A judge that only reads words cannot solve these, because both versions share the same words.

They tested two protocols. Pointwise: the judge hears one audio context and decides. Pairwise: the judge hears both versions and matches them. On the single-turn task with an explicit cue in the prompt, Gemini-3-Pro scored 91.0% pairwise but only 65.3% pointwise. Gemini-2.5-Pro dropped from 86.0% to 60.1%. GPT-4o scored 53.8% pointwise and 55.9% pairwise, close to the 50% chance line either way. Open models such as Qwen-2.5-Omni-7B and Phi-4-Multimodal stayed near chance in both.

On the harder multi-turn task, where the judge must find when in the conversation the caller became frustrated, pointwise accuracy for every model sat between about 47% and 55%. Gemini-3-Flash reached 79.4% pairwise with a cue, but 53.0% pointwise. The authors' conclusion is direct: contrastive success is not deployable judge reliability.

In production terms, "listen to this call and rate the agent's tone from 1 to 5" is the pointwise protocol, the one most teams try first and the one the evidence says fails for paralinguistic judgments. Comparing this call's audio against a reference clip of known quality is the form with real signal.

Judges are good at global cues, bad at local ones

ParaPairAudioBench (arXiv 2606.24648) tested 5,175 audio pairs across style, speaking rate, emphasis, age and gender. On average, LALM judges trailed humans by 32 percentage points. The detail is more useful than the average.

Gemini 2.5 Flash, the best judge tested, scored 88.9% on speaking rate against a human score of 91.0%. On emphasis, which word in the sentence carries stress, it scored 49.7% against 85.7% for humans. Style showed a lexical shortcut: with identical transcripts Gemini scored 83.8%, with different transcripts 36.6%. Humans held about 85% in both. The model was partly reading the words, not hearing the style.

Calibration failed badly. When both clips were equally good or equally bad and the right answer was "tie," Gemini's emphasis accuracy fell from 82.3% to 19.3%. GPT-4o Audio's style accuracy fell from 69.0% to 3.8%. Judges force a preference when they should abstain. Position bias, measured by swapping clip order, reached up to 29.4 points.

For voice agents this maps cleanly. Rate and pacing are global cues, so an audio judge can reasonably flag "agent speaks too fast." Wrong emphasis on a single word, a name said with the stress on the wrong syllable, or a statement that should have sounded like a question are local cues. Do not delegate those to an audio judge alone.

Prompt design moves the numbers a lot

AudioJudge (EACL 2026) studied what makes these judges work. With basic prompting, pronunciation-match accuracy was 46.0% and speaking-rate accuracy 46.9%, near random. Concatenating the audio clips and adding in-context examples raised pronunciation to 66.5%. A multi-aspect ensemble, with separate judges for lexical content, speech quality and paralinguistics, reached 0.91 Spearman correlation with human preferences at the system-ranking level. The authors also found strong verbosity and position biases.

That 0.91 is for ranking systems over many samples, not for scoring one call. It matches a third study, Audio-Aware LLMs as Judges for Speaking Styles, which found Gemini-2.5-pro's agreement with humans comparable to human-human agreement when judging spoken-language-model outputs on style tasks. Audio judges are more trustworthy for comparing two versions of your agent than for grading one call.

What to take from the research

• Use audio judges pairwise. Compare against a reference clip or the previous release, not against an abstract 1 to 5 scale.
• Swap order and run both. Count a result only when both orders agree.
• Allow and expect "no difference." Track how often the judge says there is a difference when humans say there is not.
• Split judges by dimension. One prompt per aspect (pacing, pronunciation, affect) beats one holistic prompt.
• Keep entity pronunciation and emphasis out of audio-judge-only checks. Re-transcription and humans handle those better.

The text-judge side has its own limits, covered in our piece on LLM-as-judge limits for voice AI. Those limits stack on top of the transcript gaps above.

The cost math: audio vs text evaluation

The common objection to audio evaluation is cost. With current pricing, that objection is mostly out of date. The real constraint is judge reliability, covered above.

How audio is tokenized

Audio LLMs bill audio as tokens, and the rates differ by vendor.

• Google states that Gemini uses 32 tokens per second of audio, or 1,920 tokens per minute. Audio is downsampled and multichannel audio is combined into one channel, which matters later.
• OpenAI's Realtime documentation states user audio is 1 token per 100 ms and assistant audio is 1 token per 50 ms, so about 600 input tokens per minute of listening.

Token counts are not comparable across vendors. Compare dollars per audio minute instead.

Prices used here (checked October 2, 2026)

• Gemini 3.8 Flash: $0.75 per million input tokens and $3.75 per million output tokens on the Gemini API pricing page, listed as valid through December 31, 2026 and doubling on January 1, 2027.
• Gemini 3.5 Flash-Lite: $0.30 per million input tokens for text, image, video and audio; $2.50 per million output.
• gpt-realtime-2.1: $32 per million audio input tokens; $4 per million text input; $24 per million text output, per OpenAI pricing. gpt-realtime-2.1-mini: $10 per million audio input.
• Deepgram Nova-3 pre-recorded transcription: $0.0043 per minute pay-as-you-go on Deepgram pricing. OpenAI gpt-transcribe: $0.0045 per minute.

Worked example: one 5-minute call

Assumptions, labeled as such: the call is 5 minutes (300 seconds). The transcript is about 700 words, roughly 950 text tokens at an assumed 1.35 tokens per word. The judge rubric and instructions add 1,500 tokens. The judge outputs 300 tokens of structured scores.

Three things fall out of this table.

First, a whole-call audio judge on Gemini Flash costs about three times a text judge, not thirty. In absolute terms it is under a cent per call.

Second, independent re-transcription costs more than audio judging on a Flash-class model. It is still worth running, because it is the only reliable fix for laundering, but it is the line to sample if budget is tight. Re-transcribing only the caller channel halves it versus both channels.

Third, model choice dominates format choice. Routing audio through a realtime speech-to-speech model as a judge costs about ten times a Flash-class model for the same audio. Realtime models are priced for live conversation, not batch grading.

The silero-vad project reports that one 30 ms audio chunk takes under 1 ms on a single CPU thread. A 5-minute channel is 10,000 chunks, so under 10 seconds of one core. Signal metrics are effectively free, which is why they should run on every call.

The hidden cost: audio judges see one mixed channel

Gemini's docs say multichannel audio is combined into a single channel. If you upload a dual-channel recording, the judge hears a mix and loses the clean "who spoke when" separation. For overlap and talk-over, measure on the separate channels with signal tools before anything is mixed. Give the audio judge short clips around flagged events instead of the whole call. That is cheaper and, given the research above, more accurate.

The recording setup that makes this possible

None of the signal metrics work without the right recording. Fix this first.

Record dual-channel. Twilio says call recordings are stored dual-channel by default for two-party calls, with the parent call in the first channel and the child call in the second. Two traps. When the `RequestedChannels` download parameter is not specified, the download follows whatever was requested for the recording, so a pipeline can quietly fetch a mono mix. And recordings made through the TwiML Conference noun or the Record verb are mixed into a single channel. If your warm transfer flow uses a conference, your recordings are mono and every overlap metric needs diarization instead.

Know which channel is which. Leg order differs between inbound and outbound flows. Confirm the mapping with a test call where only the agent speaks.

Keep one clock. Log every VAD, STT, TTS, tool and LLM event against the recording's start time. Backend logs on unsynchronized wall clocks can drift enough to pin a slow tool call on the wrong turn. Our guide on what to log on every voice agent call lists the fields.

Merging tool calls into the transcript timeline

Text judges do much better when they see one merged timeline instead of a bare transcript. Tool calls, played-versus-generated text, and re-transcription disagreements all go in. A minimal event shape, illustrative:

{
  "call_id": "c_8812",
  "t0_utc": "2026-10-02T14:03:11.204Z",
  "events": [
    {"t_ms": 18400, "src": "vad", "ch": "caller", "type": "speech_end"},
    {"t_ms": 18410, "src": "stt_live", "ch": "caller", "text": "fifty milligrams"},
    {"t_ms": 18410, "src": "stt_indep", "ch": "caller", "text": "fifteen milligrams", "entity_mismatch": ["dose"]},
    {"t_ms": 18950, "src": "tool", "name": "update_rx", "args": {"dose_mg": 50}, "latency_ms": 640},
    {"t_ms": 19870, "src": "vad", "ch": "agent", "type": "speech_start"},
    {"t_ms": 19870, "src": "agent_text", "generated": "Got it, fifty milligrams. Anything else?", "played": "Got it, fifty milligrams."}
  ]
}

With this, a text judge can see that the tool acted on a contested entity, that the agent took 1,470 ms to respond, and that half the reply was never heard. None of that is visible in a plain transcript. Silent tool failures are a related problem, covered in our guide to detecting silent tool failures.

Signal-level metrics you can compute today

Here is an illustrative script that computes the timing and level metrics from a dual-channel WAV. It uses the `silero-vad` package's documented `load_silero_vad` and `get_speech_timestamps` functions and `soundfile` for reading. It assumes 8 kHz or 16 kHz audio, the two rates Silero supports, and channel 0 as the caller. Verify both for your stack.

# Illustrative: dual-channel signal metrics for one call.
import numpy as np
import soundfile as sf
import torch
from silero_vad import load_silero_vad, get_speech_timestamps

model = load_silero_vad()

def segments(x, sr, merge_gap=0.5):
    ts = get_speech_timestamps(torch.from_numpy(x).float(), model,
                               sampling_rate=sr, return_seconds=True)
    segs = []
    for t in ts:  # merge short pauses into one turn
        if segs and t["start"] - segs[-1][1] < merge_gap:
            segs[-1] = (segs[-1][0], t["end"])
        else:
            segs.append((t["start"], t["end"]))
    return segs

audio, sr = sf.read("call.wav", dtype="float32")
assert audio.ndim == 2 and sr in (8000, 16000), "need dual-channel 8k/16k"
caller, agent = audio[:, 0], audio[:, 1]
u, a = segments(caller, sr), segments(agent, sr)

# 1. Response gap: caller turn end -> next agent speech start
gaps = []
for _, ue in u:
    nxt = [s for s, _ in a if s >= ue]
    if nxt:
        gaps.append(nxt[0] - ue)

# 2. Talk-over: agent starts while caller is mid-turn
talk_over = [(s, e) for s, e in a for us, ue in u if us < s < ue - 0.3]

# 3. Barge-in stop latency: caller starts while agent speaks
stop_lat = [ae - us for us, _ in u for as_, ae in a if as_ < us < ae]

# 4. Level and clipping on the agent channel
rms_dbfs = 20 * np.log10(np.sqrt(np.mean(agent ** 2)) + 1e-12)
clip_ratio = float(np.mean(np.abs(agent) >= 0.999))

print({
    "gap_p50_ms": round(np.percentile(gaps, 50) * 1000) if gaps else None,
    "gap_p95_ms": round(np.percentile(gaps, 95) * 1000) if gaps else None,
    "talk_over_events": len(talk_over),
    "barge_in_stop_p95_ms": round(np.percentile(stop_lat, 95) * 1000) if stop_lat else None,
    "agent_rms_dbfs": round(float(rms_dbfs), 1),
    "agent_clip_ratio": clip_ratio,
})

A few notes on reading the output. Short caller sounds like "uh-huh" during agent speech are backchannels, not barge-ins. Filter caller segments under about 400 ms before computing stop latency, or you will punish the agent for correctly ignoring them. The 0.3-second margin on talk-over keeps the agent's normal turn-start from counting as an overlap. Tune both on 20 to 30 calls you have listened to.

For noise and quality, DNSMOS P.835 gives three non-intrusive scores: speech quality (SIG), background noise (BAK) and overall (OVRL). The paper reports Pearson correlation with human P.835 ratings of 0.94 for SIG and 0.98 for BAK and OVRL. Those numbers come from noise-suppression test sets, not narrowband phone calls, so treat DNSMOS as a relative signal within your own traffic. A caller-channel BAK drop is a good predictor of where STT errors will cluster. For TTS quality more broadly, see our mean opinion score guide.

Independent re-transcription in code

This illustrative snippet sends the dual-channel file to Deepgram's pre-recorded endpoint with `multichannel=true`, so each channel is transcribed separately, and compares the caller channel to the live transcript. It uses the REST endpoint directly so it does not depend on SDK version changes.

# Illustrative: re-transcribe and diff against the live STT transcript.
import os, re, requests, jiwer

DG_KEY = os.environ["DEEPGRAM_API_KEY"]
with open("call.wav", "rb") as f:
    r = requests.post(
        "https://api.deepgram.com/v1/listen",
        params={"model": "nova-3", "multichannel": "true",
                "smart_format": "false", "keyterm": ["Nguyen", "metformin"]},
        headers={"Authorization": f"Token {DG_KEY}", "Content-Type": "audio/wav"},
        data=f,
    )
r.raise_for_status()
channels = r.json()["results"]["channels"]
indep_caller = channels[0]["alternatives"][0]["transcript"]

def norm(s):
    s = s.lower()
    return re.sub(r"[^a-z0-9' ]+", " ", s).strip()

live_caller = open("live_caller_transcript.txt").read()
disagreement = jiwer.wer(norm(indep_caller), norm(live_caller))
print(f"cross-ASR disagreement: {disagreement:.1%}")

`smart_format` is set to false here so numbers stay as words on the independent side. If your live STT formats numbers as digits, normalize both sides into one form before comparing. `keyterm` is Deepgram's keyterm prompting for Nova-3, which helps the independent pass get your domain words right. It is billed as an add-on, so budget for it. Check Deepgram's pricing FAQ on how multichannel audio is billed before you send both channels at volume.

Then extract entities from both transcripts with the same deterministic parser and compare slot by slot. Any slot that drove a tool call and disagrees goes straight to review. Noise is the main cause of these disagreements, which is why we pair this with the conditions in our guide to testing STT under background noise.

How to run a hybrid transcript and audio evaluation protocol

This is the protocol. It runs cheap, deterministic checks on every call and spends listening effort only where the cheap layers point.

1. Fix the recording. Confirm dual-channel recordings, the channel mapping per flow, and one shared clock for every event. Do not start scoring until a test call proves all three.

2. Run signal metrics on 100% of calls. Compute response gap p50 and p95, talk-over events, barge-in stop latency, intra-turn silences, RMS level, clip ratio and caller-channel DNSMOS BAK. Store them per turn, not only per call.

3. Run independent re-transcription on the caller channel. Use a different STT model family than production. Compute cross-ASR disagreement and entity mismatches. Run it on 100% of calls if budget allows, or on all calls with low BAK scores plus a random 10%.

4. Re-transcribe the agent channel on a sample. Diff it against logged agent text to catch unplayed text (failure 4), TTS mispronounced entities (5), and repeated chunks (7). Run it on 100% of calls for regulated flows where read-backs matter.

5. Build the merged timeline. Join live STT, independent STT, played-versus-generated agent text, tool calls with latencies, and signal flags into one event list per call.

6. Run text judges and rules on the merged timeline. Policy, disclosure, facts, tool arguments, task outcome. Tell the judge that entity mismatches exist so it does not treat the live STT as truth.

7. Route flagged spans to a pairwise audio judge. Cut 10 to 20 second clips around flagged events. Compare each against a reference clip of correct behavior. Run both orders, and keep only verdicts that agree.

8. Add a random audio sample for drift. Pull a random 2% to 5% of calls through the same pairwise audio checks, so you catch failure types your flags were not built for.

9. Calibrate with humans. For each audio-judge dimension, have humans label about 250 clips (the worked math is below). Measure agreement, tie behavior and order bias before trusting the judge's numbers.

10. Close the loop. Every audio-found failure becomes either a new signal rule, a new entity check, or a fixed scenario in your pre-release suite.

Our guide to sampling live calls covers how to size the random sample, and human-in-the-loop evaluation covers the review workflow.

Starting thresholds to tune

These are starting points, not standards. Tune them on calls you have listened to.

Sample-size math for audio judges

Two calculations come up constantly.

How many human labels to calibrate a judge. To estimate judge-human agreement within plus or minus 5 points at 95% confidence, with agreement expected near 80%: n = z² x p(1 - p) / e² = 1.96² x 0.8 x 0.2 / 0.05² ≈ 246 labeled clips per dimension. Need plus or minus 10 points? About 62.

How many random calls to sample to catch a rare failure. If a failure appears in 1% of calls, the chance a random sample of k calls contains at least one is 1 - 0.99^k. For a 95% chance, k = ln(0.05) / ln(0.99) ≈ 298 calls. For a 0.5% failure, about 598. This is why the random audio sample alone cannot carry the protocol, and why the 100% signal layer matters.

Copyable decision table: which artifact for which check

Use this table to assign every check in your suite to the cheapest artifact that can see it.

The "Blind" cells in the transcript column are what a transcript-only program will never catch. The audio judge column is narrow on purpose.

Emotional tone has its own measurement problems, covered in our guide to emotional tone evaluation.

Common mistakes

Grading the live transcript as ground truth. This is the laundering problem. The fix is independent re-transcription, and it is non-negotiable for any flow where the agent acts on spoken numbers or names.

Asking an audio LLM for a 1 to 5 tone score on a whole call. The research says pointwise paralinguistic judging is near chance for most models. Use pairs, clips and calibration.

Measuring latency from transcript timestamps. Transcript timestamps often mark when text was finalized, not when sound started or stopped. Use VAD on the audio. Time-to-first-audio is a related metric, covered in our time to first audio guide.

Logging generated text instead of played text. If your logs hold streaming deltas, they include text the caller never heard. Check whether your framework truncates on interruption and log the final, truncated item. One LiveKit GitHub issue reports interrupted speech dropped from history in an edge case where audio pauses before the first frame. Edge cases like that are why the agent-channel re-transcription check exists.

Where independent evaluation fits

Transcript-versus-audio is mostly a question of who grades whom. A team grading its own agent with its own STT output and its own judge prompt has no outside check on the laundering problem. This is where Evalgent helps. As an independent evaluator, Evalgent runs scripted and synthetic calls against your agent over real telephony audio, then scores the recording with its own re-transcription and signal checks rather than your pipeline's transcript.

Three moments benefit most. Before launch, an audit catches unplayed text, mispronounced entities and talk-over that internal transcript tests passed. In a vendor bake-off, every vendor is scored on the same audio, not its own transcript. After launch, regression runs and production scoring track signal metrics and entity mismatch rate across releases.

Frequently asked questions

What is the difference between transcript and audio evaluation?

Transcript evaluation grades the text of a call: content, policy, tool use and outcome. Audio evaluation grades the recording: timing, overlap, pronunciation, delivery and what the caller heard. The transcript is built from your own STT and the LLM's intended text, so it can hide recognition errors and unplayed speech. Audio evaluation includes signal metrics, re-transcription and audio judges.

What is ASR laundering?

ASR laundering is when a speech recognition error becomes invisible because the evaluation reads the recognizer's own output. If the caller said fifteen and STT heard fifty, the transcript shows a correct conversation about fifty, and a text judge passes it. The fix is to re-transcribe the recording with an independent STT and flag entity mismatches.

How do I evaluate transcription accuracy on production calls?

Without reference transcripts, measure cross-ASR disagreement by re-transcribing the caller channel with a second STT and comparing after normalization. Track entity mismatches on slots the agent acted on. Calibrate with true WER on a human-labeled slice of about 200 utterances per condition. Normalize numbers and punctuation the same way on both sides first.

Are audio language models reliable judges of tone and emotion?

Not when scoring a single call alone. A 2026 counterfactual audit found most audio-LLM judges near 50% chance in pointwise paralinguistic judging. Gemini models reached 74% to 91% only when comparing two versions side by side. ParaPairAudioBench found judges 32 points behind humans on average, with weak emphasis detection. Use pairwise comparisons and human calibration.

How much does audio evaluation cost compared to text?

With October 2026 pricing, a whole-call audio judge on Gemini 3.8 Flash costs about $0.0095 for a 5-minute call, about three times a text judge at $0.0030. Independent re-transcription with Deepgram Nova-3 costs about $0.0215 for the caller channel. Realtime speech models used as judges cost roughly ten times more than Flash-class models.

Can I measure voice agent latency from the transcript?

Not reliably. Transcript timestamps usually mark when text was finalized, not when the caller stopped speaking or the agent's audio began. Measure response gap with voice activity detection on a dual-channel recording: caller speech end to agent speech start. Report p50 and p95 per call, and filter backchannels before computing barge-in stop latency.

How do I merge tool call data into voice agent transcripts for evaluation?

Log every event against the recording's start time: VAD segments, live STT, independent STT, generated and played agent text, and tool calls with arguments and latency. Merge them into one ordered event list per call. Text judges reading this timeline can see slow tools, tools acting on contested entities, and replies that were cut off.

Do I need dual-channel recordings for audio evaluation?

For timing and overlap metrics, yes. Separate channels let you measure talk-over and barge-in directly with VAD. Twilio stores two-party recordings dual-channel by default, but conference and record-verb recordings are mixed to mono, and downloads may be mixed unless requested otherwise. With mono audio you need diarization, which adds its own errors.

The bottom line

A transcript records what your system believed happened, so it cannot catch the recognition errors, unplayed speech and timing failures that callers actually experience. Measure timing and level on every recording, re-transcribe independently to break ASR laundering, grade content on a merged timeline, and use audio judges only in calibrated pairwise checks.