Voice Agent Evaluation: The Complete 2026 Guide

Most teams can quote their word error rate and their average latency. Far fewer can say how often the agent completes the same task twice in a row, how wide the confidence interval is on last night's 92% pass rate, or whether their LLM judge agrees with a human reviewer beyond what chance alone would produce. Those three questions decide whether an evaluation result means anything. Vendor blogs rarely ask them.

This guide gives you an eight-layer evaluation model with a full metric catalog, the reliability metric that single-run success hides, sample-size math for pass rates, and a protocol for validating an LLM judge. It also covers dataset design, CI gating, and the rhythm that keeps evaluation honest after launch.

What is voice agent evaluation?

Voice agent evaluation is the repeatable measurement of whether an AI voice agent does its job, judged against a written standard. The output is a verdict: pass or fail, a score with an interval, or a comparison against the previous version. Running calls produces evidence. Evaluation turns that evidence into a decision you can defend.

A suite of 200 scenarios that "mostly passes" is activity. A statement like "task success is 91% (95% CI 88–93%) on 600 paired scenarios, no worse than the last release at α = 0.05, zero critical policy violations" is evaluation. The rest of this guide is about getting from the first to the second.

Why voice breaks the assumptions of text evaluation

Three mechanisms make voice evaluation harder than evaluating a chatbot or a bare LLM.

The input is lossy before reasoning starts. On a phone call, caller audio usually arrives as 8 kHz G.711 µ-law. The platform decodes it, often resamples it to 16 kHz for the recognizer, and streams it in 20 ms frames. Every step can drop information. Then the recognizer turns the audio into text with errors. The SpokenWOZ benchmark (Si et al., NeurIPS 2023) built 249 hours of human-to-human spoken task dialogues. The best dialogue state tracker reached only 25.65% joint goal accuracy, and the best end-to-end model completed the user's request in 52.1% of dialogues. The authors trace much of the gap to spoken behavior that text datasets lack. Callers spell names letter by letter across several turns and correct themselves mid-sentence. In practice, an evaluation that feeds clean text into your LLM measures a different system from the one callers reach.

Timing is part of correctness. The agent has to decide when the caller has finished, using voice activity detection plus an endpointing timer. It has to decide whether a sound is an interruption or a backchannel ("mm-hm"). Then it has to start speaking fast enough that the silence doesn't feel like a dead line. The ITU-T G.114 telephony standard treats 150 ms one-way delay as the comfortable limit for human conversation. Voice agents run far above that, so every hop counts. None of these failure modes exist in a text transcript.

The agent is stochastic, and so is the caller. Sampling temperature, streaming partial transcripts, and network jitter mean the same scenario can go differently on two runs. So can the same synthetic caller. One passing run is one draw from a distribution. That is why reliability needs its own metric, covered below.

The eight-layer voice agent evaluation model

A voice agent is a pipeline, and any layer can sink a call. Measuring only the end result hides where the failure started. The model below splits the call into eight layers. Each layer has its own failure mechanism, its own metric, and its own data requirement. Speech-to-speech models collapse layers 2 through 6 into one network. You still evaluate those layers, but from the outside, through audio. Our speech-to-speech testing guide covers how.

Layer 1: Signal and audio

What breaks: codec and sample-rate mismatches, clipping, packet loss, and echo from the agent's own voice leaking back into the input. Audio at the wrong sample rate sounds fine to a human but degrades recognition, and packet loss becomes dropped syllables that the recognizer turns into deletions. Log RTP statistics and audio-quality estimates per call, then segment every downstream metric by audio condition. A WER average across clean and noisy calls describes neither.

Layer 2: Speech recognition (ASR)

What breaks: substitutions on names, digits, and amounts, and hallucinated text during silence. Word error rate, (S + D + I) / N, is the standard. But WER weights "the" the same as a policy number. A 4% WER can still mean one wrong digit in every third account number. Score critical entities separately. Our STT evaluation guide and entity accuracy guide cover both protocols.

Silence deserves its own test set. Koenecke et al. (FAccT 2024) found that about 1% of Whisper transcriptions contained entire hallucinated phrases. About 38% of those included explicit harms, and hallucinations were more likely when speech had longer non-vocal stretches. Callers who pause to find a card number produce exactly that audio. Build a fixture of silence, hold music, and long pauses, and require zero inserted words.

Layer 3: Understanding

What breaks: the transcript is right but the intent or slot is wrong. Callers also correct themselves ("Tuesday, no, Wednesday") or spread one value across several turns. Measure intent accuracy and slot accuracy per turn. Also measure state accuracy: the share of turns where every tracked slot is correct, which dialogue research calls joint goal accuracy. Per-intent recall matters more than the average, because the rare intents tend to be the costly ones.

Layer 4: Policy and reasoning

What breaks: skipped identity verification, a missing required disclosure, a promise the business can't keep, or a confident answer with no support in the knowledge base. This layer is where LLM judges do most of their work, and where they most need validation. Split rules into ones you can check in code ("verification tool called before balance read") and ones that need judgment ("tone stayed professional"). Never send a code-checkable rule to a judge. The policy adherence guide gives a rule-by-rule method.

Layer 5: Actions and tools

What breaks: wrong tool, right tool with wrong arguments, a tool error the agent never mentions to the caller, or a double booking on retry. The strongest evidence of task success is the end state of the backend, not the transcript. tau-bench (Yao et al. 2024) grades each episode by comparing the database at the end of the conversation with an annotated goal state. Different conversations can reach the same correct state, so the grade ignores conversational wording. Copy that design. See tool argument accuracy and silent tool failures.

Layer 6: Speech output

What breaks: slow first audio, mispronounced names, numbers read too fast to catch, and prosody that sounds flat on bad news. Time to first audio (TTFA) is measured from the caller's end of speech to the first agent audio reaching the caller. That covers endpointing wait, ASR finalization, LLM time to first token, TTS time to first byte, and network legs. Report it at p50 and p95, never as a mean. Check pronunciation with a round-trip test: synthesize, re-transcribe, compare entities. Details are in TTS evaluation and time to first audio.

Layer 7: Conversation dynamics

What breaks: the agent cuts callers off mid-thought (endpointing too aggressive). It leaves long dead air (too conservative). It stops talking because the caller said "uh-huh" (false barge-in), or it talks over a real interruption (missed barge-in). Full-Duplex-Bench (Lin et al., ASRU 2025) tests four behaviors with automatic metrics: pause handling, backchanneling, turn-taking, and interruption management. That is a useful checklist even for cascaded agents. These failures appear only in audio, which is why transcript-only evaluation misses them. The turn-taking evaluation guide covers measurement.

Layer 8: Business outcome

What breaks: the agent contains a call without solving it, escalates too late, or "resolves" an issue the caller phones back about the next day. Containment is easy to measure and easy to game. Pair it with resolution checks and repeat-contact rates. Task success and escalation accuracy are the outcome metrics worth gating on.

The voice agent metric catalog

The table below is the working catalog. The gates are starting points for a typical support or scheduling agent, not industry standards. Tighten them for regulated flows, and re-baseline them against your own production data after the first month. Gate metrics block a release. Monitor metrics trigger investigation. For threshold tuning and alert tiers, see voice agent metric thresholds.

Two rules make this table usable. Every gate metric reports its sample size and interval, not just a point estimate. And every metric is stored per call with scenario, persona, audio condition, and build hash, so you can slice a regression to its cause. Our call logging guide lists the fields.

Reliability: why one passing run proves little

Here is the insight from agent research that most voice teams still miss. tau-bench introduced pass^k: the probability that an agent succeeds on all k independent runs of the same task, averaged over tasks. With n runs per task and c successes, the unbiased estimate per task is C(c, k) / C(n, k), averaged across tasks. Code-generation benchmarks use pass@k, which is the opposite: at least one success in k tries. pass@k rewards an agent that sometimes gets it right. pass^k rewards an agent that gets it right every time, and a caller only gets one try.

The tau-bench results show how big the gap is. GPT-4o with function calling succeeded on about 61% of retail tasks in a single run (pass^1). When the same tasks had to succeed in all eight runs, the figure fell to about 25% (pass^8). The authors attribute the failures to complex reasoning over databases, ad-hoc policies, and compound requests. Voice calls are full of all three.

The worked example below shows why pass^1 alone can mislead you. It is illustrative, not measured data. Two agents both score 80% task success on a single run of 100 tasks:

• Agent A passes 80 tasks every time and fails the other 20 every time. Its failures are consistent, so you can find them, fix them, or route them to humans. pass^8 = 80%.
• Agent B passes every task 80% of the time, at random. pass^8 = 0.8^8 ≈ 17%. Run it for a month and every task fails sometimes. No failure reproduces cleanly, and callers see an agent that works "most of the time" on everything.

A single-run suite can't tell these agents apart. In practice, pick 20 to 50 core tasks that carry most of your volume or risk, run each 8 times nightly with varied personas and seeds, and gate on pass^4. The 8x cost applies to a small subset, not the whole suite. If you need personas for those reruns, see synthetic callers.

How many calls do you need? Sample-size math for pass rates

A pass rate without an interval is a guess. Use the Wilson score interval (see binomial proportion confidence intervals) rather than the textbook p ± 1.96·√(p(1−p)/n). The textbook interval misbehaves near 0% and 100%, which is where gate metrics live. The Wilson interval is:

center = (p + z²/2n) / (1 + z²/n)
half   = z · √( p(1−p)/n + z²/4n² ) / (1 + z²/n)
CI     = center ± half        (z = 1.96 for 95%)

Here is what a 90% observed pass rate is worth at different suite sizes (95% Wilson intervals, computed):

So a 50-scenario suite that reads "90%" is consistent with a true rate anywhere from the high 70s to the mid 90s. To estimate a rate near 90% within ±3 points, you need n = z²·p(1−p)/E² = 3.84 × 0.09 / 0.0009 ≈ 384 runs. For ±5 points, about 139.

Detecting a regression needs more data than estimating a level. Suppose the current release succeeds 90% of the time and you want to catch a drop to 87%. That calls for 80% power at a two-sided α of 0.05. With two independent samples, the standard two-proportion formula gives about 1,774 runs per arm. Most suites are nowhere near that, which is why "it went from 91% to 89%" usually means nothing.

The fix is pairing. Run the old and new builds on the same scenarios, with the same personas and seeds, and test the discordant pairs with McNemar's test. Suppose 5% of scenarios flip from pass to fail and 2% flip from fail to pass (a net 3-point drop; these rates are assumptions for illustration). Then you need about 608 paired scenarios for the same power. Evan Miller's Adding Error Bars to Evals makes the same recommendation for LLM evals: compare models on question-level paired differences, not on two summary numbers.

Two more corrections matter in voice suites:

• Clustering. If each scenario runs across 6 personas, those 6 results are correlated. Miller shows that clustered standard errors can be more than 3x larger than naive ones on real evals. A rough guide is the design effect, 1 + (m − 1)·ρ. With m = 6 runs per scenario and an assumed intra-scenario correlation ρ = 0.5, 600 runs carry about the information of 171 independent ones. Report the number of scenarios alongside the number of runs.
• Zero-failure claims. For safety gates, the rule of three applies. With 0 failures in n trials, the 95% upper bound on the true failure rate is about 3/n. To claim prompt injection succeeds less than 1% of the time, you need about 300 clean attempts. For under 0.1%, about 3,000. A red-team suite of 40 attacks with zero successes supports only "below roughly 7.5%." The red-team audit guide covers attack design.

Judge reliability: validating an LLM judge against humans

LLM judges make layer 4 and layer 8 evaluation affordable at scale. They are also measuring instruments, and an uncalibrated instrument produces numbers that look precise and aren't.

Zheng et al. (2023), the MT-Bench and Chatbot Arena paper, found that GPT-4 matched human preferences over 80% of the time, about the same as human-human agreement. The same paper measured the biases behind that headline:

• Position bias. When the order of two answers was swapped, GPT-4 gave a consistent verdict only 65% of the time and favored the first answer in 30% of cases. Claude-v1 was consistent 23.8% of the time.
• Verbosity bias. In a "repetitive list" attack, answers were padded with rephrased duplicate items. The attack fooled GPT-3.5 and Claude-v1 91.3% of the time, and GPT-4 8.7% of the time.
• Grading without a reference. On math questions, GPT-4 marked a wrong answer correct in 14 of 20 cases with the default prompt, 6 of 20 with chain-of-thought, and 3 of 20 when given a reference answer.

These are 2023 models, and newer judges do better. The mechanisms haven't gone away, though. For voice, the practical lessons are to give the judge the reference (the expected end state, the policy text, the tool outputs), to ask one narrow question per call, and never to let response length or order carry information.

Thakur et al. (2024, GEM 2025) tested 13 judge models in a deliberately easy setup where humans agree strongly. Only the largest models aligned reasonably well, and even those trailed human agreement. Their scores could differ from human scores by up to 5 points, and judges leaned lenient. The finding that matters most for builders: high percent agreement can hide very different scores. The authors recommend chance-corrected agreement metrics over raw percent agreement.

Why percent agreement lies: a worked example

Take 500 calls where humans fail 50 (10%) for a policy violation. Your judge also fails 50, agreeing with humans on 460 of 500. That's 92% agreement, which sounds excellent. Now compute Cohen's kappa, κ = (pₒ − pₑ) / (1 − pₑ):

• Observed agreement pₒ = 0.92.
• Chance agreement pₑ = 0.9 × 0.9 + 0.1 × 0.1 = 0.82. Two raters who both pass about 90% of calls agree 82% of the time by luck.
• κ = (0.92 − 0.82) / (1 − 0.82) = 0.56.

On the common Landis and Koch scale, 0.56 is only "moderate." The 40 disagreements split 20 and 20, so the judge caught 30 of the 50 real violations. It missed 40% of the failures you built it to catch. Percent agreement looks good because most calls are passes and both raters say "pass." When failures are rare, report kappa plus the judge's recall on human-labeled failures.

For more than two raters, or for ordinal 1-to-5 scales, use Krippendorff's alpha. By common social-science convention, α ≥ 0.800 is reliable, 0.667 to 0.800 supports only tentative conclusions, and anything lower should be discarded. Use those bands as your judge's promotion criteria.

The judge validation protocol

1. Write the rubric as narrow binary questions. Ask "Did the agent read the cancellation fee before confirming?", not "Rate policy adherence 1 to 10."

2. Label a calibration set with two humans. Stratify the sample so it holds about 100 human-confirmed failures per criterion. Estimating failure recall within ±10 points takes roughly that many. With a 10% base rate, random sampling would need 1,000 labeled calls, so pull suspected failures on purpose. Measure human-human agreement first. If humans can't agree (α below 0.667), fix the rubric, not the judge.

3. Run the judge and compute κ or α, failure recall, and false-fail rate. Promote a criterion to automated gating only at α ≥ 0.8 and failure recall at or above your tolerance.

4. Run a swap-and-repeat check. Run each pairwise judgment in both orders, and run absolute judgments 3 times. Report self-consistency as its own number.

5. Re-validate monthly and whenever the judge model, prompt, or agent changes. A new agent version can produce failure styles the judge has never seen.

For high-volume classification questions, a purpose-built classifier can stand in for a general LLM. Jev (TypeSafe System One) is highly consistent across repeated runs, costs $0.042 per million input tokens, and responds in 70 to 500 ms. That makes scoring every call affordable. It still goes through the same validation protocol. Our Jev evaluation guide shows the setup, and LLM-as-judge limits and human-in-the-loop evaluation cover the rest.

The statistics in code

The helpers below compute the three numbers this guide keeps returning to. They are simplified, standard-library Python, and the example outputs in the comments match the figures above.

# Simplified: evaluation statistics with no third-party dependencies
from math import comb, sqrt

def wilson_interval(passes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = passes / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return center - half, center + half

def pass_hat_k(trials_per_task: list[list[bool]], k: int) -> float:
    """tau-bench pass^k: mean over tasks of C(c, k) / C(n, k)."""
    scores = []
    for trials in trials_per_task:
        n, c = len(trials), sum(trials)
        if n < k:
            raise ValueError("each task needs at least k trials")
        scores.append(comb(c, k) / comb(n, k))
    return sum(scores) / len(scores)

def judge_agreement(human_fail: list[bool], judge_fail: list[bool]) -> dict:
    n = len(human_fail)
    po = sum(h == j for h, j in zip(human_fail, judge_fail)) / n
    ph, pj = sum(human_fail) / n, sum(judge_fail) / n
    pe = ph * pj + (1 - ph) * (1 - pj)
    caught = sum(h and j for h, j in zip(human_fail, judge_fail))
    return {
        "percent_agreement": po,
        "cohens_kappa": (po - pe) / (1 - pe),
        "failure_recall": caught / sum(human_fail),
    }

print(wilson_interval(45, 50))  # (0.786, 0.957)
print(pass_hat_k([[True] * 6 + [False] * 2], k=4))  # 0.214: 6 of 8 runs passed

Designing the evaluation dataset

A useful evaluation set is designed like a test matrix, not collected like a pile of transcripts. Our golden dataset guide goes deeper.

Build a coverage matrix. One axis lists your intents or tasks, weighted by volume and by risk. The other lists conditions. Each cell is a scenario with a written goal state. A typical starting matrix:

You can't run the full cross product, so sample it. Cover every task with the cooperative, clean baseline. Then use pairwise combination so every pair of condition levels appears together at least once, and add extra scenarios to high-risk cells.

Write goal states, not expected transcripts. As in tau-bench, define success as the backend state and the required disclosures, not a word-for-word script. Agents that phrase things differently but reach the right state should pass.

Keep adversarial cases separate and growing. Red-team scenarios go in their own suite with zero-tolerance gates and rule-of-three sizing. Every production incident becomes a new adversarial case within a week.

Guard against contamination. If you tune prompts against the same scenarios you gate on, the gate stops measuring generalization. Split the set into a development set (tune freely), a held-out gate set (never shown to prompt writers), and a small locked canary set that never changes, so you can compare across months. Rotate a share of the gate set each quarter.

Refresh on a cadence. Add 10 to 20% new scenarios a month, drawn from production failures, new intents, and drift alerts. Retire scenarios that haven't failed on any build in six months only after keeping them in the canary set. Version scenarios, prompts, judge rubrics, and results together. A result you can't trace to the exact scenario version and build hash can't be reproduced.

Offline vs online evaluation

Offline evaluation runs controlled scenarios before release. Online evaluation scores real calls after release. They answer different questions, and each covers the other's blind spot. The full comparison is in offline vs online voice agent evaluation.

Online evaluation has its own traps. Score a stratified sample (see sampling live calls) or every call if the scorer is cheap. Alert on changes outside a control band, not on single bad days. Watch for population shift: a new caller cohort can move every metric with no change to the agent, as metric drift explains. Live comparisons between builds (A/B testing) use the same sample-size math, with calls instead of scenarios.

Regression gating in CI

A gate is a rule a build must pass before it ships. The goal is to block real regressions without blocking every build on noise. Three tiers work for most teams:

1. Per pull request (minutes). Run a 30 to 60 scenario smoke suite covering core tasks and critical rules, plus deterministic checks: tool schemas, required disclosures, the silence fixture. Fail on any critical violation. Don't gate on pass-rate deltas here, because the sample is too small to mean anything.

2. Nightly or pre-release (hours). Run the full gate set, paired against the current production build with the same seeds. Add pass^k on core tasks and audio-condition slices. Gate on non-inferiority: block if the paired test shows a drop larger than your margin.

3. Model or provider change (a day). Any LLM, STT, TTS, or orchestration upgrade runs the full suite plus the judge re-validation sample. A model upgrade changes the agent and can also change how your judge scores it. Our LLM update regression guide walks through a real upgrade path.

An illustrative gate specification (not any specific tool's schema) looks like this:

# Illustrative release gate for a scheduling agent
suite: gate_set_v14          # held-out, versioned
baseline: prod_build_2026_09_28
pairing: same_scenarios_same_seeds
gates:
  critical_rule_violations: {max: 0}
  silence_hallucination:    {max: 0, suite: silence_fixtures_v3}
  task_success:             {min_wilson_lower: 0.85, min_scenarios: 400}
  task_success_vs_baseline: {test: mcnemar, max_drop: 0.02, alpha: 0.05}
  core_pass_hat_4:          {min: 0.75, tasks: core_25, trials_per_task: 8}
  critical_entity_accuracy: {min: 0.98}
  ttfa_p95_ms:              {max: 1500}
report:
  per_slice: [audio_condition, persona, intent]
  include_intervals: true

Choose the non-inferiority margin before you run, not after you see the result. When a gate fails, store the failing calls with audio so reviewers can hear the regression.

The evaluation operating rhythm

Evaluation decays if it only happens at launch. Prompts change, models get upgraded underneath you, caller mix shifts, and judges drift. Each cadence below has an owner and an output.

The weekly human review does double duty: it finds failures the judges missed, and it produces fresh labels for the monthly judge check. Feeding confirmed failures back into the suite is what makes evaluation improve over time. The production feedback loop guide describes the plumbing.

How to evaluate an AI voice agent: the step-by-step program

1. Write the standard. For each task, write the goal state, the must-do and must-not rules, and the escalation conditions.

2. Instrument every layer. Record both audio channels with timestamps, ASR finals, LLM inputs and outputs, tool calls with arguments, and the build hash.

3. Build the coverage matrix. Cover intents across caller, speech, audio, network, data, and safety dimensions. Split it into development, gate, and canary sets.

4. Pick metrics from the catalog. Separate gate metrics from monitor metrics. Code-check every rule you can, and send only the rest to judges.

5. Validate your judges. Compute κ or α and failure recall against two human labelers, and automate only criteria that clear α ≥ 0.8.

6. Size the suite. Use Wilson intervals for levels, paired McNemar sizing for regressions, the rule of three for safety, and 4 to 8 runs per core task.

7. Wire the three CI tiers. Smoke per PR, paired gates nightly, and the full suite plus a judge check on any model or provider change.

8. Run the operating rhythm. Review weekly, re-validate and refresh monthly, re-baseline quarterly, and feed production failures back every week.

Common voice agent evaluation mistakes

• Reporting a pass rate without an interval, or comparing two builds on unpaired runs.
• Running each scenario once, which hides agents that fail randomly on every task.
• Trusting percent agreement for a judge when failures are rare.
• Gating on containment, or scoring transcripts when the failure lives in the audio.
• Tuning prompts on the gate set, so the gate ends up measuring memorization.

Where independent evaluation fits

You can build everything in this guide in-house, and many teams should own the CI tier themselves. An independent evaluator earns its place where the team that built the agent is the wrong one to grade it. That means pre-launch audits, vendor bake-offs where every vendor must face the same scenarios under the same conditions, and recurring production scoring that a vendor's own dashboard can't credibly provide. The reasoning is laid out in independent voice AI evaluation.

Evalgent runs calls over real audio with synthetic callers across personas and audio conditions, scores each layer separately, and reports gate metrics with intervals and replayable recordings. It doesn't replace your judgment about what "good" means for your callers. It makes the measurement repeatable and defensible.

Frequently asked questions

What is voice agent evaluation?

Voice agent evaluation is the repeatable measurement of whether a voice agent meets a written standard, across eight layers: audio, speech recognition, understanding, policy, tool actions, speech output, conversation dynamics, and business outcome. It produces a verdict, such as a pass rate with a confidence interval or a non-inferiority result against the previous build, rather than a pile of call recordings.

What metrics matter most when evaluating AI voice agents?

Gate on end-state task success, critical rule violations, tool-call accuracy, critical entity accuracy, time to first audio at p95, cut-off and barge-in rates, and escalation recall. Add pass^k on core tasks for reliability. Monitor containment, naturalness, and verified resolution, but don't gate on containment alone, because it rises when callers get trapped.

What is pass^k and why does it matter for voice agents?

pass^k, from the tau-bench paper, is the probability that an agent succeeds on all k runs of the same task, averaged over tasks. Single-run success overstates reliability. In tau-bench, GPT-4o fell from about 61% (pass^1) to about 25% (pass^8) on retail tasks. Callers get one try, so consistency matters as much as average success.

How many test calls do you need to evaluate a voice agent?

To estimate a pass rate near 90% within ±3 points, you need about 384 runs. To detect a drop from 90% to 87% between two builds at 80% power, you need about 1,774 runs per arm unpaired, or roughly 600 paired scenarios. For safety claims with zero failures, the 95% upper bound is about 3 divided by n.

How do you validate an LLM judge against human labels?

Have two humans label a stratified calibration set with about 100 confirmed failures per criterion. Then compare the judge with Cohen's kappa or Krippendorff's alpha, not percent agreement, and measure the judge's recall on failures. Automate a criterion only at alpha of 0.8 or higher. Re-validate monthly and after any model or prompt change.

What is the difference between offline and online voice agent evaluation?

Offline evaluation runs versioned synthetic scenarios with known goal states before release, using paired comparisons to decide if a change is safe. Online evaluation scores real production calls to catch failures and drift your scenarios never imagined. Offline gives exact ground truth but limited realism. Online gives realism but needs judges, outcome joins, and human review.

How do you gate voice agent releases in CI?

Use three tiers. Run a smoke suite with deterministic and critical-rule checks on every pull request. Run the full gate set nightly, paired against production with the same seeds and a pre-set non-inferiority margin. Run the full suite plus a judge re-validation on any LLM, STT, TTS, or orchestration change. Store failing calls with audio for review.

How often should you refresh a voice agent evaluation dataset?

Add 10 to 20% new scenarios each month from production failures, new intents, and drift alerts, and turn every confirmed incident into a test within a week. Keep a locked canary set for comparisons across months. Rotate part of the held-out gate set each quarter so prompt tuning doesn't quietly overfit to it.

The bottom line

Voice agent evaluation is trustworthy only when every layer has its own metric and gate, every pass rate carries an interval, core tasks run enough times to expose inconsistency, and every judge is calibrated against humans. Build that measurement once, run it on a fixed rhythm, and feed every production failure back into it, and your evaluation will catch regressions before callers do.