AI voice agent testing: the complete guide to layers, failure modes and regression (2026)

Most in-house voice teams test one layer and ship the other five. They write a few prompt checks, make some calls from a laptop, and push to live callers. Then the agent talks over a caller on a cell phone, books the wrong date because STT heard "fifteen" as "fifty", or leaves a voicemail half a sentence long.

This guide is our hub for voice agent testing. It maps test layers to failures, covers outbound calls, and gives a minimal test plan, thresholds, a regression workflow and an example harness. Each section links to a deep guide. New to the stack? Start with the voice agent stack guide.

What is AI voice agent testing?

AI voice agent testing: verifying that a voice agent behaves correctly from audio in to audio out, under realistic caller and network conditions, before and after every change. It covers what the agent hears, decides, does with tools, says, and when it says it.

Voice agent testing is not text agent testing with a microphone added. A text harness checks inputs and outputs. A voice call adds four things a text harness never sees:

• An acoustic channel. Phone audio is 8 kHz G.711 on most US PSTN legs. Your STT may be fed resampled 16 kHz audio that has already lost everything above 3.4 kHz.
• A clock. Silence is meaningful. A 1.5-second gap feels broken, and a 300 ms early reply cuts the caller off.
• Overlap. Callers interrupt, backchannel ("mm-hm"), and pause mid-sentence. The agent's turn-taking logic decides what happens.
• Irreversible side effects. A wrong tool call books, charges or cancels something while the conversation still sounds fine.

So a voice agent can pass every transcript check and still fail the call. See transcript vs audio evaluation for the failures a transcript hides.

The five layers of voice agent testing

Each layer exercises a different slice of the pipeline at a different cost. The mistake is assuming a cheap layer covers an expensive one.

Layer 1: text unit tests

You drive the agent with text turns and assert on events: which tool it called, with which arguments, and whether the reply meets an intent.

LiveKit ships this in the framework. `AgentSession.run(user_input=...)` returns a result you can walk with `result.expect.next_event().is_function_call(name=...)`, and `.judge(llm, intent=...)` grades a message. Pipecat has `run_test` in `pipecat.tests.utils` for frame processors and Pipecat Evals for turn-level behavior. Both run your logic. Neither runs your VAD, your STT or your turn detector in this mode, as LiveKit's testing docs make clear by design.

Use this layer for the bulk of your behavioral cases: tool selection, argument normalization, refusals, escalation rules and disclosures. The tool-calling test cases guide has a full library, and regulated policy test cases turns rules into checkable assertions.

Layer 2: audio replay

Audio replay feeds recorded or synthesized caller audio through the real STT, VAD and endpointing settings, without a live phone call. It is where you find out that "my member ID is B as in boy, 4, 7, 7, 1" comes back as "be as in boy for 771."

Build the audio set deliberately: 8 kHz phone-band versions of every clip, two or three noise levels, and the accents your callers actually have. Score entity accuracy, not just WER, because a 3% WER can still miss the one digit that matters. See STT entity accuracy, testing STT under background noise and accent robustness.

One non-obvious check belongs here. Whisper-style models can invent text from silence. Koenecke et al. (FAccT 2024) found that about 1% of Whisper transcriptions contained entire hallucinated phrases, and the hallucinations clustered in speakers with longer non-vocal stretches. Include clips with long pauses and pure line noise, and assert the transcript stays empty.

Layer 3: simulated phone calls

A simulated caller dials your real number, or joins your SIP trunk, and holds a scripted or goal-driven conversation. This is the first layer that exercises the whole path: carrier, codec, jitter buffer, resampling, VAD, endpointing, LLM, tools, TTS and back.

It catches the barge-in that works in the browser but not on G.711, the reply that is fast on WebRTC and slow over SIP, and the greeting that clips its first word. Synthetic callers for voice agent testing and voice agent end-to-end testing cover scenario design. If you are still doing this by hand, replace manual test calls shows how to automate the twenty calls you make before every release.

Layer 4: load tests

Load tests answer one question: does the agent behave the same at 50 concurrent calls as at one? Usually not. Latency tails grow as STT and LLM providers queue requests, workers run out of CPU for VAD and noise cancellation, and rate limits trigger silent retries.

Ramp concurrency in steps, hold each step long enough to see the tail, and report P95 time to first audio at each step. LiveKit has `lk perf agent-load-test` for dispatch and greeting. For provider-side limits, see load testing Deepgram's Voice Agent API and concurrency failures in voice agents.

Layer 5: production scoring

Production is the only test set with your real callers in it. Score every call, not a sample, on a small set of outcome checks: task completed, tool outcome correct, escalation correct, disclosures present, dead air under threshold. Sampling 2% of calls is how a 1-in-200 failure stays invisible for a month. Score every call, not a sample and monitoring AI voice agents in production cover the setup, and what to log on every call lists the fields you need first.

Every production failure should become a test case in layers 1 to 3.

A testing taxonomy by failure layer

Layers of testing describe how you test. Failure layers describe where things break. Map each failure layer to the cheapest test layer that can catch it, and you stop paying for phone calls to find prompt bugs.

Telephony

The phone network sets the floor. LiveKit's telephony troubleshooting guide gives usable thresholds: packet loss under 1% is healthy and over 3% causes audible breakup; mean jitter over 20 ms causes choppy audio; one-way latency over 300 ms makes people talk over each other. The 150 ms one-way target matches ITU-T G.114. More in phone audio quality for voice agents and SIP vs WebRTC.

Test the first two seconds of every call separately. That is where media setup races the agent's greeting and the first word gets clipped. It is also where DTMF and IVR flows break, covered in testing DTMF navigation.

STT

STT errors that matter are concentrated in entities: names, account numbers, dates, amounts, addresses. Report entity error rate per entity type and per caller cohort. A blended WER of 6% can hide a 20% error rate on alphanumeric IDs for one accent group. The STT evaluation guide explains the scoring, and WER for voice agents covers the formula and its limits.

Turn-taking

Turn-taking is a timing problem. People leave short gaps between turns. In Stivers et al. (PNAS 2009), across 10 languages, response offsets clustered tightly around a fraction of a second, so even a one-second gap reads as hesitation. Agents add STT finalization, endpointing delay, LLM time to first token and TTS time to first byte on top of that.

The other direction is just as costly. An agent that treats "mm-hm" or a mid-sentence pause as a turn end interrupts the caller. Full-Duplex-Bench (arXiv 2503.04721) formalizes this: it scores a model's response to a user pause as a "takeover" when the model produces speech that is not a backchannel. Use the same idea in your tests. Script pauses of 600, 900 and 1,500 ms inside a caller's sentence, and count takeovers. The research on predicting turn ends goes back to Voice Activity Projection (Ekstedt and Skantze, Interspeech 2022), which is the idea behind many modern turn detectors.

Deep guides: end-of-turn detection with Flux, Silero and turn detectors, interruption detection evaluation, LiveKit false interruptions, testing VAD misfires and pause handling.

LLM and policy

Most teams test here but run each case once. LLM behavior is probabilistic. A case that passes 95% of the time fails 1 in 20 callers. Run critical cases several times and use pass^k (defined below). Policy tests need three kinds of check: exact phrase (a required disclosure), semantic (did it promise a refund it cannot give), and timing (was the disclosure said in the first 10 seconds). See testing policy adherence, testing guardrails, hallucination rate and knowledge grounding.

Tools

Tool failures are the most expensive and the least visible. The worst one is the silent failure: the API returned an error or timed out, and the agent said "You're all set." Test with a mock backend that can fail on demand: timeouts, 500s, partial success and duplicate submissions after a barge-in. Assert on the tool's final state, not the agent's words. See detecting silent tool failures, tool argument accuracy and function schema validation.

TTS

TTS failures are mostly text normalization: "$1,250.00" read as "one thousand two hundred fifty point zero zero", "Dr." read as "drive", an email address spelled letter by letter at full speed. The other class is truncation when an interruption cancels the stream mid-word and the context still records the full sentence as spoken. That mismatch makes the LLM believe the caller heard something they did not. See TTS evaluation and mean opinion score for voice agents.

Inbound vs outbound voice agent testing

Most testing guides assume inbound calls: the caller dials you and is ready to talk. Outbound calls flip that. Your agent dials, and the first seconds decide whether a human, a voicemail, an IVR or a call screener picked up. Outbound voice testing needs its own cases.

Voicemail and answering machine detection

Answering machine detection (AMD) is a classifier, so test it like one. Build a labeled corpus of real answer audio: humans saying "Hello?" in different ways, short and long voicemail greetings, carrier messages, and call screeners. Measure four rates: false machine (a human treated as voicemail and hung up on), false human (the agent pitches to a voicemail), dead air on humans, and clipped voicemail messages.

Two research findings are worth knowing. Saurav (2026, arXiv 2604.09675) reports a timing-feature AMD model that scored 99.3% on a 140-call expert-labeled set but about 88% during load testing on a broader production mix, which the authors attribute to AI screening services, carrier pre-announcements and very short greetings. The same paper found a frequency-domain beep detector nearly useless. Curated accuracy does not survive production, so your corpus must come from your real calls. Twilio's AMD docs also note that short voicemails often contain a pause that looks like the end of a human greeting at default settings.

Full mechanism and tuning: voicemail detection in LiveKit and Pipecat.

Consent, calling hours and abandonment

Outbound calls with an AI voice fall under the TCPA. The FCC's February 2024 declaratory ruling confirmed that AI-generated voices count as "artificial" voices under the TCPA. Several rules in 47 CFR 64.1200 become test cases:

• Calling hours. Telephone solicitations to residential subscribers cannot be made before 8 a.m. or after 9 p.m. local time at the called party's location. Test the scheduler with callee time zones, area codes that do not match the current location, and daylight saving transitions.
• Opt-out. Covered artificial-voice calls must offer an automated opt-out within 2 seconds of the identification. Test the timestamp, not just the words.
• Abandonment. For telemarketing, a call counts as abandoned if not connected within 2 seconds of the completed greeting, capped at 3% per campaign over 30 days. Ask counsel how this applies to your agent. Either way, P95 dead air after "Hello?" becomes a compliance metric, and slow synchronous AMD can blow through it.

None of this is legal advice; it is why outbound suites need timing assertions. Our regulated policy test cases and voice agent compliance audit go further.

Caller ID and answer rates

Outbound agents also fail before anyone speaks. In a lab study, Sherman et al. (NDSS 2020) found participants answered 35% of unknown-number calls with no warning, but only 3% to 5% when a spam warning was shown. Test from the numbers you will actually use, on major US carriers, and track answer rate per number. Setup and carrier tests are in LiveKit and Twilio outbound calling and Pipecat with Twilio and Telnyx.

Metrics that matter, with definitions

Define each metric precisely. The voice agent metrics scorecard covers the full set.

• Task success rate. Calls where the caller's goal was completed and the backend state is correct, divided by calls with a valid goal. Check state, not words. See task success rate.
• pass^k. From tau-bench (arXiv 2406.12045): the probability a case passes all k independent runs. For a case with n runs and c passes, pass^k = C(c, k) / C(n, k), averaged over cases. tau-bench reported pass^8 below 25% in its retail domain for strong function-calling agents, far below their single-run scores.
• Entity error rate. Entities transcribed or extracted wrong, divided by entities spoken. Report per entity type.
• Response gap. Time from the caller's end of speech to the agent's first audible audio, measured on the call recording. Report P50 and P95. This differs from framework metrics, which usually start the clock at the end-of-turn decision. See time to first audio.
• False interruption rate. Agent turns that started while the caller was still mid-utterance, divided by caller turns. See interruption rate.
• Silent tool failure rate. Tool errors or timeouts where the agent told the caller the action succeeded, divided by tool errors.
• Dead air rate. Calls with any silence over a threshold (for example 3 seconds) not caused by the caller. See silence rate and going-silent voice agents.
• Escalation accuracy. Correct transfers plus correct non-transfers, divided by calls where escalation was a valid option. See escalation accuracy.

Why pass^k and not plain pass rate? Because callers do not get a second try. A case with a 95% single-run pass rate has pass^5 of about 0.95^5 = 0.77. A suite that shows "95% passing" can still mean one in four callers on that path hits a failure across a week of repeated calls.

LLM judges are good at intent and tone and poor at exact facts, timing and backend state. Put hard checks in code. See LLM-as-judge limits.

A minimal test plan and pass/fail thresholds

You do not need 2,000 scenarios to start. You need the right 60, run the right number of times, against thresholds you wrote down before you looked at results.

The minimal suite

Start with this composition and grow it from production failures:

The three audio conditions are clean 16 kHz, 8 kHz phone band, and 8 kHz with background noise. That is roughly 60 cases and 300 to 400 runs per full pass, most of them cheap text runs.

Starting thresholds

These are starting points to tune, not industry standards. A pizza line and a collections agent should not share thresholds. The metric thresholds guide explains how to set them.

How many runs prove anything?

With zero failures in n independent runs, the 95% upper confidence bound on the failure rate is about 3/n (the rule of three). Ten clean runs still allow a 30% failure rate. Sixty clean runs bound it near 5%. That is why critical cases run several times per release and accumulate runs across releases.

Comparing two versions needs more. To detect a pass rate moving from 90% to 95% at 5% significance and 80% power, the standard two-proportion formula gives n = (1.96 + 0.84)² × (0.9 × 0.1 + 0.95 × 0.05) / 0.05² = 7.84 × 0.1375 / 0.0025 ≈ 431 runs per version. So an offline suite catches large regressions. Small ones need a canary on live traffic or batched simulations. See A/B testing voice agents.

Voice agent regression testing workflow

Regressions come from model bumps, STT upgrades, new TTS voices, endpointing values, tool schema edits and silent provider-side updates, not only prompts. Our post on LLM update regressions lists five ways model updates break live agents.

The workflow that holds up:

1. Version the bundle. Prompt, flow, tool schemas, model IDs, STT and TTS settings and turn-handling values ship together as one versioned config. Pin each call to one bundle at call start so production results are attributable.

2. Layer 1 on every commit. Text unit tests with pass^k on critical tags. Fast, cheap, blocking.

3. Layers 2 and 3 before merge or nightly. Audio replay on any change to STT, VAD or turn settings. Simulated phone calls for any release candidate.

4. Compare to the last known-good bundle, not to an absolute bar alone. Flag any metric that moved past its watch threshold, and diff the failing cases by tag.

5. Canary on live calls with automatic rollback guardrails on task success, dead air and escalation rate.

6. Score every production call, and turn each new failure into a case in the golden set.

Ship voice agent prompt changes safely covers bundles, canary sizing and rollback. Voice agent staging environments covers where layers 2 and 3 run, shadow testing covers replaying live traffic against a candidate, and voice agent test automation covers CI wiring. Your golden dataset is the asset that makes all of it work.

Framework-specific pointers

The layers are the same on every stack. The tools and the traps differ.

LiveKit

LiveKit Agents has pytest helpers with built-in judges, goal-driven Agent Simulations in text and audio modes, and `lk perf agent-load-test`. Endpointing settings now live under `turn_handling`, and workers are called agent servers. The trap: unit tests and text simulations skip VAD, STT and the turn detector, and some components behave differently in dev mode, on LiveKit Cloud and on a self-hosted agent server. Read the LiveKit voice agent testing guide, then debugging LiveKit agent latency and LiveKit noise and echo cancellation when self-hosted.

Pipecat

Pipecat has `run_test` for frame processors and, since version 1.4.0, Pipecat Evals for behavior. `PipelineTask` was renamed `PipelineWorker`, so older test code may exercise deprecated paths. The traps sit in VAD volume gating, Smart Turn release timing, 8 kHz resampling on phone serializers, and assistant context that keeps text the caller never heard after an interruption. Read the Pipecat voice agent testing guide, measuring Pipecat latency and Pipecat interruption strategy and Smart Turn.

Vapi and Retell

Managed platforms give you built-in testing, with limits worth knowing. Vapi's Test Suites connect an AI tester to your assistant over a real call and have an LLM score the transcript against a rubric. Vapi marks Test Suites as deprecated in favor of Simulations. Retell's simulation testing runs an AI-simulated user in a text conversation with success criteria and function mocks, which means it cannot test background noise or interruptions. Both grade transcripts, so timing and audio failures need another layer. See the Vapi voice agent testing guide and Vapi vs Retell.

A short example: a framework-agnostic regression gate

This harness is simplified and illustrative. It loads run results from any layer as JSON lines, computes pass^k per tag, applies thresholds and exits non-zero if a gate fails, so CI blocks the release. Your framework adapter (LiveKit `AgentSession.run`, Pipecat Evals, or a phone simulator) writes one line per run.

# regression_gate.py  (illustrative, simplified)
# Input: runs.jsonl, one line per run:
# {"case": "reschedule_tue", "tags": ["critical","date"], "passed": true,
#  "response_gap_ms": 940, "silent_tool_failure": false}
import json, sys
from collections import defaultdict
from math import comb

K = 4
GATES = {  # tag -> minimum pass^k; tune per use case
    "critical": 0.95,
    "money": 0.95,
    "date": 0.90,
}
MAX_GAP_P95_MS = 1500

def pass_hat_k(results: list[bool], k: int) -> float | None:
    n, c = len(results), sum(results)
    if n < k:
        return None  # not enough runs to estimate pass^k
    return comb(c, k) / comb(n, k)

def p95(values: list[float]) -> float:
    s = sorted(values)
    return s[min(len(s) - 1, int(round(0.95 * (len(s) - 1))))]

def main(path: str) -> int:
    by_case, case_tags = defaultdict(list), {}
    gaps, silent_failures = [], 0
    with open(path) as f:
        for line in f:
            r = json.loads(line)
            by_case[r["case"]].append(bool(r["passed"]))
            case_tags[r["case"]] = set(r.get("tags", []))
            if r.get("response_gap_ms") is not None:
                gaps.append(r["response_gap_ms"])
            silent_failures += bool(r.get("silent_tool_failure"))

    failed = []
    for tag, floor in GATES.items():
        scores = [pass_hat_k(runs, K) for case, runs in by_case.items()
                  if tag in case_tags[case]]
        scores = [s for s in scores if s is not None]
        if not scores:
            continue
        score = sum(scores) / len(scores)
        print(f"{tag:10s} pass^{K} = {score:.3f}  (gate {floor})")
        if score < floor:
            failed.append(f"{tag} pass^{K} {score:.3f} < {floor}")

    if gaps:
        gap = p95(gaps)
        print(f"response gap P95 = {gap:.0f} ms  (gate {MAX_GAP_P95_MS})")
        if gap > MAX_GAP_P95_MS:
            failed.append(f"gap P95 {gap:.0f} ms")
    if silent_failures:
        failed.append(f"{silent_failures} silent tool failures")

    for reason in failed:
        print("GATE FAILED:", reason)
    return 1 if failed else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Two choices matter more than the code. Report by tag, because a blended score hides the slice that hurts callers. Take `response_gap_ms` from the call recording on layer 3 runs, not from framework metrics, because the recording is what the caller heard. The tool-calling test cases guide includes a fuller harness with a LiveKit adapter.

How to test an AI voice agent before launch

1. Write the risk list. List the ten calls that would hurt most if they went wrong: wrong booking, missed disclosure, double charge, a caller trapped in a loop. Tag them `critical`.

2. Instrument first. Log per-turn timestamps, STT text, tool calls with arguments and results, and stereo recordings. Without these you cannot score layers 3 and 5.

3. Build layer 1. Turn the risk list and top intents into text unit tests with a mock backend that can fail. Run critical cases at least four times for pass^k.

4. Build the audio set. Record or synthesize entity-heavy utterances, convert them to 8 kHz phone band, add noise, include silent and line-noise clips. Score entity error and hallucination.

5. Run simulated phone calls. Drive scripted callers through your real number with pauses, backchannels, barge-ins and accents. Measure response gap, false interruptions and dead air from the recording.

6. Add outbound cases if you dial out. Labeled answer corpus for AMD, calling-hours scheduler tests, opt-out timing and caller ID checks.

7. Load test. Ramp concurrency to your expected peak plus margin, and confirm P95 response gap and call setup success hold at each step.

8. Write thresholds and gate. Set gate and watch thresholds per metric before running the release candidate. Wire the gate into CI.

9. Get an outside view. Have someone who did not build the agent run the suite on your real phone path. Teams grade their own agents on the paths they already know work.

10. Launch with a canary and score every call. Roll back on guardrail breaches, and turn each production failure into a new test case.

The voice agent testing checklist is the printable version of this plan, and the production readiness bar defines when you are done. This is also where independent evaluation helps most. Evalgent runs the pre-launch audit and the recurring regression suite over real calls, with scoring you did not write yourself, and keeps scoring production calls after launch. Read why independent evaluation matters for the reasoning.

Frequently asked questions

What is AI voice agent testing?

AI voice agent testing verifies a voice agent from audio in to audio out under realistic conditions. It covers what the agent hears, decides, does with tools and says, and the timing of each turn. It runs in five layers: text unit tests, audio replay, simulated phone calls, load tests and production scoring.

What is the difference between voice agent test cases and assertions?

A test case is the scenario: who calls, what they want, how they behave and under what audio conditions. An assertion is a checkable rule that decides pass or fail, such as "booking tool called with date 2026-10-13" or "disclosure spoken within 10 seconds." One case usually carries several assertions, mixing code checks and judge checks.

How do you do AI voice agent regression testing?

Version the full config as one bundle. Run text unit tests on every commit, audio replay and simulated phone calls on release candidates, and compare every metric to the last known-good bundle. Gate on pass^k for critical tags, canary on live calls with rollback guardrails, and add each production failure to the golden set.

How is outbound voice testing different from inbound?

Outbound agents must handle who answers: a person, voicemail, IVR or call screener. That adds answering machine detection tests on a labeled corpus, dead-air timing after "Hello?", calling-hours checks in the callee's time zone, opt-out timing, and caller ID and spam-label checks. Inbound testing focuses on routing, response gaps and transfers.

Can I test a voice agent with text-only tests?

Partly. Text tests cover the LLM, policy and tools cheaply and should hold most of your cases. They skip VAD, STT, turn-taking, TTS and telephony, which cause many production complaints. Add audio replay for STT and endpointing, and simulated phone calls for timing, barge-in and codec issues.

How many test runs do I need per case?

For a critical case, run it at least four times per release and track pass^k. With zero failures in n runs, the 95% upper bound on the failure rate is about 3/n, so 10 clean runs still allow 30%. Detecting a 90% to 95% change between versions takes roughly 430 runs per version.

What are voice evals?

Voice evals are scored evaluations of voice agent calls or turns against defined criteria: task success, tool correctness, policy adherence, entity accuracy, response gap and interruptions. They are the scoring half of voice agent testing. They run offline on test calls before release and online on production calls after launch.

Which metrics should gate a voice agent release?

Gate on pass^k for critical policy and tool cases, task success on happy paths, silent tool failures (zero tolerance), entity error rate on 8 kHz audio, P95 response gap measured from recordings, and false interruption rate. Set thresholds per use case before testing, and compare each release to the last known-good version.

The bottom line

Voice agent testing works when each failure layer is caught by the cheapest test layer that can see it, and each release is gated on repeated runs against thresholds you set in advance. Start with 60 well-chosen cases across five layers, score every production call, and let real failures grow the suite.