How to Evaluate Voice Variations With A/B Tests in Deepgram (2026)

Most voice tests fail before the first call connects. A team auditions six Aura-2 voices on laptop speakers, picks two, flips a coin per call, checks the dashboard daily, and ships whichever arm leads on day four. Every step has a known flaw. The Deepgram docs explain how to set a voice, not how to run an experiment on one.

This guide covers both halves. First, the Deepgram mechanics as of October 2026: the two TTS families, the exact Settings fields, the defaults that quietly change your control arm, and the events to log. Second, experiment design for voices: caller-level randomization, sample-size math, peeking, novelty, and the part almost nobody tests, which is how a voice change shifts echo and false barge-ins on the STT side.

If you want the general method for comparing agent versions or prompts, our A/B testing voice agents guide covers that. This post is only about the voice.

What a "voice" is in Deepgram in 2026

Deepgram now ships two text-to-speech families, and a voice A/B test has to treat them as different products rather than different voices. Per the Voice Agent TTS models page, the agent picks the TTS endpoint from the `version` field of `agent.speak.provider`:

The Aura-2 catalog on the voices page lists dozens of English voices with tags for accent (American, British, Australian, Filipino), age, and suggested use. Six are marked "featured": thalia, andromeda, helena, apollo, arcas, and aries. Several others, including harmonia, mars, neptune, phoebe, orpheus, and vesta, are tagged for customer service. Those tags are a starting shortlist, not evidence. They describe how the voice sounds in a studio rendering, not how it survives an 8 kHz phone line.

Aura-2 also covers Spanish, German, French, Dutch, Italian, and Japanese. For Spanish voices, the voice controls page recommends speed 0.9 to 1.5 because values below 0.9 "may introduce disfluencies." If your test includes a Spanish line, a slow-speed arm is already a known risk.

Five configuration gotchas that break a voice test

These come straight from the Deepgram docs footnotes. Each one can silently move calls into the wrong arm.

1. Omitting `agent.speak` does not give you Aura. The configure page states that Flux TTS with the `flux-kit-en` voice is the default when `agent.speak` is omitted. If your "control" arm is "whatever we had before" and that config has no speak block, your control may have changed under you.

2. `version` and `model` must change together. Flux voices are served only on `v2` and Aura voices only on `v1`. An arm config that sets `model: "flux-alexis-en"` but leaves `version` at its `v1` default asks for a voice that family does not serve.

3. Flux rejects compressed output. Requesting `mp3`, `opus`, `flac`, `aac`, or a `wav` container with Flux TTS returns `INVALID_SETTINGS`. If one arm is Flux and your web client asks for Opus, that arm fails at session start, and those calls vanish from your data. That is a sample-ratio mismatch waiting to happen.

4. Deepgram voices reject a `language` field. The language lives in the model name. Sending `agent.speak.provider.language` with a Deepgram voice returns `UNPARSABLE_CLIENT_MESSAGE`.

5. Fallback chains can switch your arm mid-call. `agent.speak` accepts an array of providers as an ordered fallback. When the first provider fails, the agent emits a `SPEAK_REQUEST_FAILED` warning and retries on the next one, per request. If you have a fallback configured, some turns in your "Aura-2 harmonia" arm may be spoken by a different voice. Log the warning and treat any call with a fallback turn as contaminated in the per-protocol analysis.

Switching the voice per session in the Voice Agent API

The Voice Agent runs on one WebSocket, `wss://agent.deepgram.com/v1/agent/converse`. The client waits for `Welcome`, sends a `Settings` message, waits for `SettingsApplied`, and only then streams audio, per the message flow docs. The voice is set once per session in that Settings message.

Here is an arm config for a Twilio-style phone leg. Everything except `agent.speak` must be byte-identical across arms:

{
  "type": "Settings",
  "tags": ["exp:voice-2026-10", "arm:B_treat"],
  "audio": {
    "input":  { "encoding": "mulaw", "sample_rate": 8000 },
    "output": { "encoding": "mulaw", "sample_rate": 8000, "container": "none" }
  },
  "agent": {
    "listen": { "provider": { "type": "deepgram", "version": "v2", "model": "flux-general-en" } },
    "think":  { "provider": { "type": "open_ai", "model": "gpt-4o-mini" }, "prompt": "..." },
    "speak":  { "provider": { "type": "deepgram", "version": "v1", "model": "aura-2-harmonia-en", "speed": 1.0 } },
    "greeting": "Thanks for calling. How can I help today?"
  }
}

Two details in that payload do real work. First, the `tags` array, which Deepgram documents as tags "for filtered searching." Put the experiment ID and arm in it so you can filter sessions by arm later without a database join. Second, the explicit `"version": "v1"`. It is the default for a provider you supply, but writing it out guards against the Flux default and documents the arm.

UpdateSpeak: useful, and a contamination risk

The `UpdateSpeak` message changes the speak model mid-conversation; the server finishes remaining audio and replies `SpeakUpdated`. It is how you would switch voices on a language change. In an A/B test it is a hazard: any code path that sends UpdateSpeak (a language detector, an escalation persona, a "calm voice" for angry callers) moves the caller out of their arm. Log every `SpeakUpdated` with the session ID, and decide before the test whether those calls stay in the intent-to-treat analysis. They should; the voice change is part of what the arm does.

Per-session voice in the standalone TTS API

If you run your own pipeline (Pipecat, LiveKit, or custom) and call Deepgram TTS directly, the voice is the `model` query parameter on `POST https://api.deepgram.com/v1/speak?model=aura-2-thalia-en&speed=0.9`. The REST response carries `dg-model-name` and `dg-speed-used` headers. Log both on every synthesis call. They are your proof that the arm you assigned is the arm the caller heard, which matters when someone asks why a "control" call sounds different in a recording.

Flux TTS runs on `/v2/speak`, and its WebSocket transport supports a mid-stream `Configure` to change speed without reconnecting, per the Flux TTS overview. Same rule as UpdateSpeak: if you use it adaptively, it is part of the treatment, and it needs logging.

What actually changes when you change the voice

Teams treat a voice swap as cosmetic, so they measure it with cosmetic metrics. Mechanically, a voice change touches five parts of the call.

1. Intelligibility on a narrowband channel. A phone leg at 8 kHz mu-law carries roughly 300 to 3,400 Hz. Deepgram's own guidance notes that for band-limited 8 kHz telephony audio, upsampling provides no benefit. A voice rendered at 24 kHz and auditioned on headphones keeps sibilants and breath detail that the phone path removes. Voices described as "breathy" or "raspy" in the catalog are the ones most likely to sound different after band-limiting. Test them through the same codec your callers hear.

2. Duration. Speed is a rate multiplier, so at 0.9 every agent utterance runs about 11% longer (1 / 0.9 = 1.11). Over a 12-turn call with 4 seconds of agent speech per turn, that is about 5 extra seconds of agent talk. Handle time moves even if nothing else does, and any metric with a fixed time window (like "hang-up within 10 seconds") shifts mechanically because the greeting itself is longer.

3. Caller speaking rate. People entrain to the system. In Bell, Gustafson, and Heldner (ICPhS 2003), 16 subjects talking to a simulated dialogue system with either a slow or fast speaking rate adapted their own rate to the system's, and none appeared aware of it afterward. Repair sequences, where users had to repeat themselves, elicited slower speech. For a voice agent, that means a faster voice can produce faster caller speech, which changes what your STT and endpointing see. The voice arm leaks into the listen side.

4. TTS latency. Different voices and families can have different time-to-first-byte. The Voice Agent emits a `LatencyReport` after each turn with `tts_latency` (first text token to first audio byte) and `total_latency` (user utterance end to first audio byte). Every field is optional and in seconds, so log defensively. Compare p50 and p95 `tts_latency` per arm; a voice that wins on containment but adds 150 ms at p95 is a trade-off you should see explicitly.

5. Echo and barge-in. This is the least tested interaction, so it gets its own section.

The voice and barge-in interaction nobody tests

Barge-in is an STT decision: the system hears caller speech while the agent is talking and stops playback. Anything that leaks the agent's own voice back into the caller's audio can trigger it. That leak path is where the voice matters.

Deepgram's audio preprocessing and barge-in guide gives the mechanics:

• Echo cancellation "prevents the agent's own TTS output from being captured by the microphone and re-transcribed, which causes the agent to respond to itself."
• AEC algorithms adapt once audio starts flowing, and "having the agent speak an opening greeting helps the AEC calibrate before the customer responds."
• Aggressive AEC "makes barge-in harder," because it can suppress the caller along with the echo.
• With Flux STT, every `StartOfTurn` is guaranteed to contain a non-empty transcript, so silence and noise cannot trigger it. With Nova-3, Deepgram recommends requiring at least 2 words in an interim result, or a final result with at least 1 word, before treating it as barge-in.
• As a software-level check, compare STT output against the TTS text the agent just spoke and discard matches.

Now connect that to a voice swap. A louder, brighter, or more energetic voice puts more energy into the band the phone keeps, so residual echo after AEC is louder on speakerphone and car calls. A faster voice with a shorter greeting gives the AEC less audio to adapt on before the caller first speaks. A voice with more pitch movement can be harder for a linear echo canceller to track. None of these are certain for a specific voice, but each predicts the same symptom: more false barge-ins in one arm, which reads as "callers interrupt this voice more" when the agent is actually interrupting itself.

That is why false barge-in is a guardrail metric in this test, separate from total barge-in rate. Define it from server events:

• Barge-in: a `UserStartedSpeaking` event received between the first agent audio frame of a turn and that turn's `AgentAudioDone`.
• False barge-in: a barge-in where the next user `ConversationText` is empty or one word, or fuzzy-matches the agent text just spoken (echo), or no user turn follows within 2 seconds.

If you use Flux TTS directly, its `Interrupt` message reports `text_spoken`, meaning exactly what the caller heard before the cut. That gives you a second measure: the share of each agent sentence delivered before interruption, per arm.

Run the echo check offline first. Play each candidate voice through a speakerphone-in-a-room setup and a car-cabin recording, feed the caller channel to your STT config, and count false triggers per minute of agent speech. Our guide to testing VAD misfires covers the rig, and the interruption rate guide covers how to read the live metric.

Offline listening tests first: the cheap pre-filter

A live test costs thousands of calls per arm; a listening test costs a few hundred judgments. Use the cheap one to cut ten voices to two.

MUSHRA, adapted for phone audio

ITU-R BS.1534-3 defines MUSHRA (Multiple Stimuli with Hidden Reference and Anchor). Listeners rate several stimuli side by side on a 0 to 100 scale against a labeled reference. The set includes a hidden copy of the reference and low-pass anchors, typically at 3.5 kHz and 7 kHz. Post-screening excludes assessors who rate the hidden reference below 90 for more than 15% of items.

Here is the twist for voice agents: a 3.5 kHz low-pass anchor is close to what a narrowband phone call sounds like. On a phone line, the "anchor" is your production channel. So adapt the protocol:

• Render every candidate voice at its native rate, then pass it through your real path (for example, 8 kHz mu-law via your telephony provider, or a G.711 encode-decode in software). Those band-limited files are the stimuli.
• Use a recording of your best human agent reading the same script, through the same phone path, as the reference.
• Use 20 to 30 scripts drawn from real call turns: numbers, names, addresses, confirmation readbacks, and an apology. Avoid demo copy.

Two research findings shape how you read the results. Wester, Valentini-Botinhao, and Henter (Interspeech 2015) re-analyzed Blizzard Challenge 2013 data and found that a MOS naturalness test reached a stable level of significance only above 30 listeners, while more than 60% of Interspeech 2014 papers drew conclusions from fewer than 20. And Rethinking MUSHRA (Varadhan et al., 2024), with 492 listeners, found that MOS tests fail to separate similar systems and that MUSHRA raters show reference-matching bias, penalizing TTS that sounds better than the human reference. For a voice shortlist, the practical reading is: use at least 30 raters, keep the human reference for calibration only, and rank candidates against each other.

For more on why single-number naturalness scores mislead, see our mean opinion score guide and the broader TTS evaluation framework.

Paired preference tests: the math

For the final two voices, a forced-choice A/B preference test is faster than MUSHRA. Each judgment is "which one would you rather hear on a support call?" You test whether the preference share differs from 50%.

The sample size for a one-sample proportion test against 0.5:

n = ( z(1-α/2) × 0.5 + z(1-β) × √(p(1-p)) )² / (p - 0.5)²

At α = 0.05 (z = 1.960) and 80% power (z = 0.842):

• To detect a 60/40 preference: n = (0.980 + 0.842 × 0.490)² / 0.01 = 194 judgments
• To detect a 55/45 preference: n = (0.980 + 0.842 × 0.497)² / 0.0025 = 783 judgments

Spread those judgments across at least 30 raters and many scripts, and randomize left/right order, so you are not measuring one rater's taste or one sentence. A 55/45 preference on a listening test is a weak signal for live outcomes. Treat it as permission to proceed to the live test, not as a result.

Experiment design for the live test

Randomize by caller, not by call

Randomizing each call independently is the default in most routing code, and it is wrong for voice tests. Repeat callers hear both voices. Their second call is shaped by the first: they know the menu, they remember the agent misheard their account number, they have adjusted their speaking rate to the first voice. That is interference between arms, and it biases the effect toward zero.

Assign by a keyed hash of the normalized caller number so the same caller hears the same voice on every call:

digest = hmac.new(SALT, caller_e164.encode(), hashlib.sha256).digest()
u = int.from_bytes(digest[:8], "big") / 2**64   # uniform in [0, 1)
arm = "A_control" if u < 0.5 else "B_treat"

A new salt per experiment reshuffles callers, so one test's assignment does not carry into the next. Store the hash, not the number, in your analysis tables.

Caller-level assignment has a statistical cost. Calls from the same caller are correlated, so a call-level standard error understates uncertainty. The inflation is the design effect:

DE = 1 + (m - 1) × ICC

where m is the average calls per caller in the test window and ICC is the intra-caller correlation of the outcome. As a worked example, with m = 1.6 and ICC = 0.3, DE = 1.18, so you need 18% more calls than the naive formula says. With m = 2.5 and ICC = 0.5 (a billing line with chronic repeat callers), DE = 1.75. Estimate ICC from a month of pre-test data before you size the test, and analyze with caller-clustered standard errors, as the script below does.

Stratify by line, time, and language

Containment on a billing line and an order-status line can differ by 20 points. If one arm happens to get more billing traffic in a small test, line mix shows up as a voice effect.

Hash assignment is balanced in expectation, not in every stratum. Three practices fix that:

• Pre-specify strata: inbound number or line, business hours versus after hours, and language. These are the variables most likely to move the outcome.
• Analyze post-stratified: compute the arm difference within each stratum and combine with stratum weights. This recovers precision that line mix would otherwise eat.
• Run whole weeks: Monday callers are not Saturday callers. Start and stop on the same weekday.

Then check for sample-ratio mismatch (SRM) daily. Count unique callers per arm and run a chi-square test against the planned split. A p-value below 0.001 means the pipeline is broken, not that the voice works. Common voice-test causes: one arm's config throws `INVALID_SETTINGS` (the Flux-with-Opus case), a fallback provider catches some sessions, or calls that drop before `SettingsApplied` are never logged with an arm.

Pick one primary metric and several guardrails

The primary metric is the one number the decision rides on. For most support lines, that is containment or task completion; see our containment rate guide for definitions that survive callbacks. Guardrails are metrics the voice must not make worse beyond a margin, even if the primary improves.

The early hang-up definition is anchored to the end of the greeting on purpose. A fixed "first 10 seconds of the call" window penalizes the slower voice, because its greeting is still playing at second 8. Anchoring to `AgentAudioDone` removes that bias. The abandonment rate guide covers the wider family of early-exit metrics.

CSAT is secondary because survey response is low and self-selected, and the voice itself may change who answers the survey. Our CSAT guide explains how to read it without overreacting. The margins in the table are illustrative starting points; set yours from the business cost of each failure before the test starts, and write them down.

Worked sample-size math

Proportions: detecting 70% to 73% containment

The standard two-proportion formula, per arm:

n = ( z(1-α/2) × √(2 p̄ (1 - p̄)) + z(1-β) × √(p1(1-p1) + p2(1-p2)) )² / (p2 - p1)²

Plug in p1 = 0.70, p2 = 0.73, p̄ = 0.715, α = 0.05 two-sided, power 80%:

• √(2 × 0.715 × 0.285) = √0.40755 = 0.6384; × 1.960 = 1.2512
• √(0.21 + 0.1971) = √0.4071 = 0.6380; × 0.842 = 0.5370
• (1.2512 + 0.5370)² = 3.1977
• 3.1977 / 0.0009 = 3,554 per arm, about 7,100 total

Apply the design effect: at DE = 1.18 that becomes about 4,190 calls per arm. At 1,000 eligible calls a day split evenly, the fixed-horizon test takes about 8.4 days, which you round up to two full weeks so you cover two weekly cycles.

The cost of chasing small effects is steep:

Halving the detectable lift roughly quadruples the sample. If your line takes 300 calls a day, a +1 point voice effect is not testable live in any reasonable window. Decide on offline and synthetic evidence instead, or accept that you are only able to detect large effects.

Guardrails need their own sizing. Early hang-ups are rare, so detecting 4% rising to 5% needs about 6,745 callers per arm, nearly twice the primary. In practice you size on the primary and treat guardrails as non-inferiority checks with a confidence interval, accepting that the interval will be wide for rare events.

Continuous metrics: handle time

For a continuous metric with standard deviation σ, the per-arm sample to detect a difference δ is n = 2 (z(1-α/2) + z(1-β))² σ² / δ².

Assume average handle time has σ = 150 s (illustrative; measure yours). To detect a 10-second change: n = 2 × 7.849 × 22,500 / 100 = 3,532 per arm. To detect 15 seconds: 1,570 per arm. Remember the duration mechanism from earlier: a speed change of 0.9 alone adds seconds of agent talk per call, so a handle-time "effect" from a slower voice is partly arithmetic.

Cutting the sample with CUPED

Deng, Xu, Kohavi, and Walker (WSDM 2013) introduced CUPED, which adjusts each unit's outcome using a pre-experiment covariate. On Bing it reduced variance by about 50%, roughly halving the required sample. For voice tests the natural covariate is the caller's containment rate in the 60 days before the test. It only exists for returning callers, so the gain depends on your repeat share; first-time callers get a covariate of the population mean. The covariate must be measured before assignment, or it can absorb part of the treatment effect.

Peeking and sequential testing

A fixed-horizon p-value is valid only at the pre-planned sample size. Johari, Pekelis, and Walsh show that continuous monitoring makes those p-values "wholly unreliable," and their numerical appendix finds Type I error inflated above 2x even at 10,000 observations, a common A/B test size.

You can reproduce the effect for a voice test in a few lines. In an illustrative simulation (3,000 A/A runs, both arms at 70% containment, 500 callers per arm per day, a two-sided z-test checked every day):

• Checked only at day 14: false positive rate 4.5%, as designed.
• Checked every day for 14 days, stopping at the first p < 0.05: 20.4%.
• Checked every day for 28 days at 250 per day: 27.6%.

One in five "winning" voices under daily peeking would be noise.

The fix: always-valid p-values (mSPRT)

The Johari et al. method, deployed on a large commercial A/B platform, uses a mixture sequential probability ratio test. For a difference in means D with estimated variance V and a normal mixing distribution N(0, τ²), the statistic is:

Λ = √(V / (V + τ²)) × exp( τ² D² / (2 V (V + τ²)) )

The always-valid p-value is p_t = min(p_(t-1), 1 / Λ_t). You may look at it as often as you like and stop the first time it drops below α. The mixing variance τ² encodes the effect size you expect; for a containment test, τ = 0.02 (2 points) is a reasonable choice.

The price is power at any fixed sample. In the same illustrative simulation with a true 3-point lift:

The trade is clear. A fixed-horizon test hits 80% power at 7 days but forbids looking. An mSPRT needs a longer maximum horizon (about 13 days here) to reach the same power, but when the effect is real it typically stops in about a week, and you can kill a harmful arm on day two without inflating error. For voice tests, where a bad voice costs real calls, that early-stop option on guardrails is worth more than the extra days.

Novelty effects: week one lies

A new voice on an existing line is a change returning callers notice. Sadeghi et al. (Microsoft, 2021) describe two forms of user learning: novelty, where engagement with something new fades over time, and primacy, where users perform better as they adapt. Both make a short test's effect differ from the long-run effect.

In a voice test, novelty shows up as returning callers reacting to the change itself, for example staying on the line out of curiosity or hanging up because "the agent sounds different." Primacy shows up as returning callers getting better with the new voice as they adapt to its pacing, which the entrainment research predicts.

Three ways to handle it:

• Split by exposure. Report the effect separately for first-ever callers (no prior voice to compare) and for returning callers. First-ever callers give the cleanest read of the steady-state effect.
• Plot effect by days since first exposure. If the treatment effect for returning callers shrinks or grows across weeks one to three, you are seeing user learning, not a stable effect.
• Keep a holdout. After shipping, leave 5% of callers on the old voice for a month. If the gap closes, the week-one result was novelty.

How to A/B test voices in Deepgram

1. Freeze everything except the voice. Snapshot the Settings message, prompt, LLM, STT model, endpointing thresholds, and greeting text. Only `agent.speak.provider` differs between arms. Hash the frozen config and log the hash per session.

2. Verify each arm's config. Open a session per arm, confirm `SettingsApplied`, synthesize the greeting, and check the output format, sample rate, and `dg-model-name` or arm tag. Fail any arm that can return `INVALID_SETTINGS` in any client you support.

3. Run a narrowband listening test. Render 20 to 30 real call scripts through your phone path, run MUSHRA with at least 30 screened raters, and keep the top two or three voices.

4. Run an echo and barge-in check. Play each finalist through speakerphone and car-cabin conditions into your real STT config and count false barge-ins per minute of agent speech. Drop any voice that is clearly worse.

5. Run a synthetic call batch. Drive the finalists through the same scripted scenarios with synthetic callers across accents and noise levels, measuring task success, repeat requests, and `tts_latency`. Our Deepgram load testing guide covers running these at concurrency.

6. Pre-register the live test. Write down the primary metric, guardrails and margins, strata, sample size with design effect, maximum duration, and the stopping rule (fixed horizon or mSPRT with τ).

7. Launch with caller-hashed assignment. Start on a weekday, tag sessions with arm and experiment ID, and check SRM daily.

8. Analyze with caller-clustered errors. Compute the primary effect with a confidence interval, check each guardrail's interval against its margin, and split results by stratum and by first-ever versus returning callers.

9. Ship with a holdout and re-test on change. Keep a small holdout for a month, and re-run the protocol whenever Deepgram updates the voice family, your prompt changes length or style, or you change telephony codecs.

The assignment and analysis script

This is simplified and illustrative: the Settings shape follows the configure docs, the connection uses `websockets` 13 or newer, and the analysis uses only numpy and pandas. Verify field names against current docs before production use.

"""voice_ab.py - caller-level voice A/B assignment + analysis (illustrative, simplified)."""
import hashlib, hmac, json, math
import numpy as np
import pandas as pd
from statistics import NormalDist

SALT = b"voice-test-2026-10-aura2"          # new salt per experiment
ARMS = {
    "A_control": {"type": "deepgram", "version": "v1", "model": "aura-2-thalia-en",   "speed": 1.0},
    "B_treat":   {"type": "deepgram", "version": "v1", "model": "aura-2-harmonia-en", "speed": 1.0},
}
SPLIT = [("A_control", 0.5), ("B_treat", 0.5)]
Z = NormalDist()

def assign_arm(caller_e164: str) -> str:
    digest = hmac.new(SALT, caller_e164.strip().encode(), hashlib.sha256).digest()
    u, cum = int.from_bytes(digest[:8], "big") / 2**64, 0.0
    for arm, share in SPLIT:
        cum += share
        if u < cum:
            return arm
    return SPLIT[-1][0]

def build_settings(arm: str, prompt: str) -> dict:
    return {
        "type": "Settings",
        "tags": ["exp:voice-2026-10", f"arm:{arm}"],
        "audio": {"input":  {"encoding": "mulaw", "sample_rate": 8000},
                  "output": {"encoding": "mulaw", "sample_rate": 8000, "container": "none"}},
        "agent": {
            "listen": {"provider": {"type": "deepgram", "version": "v2", "model": "flux-general-en"}},
            "think":  {"provider": {"type": "open_ai", "model": "gpt-4o-mini"}, "prompt": prompt},
            "speak":  {"provider": ARMS[arm]},
            "greeting": "Thanks for calling. How can I help today?",
        },
    }

async def run_session(caller_e164: str, prompt: str, api_key: str, log):
    import websockets
    arm = assign_arm(caller_e164)
    url = "wss://agent.deepgram.com/v1/agent/converse"
    async with websockets.connect(url, additional_headers={"Authorization": f"Token {api_key}"}) as ws:
        welcome = json.loads(await ws.recv())            # never send before Welcome
        await ws.send(json.dumps(build_settings(arm, prompt)))
        async for msg in ws:
            if isinstance(msg, bytes):
                continue                                  # agent audio -> telephony leg
            evt = json.loads(msg)                         # SettingsApplied, UserStartedSpeaking,
            log({"caller_hash": hashlib.sha256(caller_e164.encode()).hexdigest()[:16],
                 "arm": arm, "request_id": welcome.get("request_id"), **evt})
                                                          # ConversationText, AgentAudioDone,
                                                          # LatencyReport, Warning, Error

def srm_check(df):
    counts = df.groupby("arm")["caller"].nunique().reindex([a for a, _ in SPLIT]).fillna(0).values
    exp = np.array([s for _, s in SPLIT]) * counts.sum()
    chi2 = float(((counts - exp) ** 2 / exp).sum())
    return chi2, 2 * (1 - Z.cdf(math.sqrt(chi2)))       # 1 df; p < 0.001 = broken pipeline

def clustered_rate(df, metric):
    g = df.groupby("caller").agg(y=(metric, "sum"), n=(metric, "size"))
    k, r = len(g), g.y.sum() / g.n.sum()
    var = ((g.y - r * g.n) ** 2).sum() / (k * (k - 1)) / (g.n.mean() ** 2)
    return r, var

def compare(df, metric, control="A_control", treat="B_treat"):
    rc, vc = clustered_rate(df[df.arm == control], metric)
    rt, vt = clustered_rate(df[df.arm == treat], metric)
    diff, var = rt - rc, vc + vt
    se = math.sqrt(var)
    return {"metric": metric, "diff": diff, "ci95": (diff - 1.96 * se, diff + 1.96 * se),
            "p_fixed": 2 * (1 - Z.cdf(abs(diff) / se)), "var": var}

def msprt_p(diff, var, tau2=0.02**2):
    lam = math.sqrt(var / (var + tau2)) * math.exp(tau2 * diff**2 / (2 * var * (var + tau2)))
    return min(1.0, 1.0 / lam)                            # keep running min across looks

GUARDRAILS = {"hangup_10s": 0.010, "false_bargein": 0.020, "repeat_req": 0.020}

def decide(df, primary="contained"):
    rows = [compare(df, primary)]
    for m, margin in GUARDRAILS.items():
        r = compare(df, m)
        r["guardrail_ok"] = r["ci95"][1] < margin         # upper bound below margin
        rows.append(r)
    return pd.DataFrame(rows)

Your call log needs one row per call with the hashed caller, arm, and 0/1 columns for each metric, derived from the server events listed in the session loop. The `clustered_rate` function treats each caller as a cluster, which is the analysis-side half of caller-level randomization. For a list of fields worth persisting, see what to log on every voice agent call.

Reading the result: a decision matrix

A flat result is a real answer. It tells you the voice choice is low-stakes for this line at the effect size you could detect, which frees you to pick on latency or cost. What you should not do is run a flat test longer and longer until something crosses 0.05; that is peeking by another name.

Failure modes seen in voice tests

• The audition-room winner. Voices chosen on studio-rate audio through laptop speakers, never heard at 8 kHz.
• Silent control drift. The control config lacks `agent.speak`, so it changed to the Flux default when Deepgram changed the default.
• One arm fails at session start. A format mismatch returns `INVALID_SETTINGS` for one arm only; the dashboard shows fewer calls, not errors.
• Mid-call voice changes. UpdateSpeak or a fallback provider changes the voice during the call, and nobody logs it.
• Call-level randomization. Repeat callers hear both voices; the effect shrinks toward zero and the standard errors are too small.
• Fixed-window guardrails. "Hang-up in 10 seconds" penalizes the slower voice because its greeting is longer.
• Echo blamed on callers. Higher barge-in in one arm read as "callers dislike it" when the agent is interrupting itself.
• Daily peeking with fixed p-values. A one-in-five chance of crowning noise.
• Testing the voice on a stale stack. STT model, endpointing, or prompt changed during the test window. Freeze the stack, and see our Deepgram STT testing guide for what to pin on the listen side.

Where independent evaluation fits

The voice test has three places where a second, independent set of eyes catches problems a team inside the project tends to miss. Before launch, Evalgent can run the offline gates (narrowband listening panels, echo and false barge-in checks, and synthetic call batches across accents and noise) so only two voices reach live traffic. During the live test, an outside review of arm configs, assignment logic, and guardrail definitions catches SRM causes and biased metrics before thousands of calls are spent. After launch, scoring production calls on the same metrics shows whether the winning voice holds up once novelty fades.

Frequently asked questions

How do you evaluate different voice variations through A/B tests in Deepgram?

Freeze the Settings message except `agent.speak.provider`, and give each arm its own version, model, and speed. Filter candidates with a narrowband listening test and an echo check. Then assign callers by hashed phone number, pre-register containment as the primary metric with guardrails, size the test with a design effect, and analyze with caller-clustered standard errors.

How do you change the voice per session in the Deepgram Voice Agent API?

Set `agent.speak.provider` in the Settings message sent after `Welcome`. For Aura-2 use `version: "v1"` and a model like `aura-2-thalia-en`; for Flux TTS use `version: "v2"` and a model like `flux-alexis-en`. Add the arm to the `tags` array. `UpdateSpeak` changes the voice mid-call, which contaminates an A/B arm.

What is the difference between Deepgram Flux TTS and Aura-2?

Flux TTS (`v2`) is streaming-first, with a turn-based lifecycle, cross-turn voice consistency, speed from 0.5 to 1.5, and beta expressivity. It outputs raw audio only. Aura-2 (`v1`) has the broadest voice catalog, speed from 0.7 to 1.5, and supports compressed formats. Deepgram defaults to Flux `flux-kit-en` when `agent.speak` is omitted.

How many calls do you need to A/B test a TTS voice?

At α = 0.05 and 80% power, detecting containment moving from 70% to 73% needs about 3,554 callers per arm before clustering. A +2 point lift needs 8,080, and +1 point needs 32,647. Multiply by the design effect, 1 + (m − 1) × ICC, if callers repeat during the test.

Should you randomize a voice test by caller or by call?

By caller. Per-call randomization lets repeat callers hear both voices, so their behavior on one arm is shaped by the other. That biases the effect toward zero and understates the variance. Hash the normalized caller number with a per-experiment salt so each caller always hears the same voice, then cluster standard errors by caller.

Can you check voice A/B results every day?

Not with ordinary p-values. In an illustrative A/A simulation with daily checks over 14 days, the false positive rate rose from 5% to about 20%. Use always-valid p-values from a mixture sequential probability ratio test (Johari et al.), which allow continuous monitoring. Or fix the sample size in advance and look only at the end.

Does the TTS voice affect barge-in and false interruptions?

Yes. Agent audio that leaks back through speakerphone or car echo can reach STT and trigger barge-in, so the agent interrupts itself. Louder or brighter voices and shorter greetings, which give AEC less time to adapt, can raise that rate. Track false barge-ins as a separate guardrail, using `UserStartedSpeaking` timing and echo-matching follow-up text.

How do you run a listening test for TTS voices before a live test?

Use MUSHRA from ITU-R BS.1534-3 with stimuli rendered through your real phone path. Include a hidden reference and low-pass anchors, and screen out raters who score the hidden reference below 90 on more than 15% of items. Research suggests at least 30 listeners. For two finalists, a 60/40 preference needs about 194 paired judgments.

The bottom line

A Deepgram voice A/B test is only as good as its plumbing and statistics: pinned arm configs, caller-level assignment, a pre-registered primary metric with guardrails, and a stopping rule that tolerates the daily dashboard check. Filter on narrowband listening and echo tests first, then spend live traffic only on the two voices that survive.