Independent Voice AI Evaluation: Why Self-Reported Metrics Fail and What Counts as Independent

Every voice AI vendor publishes numbers. Sub-second latency. Ninety-something percent containment. Word error rates in the low single digits. Those numbers are rarely false. They are measured on the vendor's definition, on the vendor's test set, after the vendor picked the best of several runs. That is enough to make them useless for a purchase decision.

Other fields learned this lesson decades ago and wrote rules for it. Financial auditors may not design the systems they audit. Testing labs must prove impartiality to stay accredited. Clinical trials must register their primary outcome before the first patient enrolls. Voice AI has none of these rules yet, so buyers have to write them into their own process.

This guide gives you those rules. It explains the specific mechanisms that make self-reported voice metrics drift from production, what the research on conflicts of interest and benchmark gaming found, the six tests an evaluation must pass to count as independent, the regulations that now require evidence rather than claims, and a checklist and RFP clause you can copy. It is the hub for our pieces on why vendor-reported metrics mislead and the vendor scorecard.

What "independent" means, in testable terms

Independent voice AI evaluation: an assessment of a voice agent by a party with no technical, managerial, or financial stake in the result, run on the buyer's data under a plan fixed before any results exist.

The definition comes from software assurance, not marketing. IEEE 1012, the standard for system and software verification and validation (current edition 1012-2024), defines independent V&V along three axes. Each one maps to a question a buyer can ask.

The NIST AI Risk Management Framework says the same thing in its own words. Subcategory MEASURE 1.3 calls for "internal experts who did not serve as front-line developers" or independent assessors to be involved in regular assessments. Neither document says the vendor's own QA team counts.

A vendor can pass none of the three tests on its own product. Your internal team can pass the first two if it is separate from the integration team. A third party passes all three only if its contract and fees do not depend on which vendor wins.

Why self-reported voice metrics diverge from production

Vendor numbers drift from production through five mechanisms. None requires dishonesty. Each is a choice that looks reasonable inside the vendor's lab and happens to favor the vendor.

Mechanism 1: the metric is defined where the vendor looks best

Latency. Most vendor dashboards report a server-side interval, such as the time from a final transcript to the first synthesized audio byte leaving the server. The caller experiences mouth-to-ear delay: the time from when they stop talking to when they hear the first sound of the reply. ITU-T G.114 frames telephony delay in mouth-to-ear terms for exactly this reason.

The hops outside the vendor's interval are the ones a server-side metric cannot see. They include the inbound media path, the endpointing silence timer that waits to decide the caller is finished, STT finalization, the outbound media path with jitter buffering, and device playout. In the illustrative budget below, a dashboard that reports 500 ms sits inside a turn gap the caller feels as roughly 1,320 ms.

The fix is to measure from the caller's side: record both channels on the far end of the phone call and time the gap between the end of caller speech and the start of agent speech. Our time to first audio guide covers the measurement in detail.

Containment. "Contained" usually means the call ended without a transfer to a human. That definition counts a caller who hangs up in frustration as a success. Here is a worked example with illustrative numbers. Of 10,000 calls, 6,200 end without transfer, so reported containment is 62%. Suppose 900 of those callers hung up before the task finished and 700 more called back within 72 hours about the same issue. Verified resolution is (6,200 − 900 − 700) / 10,000 = 46%. Same calls, a 16-point gap, and nobody lied. The containment rate guide gives the full definitions.

Word error rate. WER is sensitive to three choices the vendor controls: the audio, the segmentation, and the text normalization.

On audio, Szymański et al. (Findings of EMNLP 2020) compared three commercial ASR systems on 50 real call-center conversations recorded at 8 kHz. While published results on LibriSpeech, a corpus of 16 kHz audiobooks, sat at 2-4% WER, the commercial systems scored 16.5-19.2% on the real calls. On the HUB5 CallHome phone benchmark they scored roughly double the best published result. The authors also named a quieter cause. Published numbers usually use oracle segmentation, where the recognizer receives pre-cut speech segments. They fed whole 5-minute chunks instead, forcing each system to detect speech itself, which is what happens in a live call.

On normalization, try this yourself. The reference "i would like to pay two hundred fifty dollars on the third" against the hypothesis "I'd like to pay $250 on the 3rd." scores 58% WER with no normalization, even though the transcript is correct. Meanwhile "four four seven one" transcribed as "four four seven nine" scores only 11% WER on a nine-word sentence, even though the account number is wrong. WER punishes the correct transcript and forgives the costly error.

Test sets also tend to omit the audio that breaks recognizers. Koenecke et al. (FAccT 2024) ran more than 13,000 AphasiaBank clips through Whisper. Roughly 1% of transcriptions contained whole hallucinated phrases that were never spoken, and 38% of those hallucinations carried explicit harms. Hallucinations were more common for speakers with aphasia, whose speech has more long pauses. A clean read-speech benchmark will never surface that failure. That is why entity accuracy matters more than WER for voice agents; see STT entity accuracy and word error rate for voice agents.

Task success. If an LLM judge decides whether a call succeeded, the judge's biases become the metric. Panickssery et al. (NeurIPS 2024) found that GPT-4 and Llama 2 judges favor text they generated themselves, and that the strength of this self-preference rises with the model's ability to recognize its own output. Wang et al. (2023) showed position bias is just as large: with ChatGPT as the judge, Vicuna-13B beat ChatGPT on 66 of 80 queries depending only on answer order. A vendor that judges its own agent with its own model family has both biases working in its favor. Our LLM-as-judge limits piece covers the mitigations.

Mechanism 2: best-of-k selection on a small test set

The second mechanism is quieter. A vendor tries several prompt versions, endpointing settings, or model snapshots, scores each on the same test set, and reports the best. Even if every variant is equally good, the maximum of k noisy scores sits above the true mean.

The Leaderboard Illusion (Singh et al., 2025) documented this at industry scale. Some providers privately tested many variants on Chatbot Arena and published only the best. At the extreme, one provider tested 27 private variants before a single public release. Dodge et al. (EMNLP 2019) made the same point for research: a reported score means little unless you also report how many configurations were tried.

You can compute the inflation. Assume true task success is 80% and the test set has n calls. The standard error is √(0.8 × 0.2 / n), which is 2.8 points at n = 200 and 4.0 points at n = 100. The expected maximum of k independent standard normal draws is about 1.16 for k = 5, 1.54 for k = 10, and 2.00 for k = 27. Multiply and you get the inflation with zero real improvement.

A vendor quoting 86% on a 100-call POC set after ten tuning rounds may have an 80% agent. The defense is a held-out set the vendor never sees until the scored run.

Mechanism 3: Goodhart's law, in four flavors

Goodhart's law says a measure stops being a good measure once it becomes a target. Manheim and Garrabrant (2018) split it into four variants. Each one has a voice AI form, and each needs a different control.

The practical rule: never accept a single headline metric. Every metric a vendor optimizes needs a paired counter-metric that moves the wrong way when the first one is gamed.

Mechanism 4: contamination of the test set

In LLM research, contamination means test items leaked into training data. Zhang et al. (NeurIPS 2024) built GSM1k, a fresh copy of the GSM8k math benchmark matched on difficulty. Some model families dropped by up to 8 points on the fresh copy. The authors found a positive relationship (Spearman's r² = 0.36) between a model's likelihood of generating GSM8k items and its gap between the two sets. Frontier models showed minimal overfitting, which is the point. You only learn which models are clean by testing on data they could not have seen.

Voice AI has its own contamination path, and it runs through your POC. You hand the vendor 200 sample calls. Their solutions engineer tunes prompts, adds keywords to the STT vocabulary, and writes intent examples from those exact calls. The POC score is now a training score. The fix is contractual and mechanical: give the vendor a development set to tune on, keep a disjoint held-out set, and score only on the held-out set. The golden dataset guide covers how to build and split one.

Mechanism 5: the outcome switches after the results arrive

Clinical research has a name for changing the metric after seeing the data: outcome switching. The COMPare project (Goldacre et al., Trials 2019) checked 67 trials in five top medical journals against their registered protocols. Of those, 58 (87%) had discrepancies serious enough to need a correction letter, and trials added a mean of 5.4 outcomes that had never been declared.

Voice POCs switch outcomes constantly. A bake-off starts with "task success" as the decider. When the favored vendor loses on it, the readout shifts to "naturalness" or "customer sentiment." The defense is the same as in medicine: write the primary metric, the threshold, and the analysis down before the first call, and hash the document so nobody can edit it later.

What other fields learned about grading your own work

The voice AI industry does not need to invent independence from scratch. Four mature disciplines solved versions of this problem, and each one hands voice AI a specific control.

Financial audit: the evaluator may not build what it grades

After the Enron and WorldCom collapses, Sarbanes-Oxley Section 201 barred audit firms from providing eight categories of non-audit services to audit clients. Those categories include designing and implementing financial information systems. Section 203 required lead and concurring audit partners to rotate off a client after five years, and SEC rules added a five-year time-out before they return (SEC, 2003).

The voice AI translation is direct. A firm that builds or tunes your voice agent should not also certify it. Over a long relationship, rotate the people who write test scenarios, and refresh the held-out set, because familiarity is how a test set quietly becomes a training set.

SOC 2: point-in-time versus over-a-period

SOC 2 reports are attestations by an independent CPA firm under AICPA standards. A Type 1 report says controls were designed properly at a point in time. A Type 2 report tests whether they operated effectively across an observation window, commonly three to twelve months. Security reviewers trust Type 2 more, because a control that worked on one day proves little about the rest of the year.

A one-time vendor bake-off is a Type 1 result. Voice agents change when an upstream model updates, which is why a pass at selection says little about month six. The Type 2 equivalent is continuous auditing: re-running the held-out suite on a schedule and after every known model change. The LLM update regression guide shows what silent upstream changes look like.

ISO/IEC 17025: impartiality, uncertainty, and proficiency

ISO/IEC 17025:2017 is the accreditation standard for testing and calibration laboratories. Four of its clauses translate into voice evaluation requirements. Clause 4.1 requires the lab to demonstrate how it maintains impartiality. Clause 7.2 requires methods to be validated before use. Clause 7.6 requires the lab to evaluate measurement uncertainty. Clause 7.7 requires ongoing monitoring of result validity, including proficiency testing or interlaboratory comparison where available.

For voice AI, that means four things. The evaluator discloses every commercial relationship with the vendors under test. The scoring method is validated against human labels before it scores anything that matters. Every reported number comes with a confidence interval. Known reference calls with known answers run in every batch, so a drifting scorer is caught.

Clinical trials: pre-registration changes the results

The strongest evidence that independence changes outcomes comes from medicine. Kaplan and Irvin (PLOS ONE, 2015) reviewed large trials funded by the US National Heart, Lung, and Blood Institute. Before 2000, 17 of 30 trials (57%) showed a significant benefit on the primary outcome. After ClinicalTrials.gov registration became required, only 2 of 25 (8%) did. The authors linked the drop to pre-declaring outcomes.

A Cochrane methodology review (Lundh et al., 2017) found that manufacturer-sponsored drug and device studies reported favorable efficacy results more often than other studies, with a risk ratio of 1.27 (95% CI 1.17 to 1.37). The sponsor's stake shows up in the data even when every individual researcher acts in good faith. Voice AI vendor benchmarks are sponsor-run studies with no registration requirement at all.

The six tests of an independent evaluation

The independence standard below turns everything above into six pass/fail checks. Score each 0 (absent), 1 (partial), or 2 (fully met, with evidence). An evaluation that scores 10 or more out of 12, with no zeros, can be called independent. Anything less is vendor testing with extra steps.

The 0.80 bar comes from content-analysis practice. Krippendorff's convention is to rely on data at alpha of 0.800 or higher, draw only tentative conclusions between 0.667 and 0.800, and discard data below 0.667. A team can apply the same bar to Cohen's kappa for two raters. If your human raters do not agree with each other at that level, no judge model can be validated against them either. The human-in-the-loop evaluation guide covers rater calibration.

The statistics that make a result defensible

A procurement decision needs two numbers per vendor, not one: the point estimate and its uncertainty. Three tools cover most voice AI comparisons.

Wilson intervals for each rate. A task success of 170/200 is 85%, but the 95% Wilson interval runs from roughly 79% to 89%. Report the interval every time.

Paired tests when vendors run the same scenarios. If both vendors handle the identical 400 scripted scenarios, the right test is McNemar's, which uses only the scenarios where they disagree. This matters for sample size. Detecting an 80% versus 85% difference at 80% power and 5% significance with independent samples needs about 903 calls per vendor:

n = (1.96 + 0.84)² × [0.80 × 0.20 + 0.85 × 0.15] / 0.05² ≈ 903

A paired design with 15% discordant scenarios needs about 469 scenario pairs for the same power. Running both vendors on the same scenarios roughly halves the cost of a fair bake-off. The POC bake-off guide shows how to script paired scenarios.

The overlapping-interval trap. In the example below, the two vendors' 95% intervals overlap, which many readers take to mean "no real difference." The paired McNemar test says otherwise (p ≈ 0.005), because pairing removes scenario-difficulty noise. Use the paired test whenever the design is paired.

# Simplified, runnable: score a pre-registered, blinded two-vendor bake-off.
# Requires: pip install statsmodels scikit-learn
import hashlib, json
from statsmodels.stats.proportion import proportion_confint
from statsmodels.stats.contingency_tables import mcnemar
from sklearn.metrics import cohen_kappa_score

# 1. Pre-registration: hash the frozen plan before any scored call runs
plan = {
    "primary_metric": "task_success_verified_in_system_of_record",
    "threshold": 0.85,
    "analysis": "McNemar exact, alpha=0.05",
    "heldout_manifest_sha256": "<sha256 of held-out call-id list>",
    "systems": {"A": "<vendor-x model+prompt hash>", "B": "<vendor-y model+prompt hash>"},
}
print("plan hash:", hashlib.sha256(json.dumps(plan, sort_keys=True).encode()).hexdigest()[:16])

# 2. Paired outcomes on the same 400 scenarios (illustrative counts)
both, a_only, b_only, neither = 300, 40, 18, 42
n = both + a_only + b_only + neither
for name, k in (("A", both + a_only), ("B", both + b_only)):
    lo, hi = proportion_confint(k, n, alpha=0.05, method="wilson")
    print(f"System {name}: {k/n:.1%}  95% CI [{lo:.1%}, {hi:.1%}]")
# System A: 85.0% [81.2%, 88.2%]   System B: 79.5% [75.3%, 83.2%]

# 3. McNemar uses only the discordant pairs (40 vs 18)
print("McNemar p =", round(mcnemar([[both, a_only], [b_only, neither]], exact=True).pvalue, 4))
# p = 0.0054, a real difference despite overlapping intervals

# 4. Inter-rater reliability on a double-scored subsample (human vs judge)
human = [1,1,0,1,0,1,1,0,1,1,0,1,1,1,0,1,0,1,1,1]
judge = [1,1,0,1,1,1,1,0,1,0,0,1,1,1,0,1,0,1,1,1]
print("Cohen's kappa =", round(cohen_kappa_score(human, judge), 2))
# 0.76: below the 0.80 bar, so this judge is not yet fit to score the primary metric

The final line is the kind of result most bake-offs never surface. A judge that agrees with humans on 18 of 20 calls sounds good. A kappa of 0.76 says it is not yet reliable enough to decide a contract.

Regulatory pull: why buyers will need evidence, not claims

Regulation is moving from "do not deceive" toward "show your work." This section summarizes the rules most relevant to voice agents as of October 2026. It is not legal advice; confirm current status with counsel.

FCC, TCPA and AI voices. In a Declaratory Ruling adopted February 2, 2024 (FCC 24-17), the FCC confirmed that the TCPA's restrictions on "artificial or prerecorded voice" cover current AI technologies that generate human voices. Outbound calls using them need the called party's prior express consent. A separate 2024 proposal (FCC 24-84) would add AI-specific consent and in-call disclosure rules. That proposal limits its definition to outbound calls, and as of mid-2026 it had not been finalized.

EU AI Act. Article 50 applies from August 2, 2026. It requires providers to design systems that interact with people so those people are informed they are dealing with an AI, unless that is obvious from context. The Digital Omnibus, approved by Parliament and Council in June 2026, moved the Annex III high-risk obligations to December 2, 2027 but left Article 50 on schedule (Jones Walker summary). Voice agents used in Annex III areas, such as creditworthiness assessment, certain life and health insurance pricing, or emergency call triage, face the high-risk regime. One detail matters for buyers. Under Article 43(2), most Annex III systems follow a conformity assessment based on internal control, with no notified body. The provider assesses itself. If you deploy such a system, independent evidence is the only check you get that the provider did not grade itself generously.

US states. California's AB 2905, effective January 1, 2025, requires calls placed with automatic dialing-announcing devices to inform the called person when the prerecorded message uses an artificial voice. Utah's SB 226 (2025) requires disclosure of generative AI when a consumer clearly and unambiguously asks. It also requires up-front disclosure for regulated occupations in high-risk interactions, such as those involving health or financial data. Colorado replaced its 2024 AI Act with SB 26-189, a notice-based law for automated decision-making in consequential decisions, effective January 1, 2027 (McDermott summary).

Each rule turns into an assertion you can test on every call, which is where evaluation meets compliance.

The voice agent compliance audit guide expands these into full test suites.

Vendor claim to verification: a lookup table

Use this table in the room when a vendor presents. Each row names the claim, the hidden choice behind it, and the question or test that settles it.

The third-party voice agent audit guide explains how to commission the tests in the right-hand column.

The audit trail: evidence that survives scrutiny

An evaluation is only as strong as the evidence behind each score. This matters most in transactional flows like voice ordering, payments, and scheduling, where a disputed outcome has a dollar value. For every scored call, keep a bundle that a skeptical reviewer could use to reproduce the score:

• Both audio channels, unmixed, with the sample rate and codec recorded.
• Timestamped transcripts from each STT pass, including interim results if latency is in scope.
• Exact configuration: LLM model ID, prompt hash, STT and TTS model versions, endpointing settings.
• Every tool call with arguments, responses, and the resulting system-of-record state.
• Each score with its rationale, the rater or judge ID, and the plan hash it was scored under.

For order-taking agents, the system-of-record check is what turns "the agent said it placed the order" into "the order exists with the right items and price." The what to log on every call guide and the food ordering testing guide cover the schema in detail.

How to run an independent voice AI evaluation

These eight steps take a team from vendor claims to a result it can defend. Steps 1 through 3 happen before any vendor sees a test call.

1. Write and hash the plan. Name the primary metric with an exact formula, the pass threshold, secondary metrics, sample size, statistical test, and decision rule. Compute a SHA-256 of the document and share the hash with all parties.

2. Build the dataset from your own traffic. Sample real calls across intents, accents, noise, and line quality. Split them into a development set the vendors may tune on and a held-out set they never see. Record the manifest hash.

3. Define conditions as a matrix. For example, 40 scenarios × 3 noise levels (clean, 15 dB SNR, 5 dB SNR) × 2 codecs (G.711 at 8 kHz and Opus at 16 kHz) gives 240 cells. Run each cell enough times to estimate variance.

4. Pin every version. Freeze model IDs, prompts, and settings for each vendor. Record them in the run log. Any change restarts the scored run.

5. Run paired and blinded. Send identical scenarios to every vendor, randomize order, and replace vendor names with codes before any human or judge scores a call.

6. Measure reliability before scoring. Double-score a subsample of 100 or more calls. If agreement on the primary metric is below 0.80, fix the rubric and re-calibrate before scoring the rest.

7. Report intervals and the paired test. Publish each vendor's rate with a Wilson interval and the McNemar result. Report the number of runs and any calls excluded, with reasons.

8. Re-run on a schedule. Repeat the held-out suite monthly and after any vendor model change, and compare against the frozen baseline.

Evalgent runs this process as a neutral party for pre-launch audits, vendor bake-offs, regression checks after model updates, and ongoing production scoring. The method works the same whether we run it or your team does. What matters is that the six tests pass.

Buyer's audit checklist

Before you accept any evaluation result, from a vendor, a consultant, or your own team, confirm each item. A "no" on any of the first six means the result is not independent.

• The test plan was dated and hashed before the first scored call.
• The held-out calls came from your traffic and were never shared for tuning.
• Raters and judges saw coded system labels, not vendor names.
• Inter-rater agreement on the primary metric is reported and at least 0.80.
• A second party could re-run the harness from the documents provided.
• Every scored call maps to frozen model, prompt, and config versions.
• The evaluator disclosed all commercial relationships with the vendors under test.
• Evaluator fees do not depend on which vendor wins.
• Latency is reported mouth-to-ear at p50, p95, and p99, not server-side medians.
• Resolution is verified against the system of record, not inferred from "no transfer."
• Entity accuracy is reported separately from WER.
• Every rate has a confidence interval, and paired designs use a paired test.
• The number of configurations tried per vendor is disclosed.
• Excluded calls are listed with reasons.
• Disclosure and consent assertions were tested on every call.
• A re-test schedule and trigger list for model changes exist.

RFP clause template

Paste this into the evaluation section of your RFP and adapt it with counsel. It turns the six tests into contractual terms a vendor must accept before the POC starts. The RFP checklist covers the rest of the document.

SECTION X. INDEPENDENT EVALUATION

X.1 Evaluator. Buyer will appoint an evaluator with no financial interest
    in any respondent. Respondents will not pay, select, or instruct the
    evaluator.

X.2 Pre-registration. Before testing, Buyer will issue a test plan naming
    the primary metric, formula, pass threshold, sample size, and analysis
    method, with its SHA-256 hash. The plan will not change after testing
    begins, except by a dated, hashed amendment issued to all respondents.

X.3 Data. Buyer will provide a development set for configuration. Scoring
    will use a separate held-out set from Buyer traffic that respondents
    will not receive. Respondents will not train, fine-tune, or add
    vocabulary from held-out data.

X.4 Versioning. Each respondent will declare the model identifiers,
    prompt versions, STT/TTS versions, and turn-taking settings used in
    the scored run. Any change during the scored run voids that run.

X.5 Blinding. Scoring will use coded system labels. Respondents will not
    contact raters during scoring.

X.6 Metrics. Latency will be measured mouth-to-ear from the caller side
    and reported at p50, p95, and p99. Resolution will be verified against
    Buyer's system of record. Rates will be reported with 95% confidence
    intervals.

X.7 Reliability. The evaluator will report inter-rater agreement on the
    primary metric. Scores with agreement below 0.80 will not decide award.

X.8 Re-evaluation. The selected vendor will notify Buyer at least
    [10] business days before any change to underlying models, and will
    support re-running the held-out suite after each such change and
    [monthly] thereafter.

X.9 Evidence. For every scored call, the evaluator will retain both audio
    channels, transcripts, configuration identifiers, tool-call logs, and
    score rationale for [24] months.

Where independent evaluation fits, and where it does not

Independent evaluation is not a substitute for a vendor's own engineering tests. Vendors should run thousands of unit and regression tests on their own stack, and those tests are useful to them. The independent layer answers a narrower question that only a party with no stake can answer: on your calls, under conditions fixed in advance, which option performs better, and by how much, with what uncertainty?

It also has costs. A held-out set from real traffic requires consent and redaction work. Paired, blinded runs take longer than a demo. Reliability checks reveal that many rubrics are too vague to score, which means rework. Those costs are the price of a number you can defend. Evalgent exists to carry most of that work for buyers, but the vendor evaluation scorecard and the steps above are enough to start on your own.

Frequently asked questions

What is independent voice AI evaluation?

It is testing a voice agent with a party that has no technical, managerial, or financial stake in the result, on the buyer's own calls, under a plan fixed before results exist. It replaces vendor-defined metrics with buyer-defined ones and makes results comparable across vendors because everyone is scored by the same rules.

Why do vendor-reported voice AI metrics differ from production?

Five mechanisms drive the gap: metrics defined where the vendor looks best, best-of-k selection on small test sets, Goodhart effects when a metric becomes a target, test-set contamination during POC tuning, and switching the deciding metric after results arrive. None requires dishonesty, which is why process controls work better than trust.

What makes an evaluation count as independent?

Six tests: a pre-registered and hashed test plan, buyer-owned held-out data, blinded scoring, reported inter-rater reliability of at least 0.80, a reproducible harness, and pinned model and configuration versions. Score each 0 to 2. Ten or more out of twelve, with no zeros, qualifies as independent.

How many calls do I need to compare two voice agents?

To detect 80% versus 85% task success at 80% power with independent samples, you need about 903 calls per vendor. If both vendors run the same scenarios, a paired McNemar design needs about 469 scenario pairs when 15% of pairs disagree. Pairing roughly halves the cost of a fair comparison.

Can my own team run an independent evaluation?

Partly. An internal team separate from the integration team can meet technical and managerial independence. Financial independence is harder if the same budget owner championed a vendor. The six tests matter more than who runs them. If your team passes all six with evidence, the result is defensible.

Does the EU AI Act require independent testing of voice agents?

Not directly. Article 50, in force since August 2, 2026, requires AI-interaction disclosure. Voice agents in Annex III areas face high-risk rules from December 2, 2027, but most use internal-control conformity assessment with no notified body. Independent evaluation gives deployers evidence the self-assessment does not provide.

Do AI voice calls need consent under the TCPA?

For outbound calls, yes. The FCC's February 2024 Declaratory Ruling (FCC 24-17) confirmed that AI-generated voices are "artificial" voices under the TCPA, so those calls need the called party's prior express consent. A 2024 proposal adding AI-specific disclosure rules had not been finalized as of mid-2026. Confirm current status with counsel.

What should a voice ordering audit trail include?

For each call, keep both audio channels, timestamped transcripts, exact model and prompt versions, every tool call with arguments and responses, and the resulting order record in the system of record. Add each score's rationale and the plan hash it was scored under, so a reviewer can reproduce any disputed result.

The bottom line

An evaluation is independent when the plan, the data, and the scoring sit outside the vendor's control, and you can prove it with a hash, a manifest, and a reliability number. Hold every voice AI claim, including your own team's, to the six tests, and you will buy on evidence instead of on the best of a vendor's many tries.