Voice Agent Accuracy Audit: What It Measures

# Voice agent accuracy audit: what it measures

> Quick answer: A voice agent accuracy audit is an independent measurement of how correctly an agent hears, understands, and acts on real calls. It scores transcription accuracy, intent and entity accuracy, task success, and tool-call correctness on your own audio, using human-labeled ground truth rather than vendor-reported numbers.

Every voice agent vendor claims high accuracy. Almost none of them define the word. Accuracy of what, on whose calls, measured how, and graded by whom. Those four questions decide whether a number means anything. A voice agent accuracy audit answers them in writing.

An audit is not a demo and not a dashboard. It is a measurement done by a party with no stake in the result. The agent is scored on your traffic, against labeled ground truth, across several distinct accuracy dimensions. This guide explains exactly what those dimensions are, which metric applies to each, and how the audit is run. For the broader discipline, see our overview of voice agent evaluation.

What "accuracy" actually means for a voice agent

Accuracy is not one number. A voice agent is a pipeline, and each stage can be right or wrong on its own terms.

The agent first hears the caller and turns speech into text. Then it interprets that text into an intent and extracts entities like dates, names, and amounts. Then it acts, often by calling a tool or an API. Then it produces a response. An error at any stage can wreck the call, and each stage needs its own metric.

> Voice agent accuracy audit: an independent, evidence-based measurement of a voice agent's correctness across its recognition, understanding, and action stages. It is scored on the buyer's own calls with human-labeled ground truth, not on the vendor's benchmark.

A single "95% accurate" headline hides this structure. It usually refers to one stage, on clean audio, in one language, under ideal conditions. It tells you nothing about how the agent handles your callers on a bad phone line. The audit's job is to unbundle the claim.

Why vendor-reported accuracy is not evidence

A vendor's own accuracy number is marketing, not evidence. This is not an accusation of dishonesty. It is a structural point.

The vendor chooses the test set. The vendor chooses the metric. The vendor chooses the conditions and grades its own work. Any party that controls all four inputs can produce almost any number. The result is not falsifiable, so it is not evidence. This is the same reason a lab does not certify its own results.

There is also a selection problem. Vendor benchmarks tend to use clean, read speech in common accents. Your calls have crosstalk, hold music, hesitations, and rare names. A number earned on the first says little about the second. Independence closes that gap. We cover the principle in depth in independent voice AI evaluation, and the formal version in third-party voice agent audit.

The accuracy dimensions an audit measures

A serious audit reports several accuracy dimensions separately. Each maps to a specific metric with a defined way of measuring it. The table below is the core of the audit scope.

Every row is scored on your calls. Every row uses human-labeled ground truth. The rest of this guide walks each dimension.

Transcription accuracy: WER and CER

The first thing an agent does is hear. If it mishears "Baker" as "maker," everything downstream inherits the error. Transcription accuracy is the foundation, and it is measured against a human transcript of the same audio.

The standard metric is word error rate. WER counts substitutions, insertions, and deletions, then divides by the number of words in the reference. Lower is better. A WER of 0.05 means one word in twenty is wrong. Character error rate does the same at the character level, which matters for spelled-out names and IDs. We compare the two in our guide on WER vs CER.

Raw WER can mislead, though. Missing the word "the" is not the same as missing a dollar amount. A good audit reports entity-weighted error too, so mistakes on names, dates, and numbers count for more. Pronunciation of unusual words is a common failure point, covered in our pronunciation for voice agents guide.

Intent and entity accuracy

Hearing the words is not understanding them. The agent must map the transcript to what the caller wants, then pull out the specific values it needs to act.

Intent recognition classifies the call into a goal. Book an appointment. Check a balance. File a claim. This is graded with precision and recall, reported per intent. Precision asks how many of the agent's guesses were right. Recall asks how many true cases it caught. The F-score combines them into one figure.

Entity extraction pulls the structured values out of the speech. This is a form of named-entity recognition, and it is graded the same way, per entity type. A date entity and a phone-number entity fail in different ways and deserve separate scores. Confusions between intents are best shown in a confusion matrix, which reveals which categories the agent mixes up.

Per-class reporting is the point here. An agent can be excellent at common intents and hopeless at the rare, high-stakes one. A blended average hides that. The audit surfaces it.

Task success and tool-call correctness

An agent can transcribe perfectly, classify the intent, extract every entity, and still fail the caller. Task success measures the outcome the caller actually cared about.

Task completion is judged by a human against a defined goal. Did the appointment get booked. Was the balance given correctly. Did the caller leave with their problem solved. This is measured against outcomes, not the agent's own claim of success, because agents often report success on calls that failed.

Tool-call correctness checks the actions the agent takes. A voice agent usually calls tools or APIs to do real work. The audit asks three things. Did it call the right tool. Did it pass the right arguments. Did it call at the right time. A booking made for the wrong date is a correct tool call with a wrong argument, and it counts as a failure.

Recovery and escalation matter because no agent is perfect. When the agent misheard or hit an edge case, did it recover gracefully or hand off cleanly to a human? Or did it loop, stall, or confidently do the wrong thing? An audit scores the failure behavior, not just the happy path.

Why you measure on your calls, not the vendor's benchmark

This is the single most important choice in an audit. The test set decides the result.

A vendor benchmark is a curated dataset chosen to look good. Your production traffic is messier and more honest. It has your callers, your accents, your background noise, your rare account numbers, and your specific tasks. An accuracy number is only meaningful on data that resembles what the agent will actually face.

Measuring on your own calls also lets you set thresholds that reflect your risk. A missed word in a food order is minor. A missed digit in a medication dose is not. The same WER means different things in different contexts, so the audit weights errors by their consequence for your use case. Our guide on how to benchmark voice agents on your own data goes deeper on building that dataset.

Ground-truth labeling and sampling

An audit is only as good as its ground truth. Ground truth is the human-verified answer for each call: the correct transcript, the true intent, the real entities, and whether the task succeeded.

Labeling has to be careful. Two labelers can disagree, so a good audit measures agreement between them, often with inter-rater reliability statistics. Disagreements get adjudicated by a third labeler. Guidelines are written down so labeling stays consistent across the whole sample.

Sampling matters just as much. You cannot label every call, so the audit draws a representative sample). The sample has to cover your real mix of accents, call types, and difficulty. Stratified sampling ensures rare but important intents get enough coverage to score reliably. A sample skewed toward easy calls produces an accuracy number that flatters the agent.

Audio versus transcript: what gets evaluated

There is a real choice about what the audit looks at. Evaluating the transcript alone is cheaper and faster. Evaluating the audio catches things the transcript can never show.

Audio carries information the text loses. Talk-over, long silences, robotic delivery, mispronounced names, and the emotional read of the caller all live in the sound. An agent can produce a clean transcript while sounding broken to the caller. A transcript-only audit would miss that entirely. We break down the trade-off in transcript vs audio voice agent evaluation.

A thorough accuracy audit uses both. The transcript drives the WER, intent, and entity metrics. The audio catches the acoustic and conversational failures. The two together give a complete picture that neither gives alone.

How to run a voice agent accuracy audit

Here is the process a rigorous accuracy audit follows, from scope to signed report.

1. Define the accuracy dimensions and thresholds. Agree in writing which dimensions you will measure and what score counts as a pass. Set thresholds per dimension, weighted for the errors that hurt your use case most. Fix this before any call is scored.

2. Assemble a representative sample. Pull real calls from production. Stratify the sample so rare, high-stakes intents and hard audio are covered, not just the easy majority. Document where the audio came from.

3. Create human-labeled ground truth. Have trained labelers produce the correct transcript, intent, entities, and task outcome for each call. Use written guidelines. Measure labeler agreement and adjudicate disagreements.

4. Score each dimension separately. Compute WER and CER for transcription. Compute precision, recall, and F1 per intent and entity type. Judge task success and tool-call correctness against defined outcomes. Report each dimension on its own.

5. Weight and consequence-check. Re-weight the raw scores by business impact so a wrong dose or amount counts more than a dropped filler word. Flag the failures that carry real risk.

6. Report with evidence. Deliver a scored report that ties every number to the calls behind it. Include per-class breakdowns, example failures, and the sampling method, so a reader who was not present can rely on it.

7. Repeat on a schedule. Re-run the audit after model changes and at set intervals. Accuracy drifts as models, prompts, and caller behavior change, so a single result has a shelf life.

For the difference between this and everyday QA, see our guide on testing vs evaluation for voice agents. For a step-by-step approach to vendor selection, see how to evaluate voice agent vendors.

Mapping accuracy to risk and compliance

Accuracy is not just a quality metric. For regulated buyers, it is a risk control. A voice agent that misreads a medication name or a claim number creates real harm and real liability.

This is why security and compliance teams increasingly ask for independent accuracy evidence. Frameworks like the NIST AI Risk Management Framework treat measurement and independent validation as core to managing AI risk. An audit produces exactly that evidence: a documented, third-party measurement that survives a review. A vendor's self-reported figure does not.

The reporting has to match the stakes. High-consequence intents get tighter thresholds and larger samples. Low-consequence intents can tolerate more error. Tying each accuracy score to its consequence is what turns a technical number into a decision a buyer can defend.

The bottom line

A voice agent accuracy audit measures correctness at every stage of the pipeline, on your own calls, against human-labeled ground truth. Vendor-reported accuracy is a marketing claim, not evidence, because the vendor controls the test.

Ready to see how your agent scores on your own calls? Book a demo and we will show you what an independent accuracy audit looks like on your traffic.

Frequently asked questions

What does a voice agent accuracy audit measure?

A voice agent accuracy audit measures correctness at each pipeline stage. It scores transcription accuracy with word error rate and character error rate, intent and entity accuracy with precision and recall, task success against defined goals, and tool-call correctness. Each dimension is scored separately on your own calls using human-labeled ground truth, not blended into one headline number.

Why is vendor-reported accuracy not enough?

Vendor-reported accuracy is not evidence because the vendor controls the test set, the metric, the conditions, and the grading. Any party controlling all four can produce almost any number. Vendor benchmarks also use clean, read speech that little resembles your real calls. Independent measurement on your traffic is what makes an accuracy claim falsifiable and trustworthy.

What is word error rate in a voice agent audit?

Word error rate, or WER, is the standard transcription accuracy metric. It counts substitutions, insertions, and deletions against a human-verified transcript, then divides by the reference word count. Lower is better. A WER of 0.05 means one word in twenty is wrong. Good audits also report entity-weighted error, so mistakes on names and amounts count more.

How is intent accuracy measured?

Intent accuracy is measured with precision, recall, and F1, reported per intent. A human labels the caller's true intent, and the audit compares it to what the agent classified. Precision measures how many guesses were correct. Recall measures how many true cases were caught. Per-intent reporting reveals which specific intents the agent handles poorly.

Why measure accuracy on my own calls instead of a benchmark?

You measure on your own calls because the test set decides the result. Your production traffic has your callers, accents, noise, rare names, and specific tasks. A vendor benchmark uses curated, clean audio that flatters the agent. An accuracy number only predicts real performance when it is earned on data that resembles what the agent will actually face.

What is ground-truth labeling in an accuracy audit?

Ground-truth labeling is the human-verified answer for each call: the correct transcript, true intent, real entities, and whether the task succeeded. Trained labelers follow written guidelines to produce it. Because labelers can disagree, audits measure inter-rater agreement and adjudicate conflicts. Every accuracy score is computed by comparing the agent's output to this labeled ground truth.

Should an accuracy audit use audio or transcripts?

A thorough accuracy audit uses both. Transcripts drive the word error rate, intent, and entity metrics. Audio catches failures the text loses, such as talk-over, long silences, robotic delivery, and mispronounced names. An agent can produce a clean transcript while sounding broken to the caller, so transcript-only audits miss real problems.

How often should a voice agent accuracy audit be repeated?

A voice agent accuracy audit should be repeated after any model, prompt, or configuration change, and at set intervals otherwise. Accuracy drifts as models update and caller behavior shifts, so a single result has a limited shelf life. Regular re-auditing catches regressions before they reach production and keeps compliance evidence current for reviews.