How to Score a Voice Agent Conversation

# How to score a voice agent conversation

Quick answer

To score a voice agent conversation, grade the whole call against a weighted rubric. Rate task success, accuracy, policy and safety, resolution, tone, and efficiency. Combine the scores into one number with clear pass bars. Score the audio, not just the transcript. Pick a human, LLM, or hybrid scorer for each dimension.

A single voice call carries a lot of signal. The agent either solved the caller's problem or it did not. It said true things or it made something up. It followed policy or it broke a rule. It sounded warm or it sounded like a machine reading a script. A useful score has to capture all of that in one pass, on one call, without collapsing into a thumbs-up.

This article covers how to grade one conversation end to end. You will build a rubric, weight the dimensions that matter, set pass bars, and choose a scorer. It is about a single call scored as a whole, not about tracking context across many turns. That cross-turn work is a separate topic. Here the unit is the finished conversation, and the goal is one honest number you can defend.

What it means to score one conversation end to end

Scoring a conversation is not the same as measuring a metric. A metric like latency describes one property. A conversation score is a judgment about the whole call, made against a fixed standard. That standard is a rubric): a set of named dimensions, each with a scale and a rule for what earns each point.

> Conversation score: a single rated judgment of one complete call, produced by applying a weighted rubric across several quality dimensions. It answers whether this specific call was good, and why.

The end-to-end part matters. You grade the call from greeting to hang-up as one artifact. A call can nail the first two turns and fail the resolution. It can solve the problem while leaking a policy violation halfway through. If you only score fragments, you miss how the call actually landed for the caller.

This is different from testing. Testing asks whether a specific input produces an expected output. Scoring asks how good the whole interaction was on a graded scale. Our guide on testing versus evaluation for voice agents draws that line in full. Scoring lives on the evaluation side, and it is where a rubric earns its keep.

Why you score on audio, not just the transcript

The transcript is the easy artifact to grade. It is text, it is small, and it is cheap to feed to a rule or a model. But the transcript is a lossy record of the call. It throws away everything that lived in sound.

A transcript cannot tell you the agent mispronounced the caller's name. It cannot show the two-second pause before a reply, or the flat, robotic delivery of a correct sentence. It hides overlapping speech and failed barge-in, where the caller interrupts and the agent talks over them anyway. Those are exactly the failures that make a technically correct call feel broken.

So the rubric should be scored on the recording wherever a dimension depends on delivery. Tone, pacing, pronunciation, and interruption handling all need the audio. Content and policy can often be scored from the transcript. Our deeper comparison of transcript versus audio evaluation explains which failures each artifact catches. The short version: grade text for what was said, grade audio for how it was said.

There is a cost angle here too. Audio is slower and harder to process at scale. Most teams run a hybrid: score every call from the transcript, then re-score a sample on audio. The sample is where you learn what the text-only pass has been missing.

The dimensions that decide a conversation score

A good rubric has few dimensions, each with a clear question and a weight. Too many dimensions and scorers disagree. Too few and the number hides real failures. Six dimensions cover most voice agent calls. The table below pairs each with what to check and a starting weight.

Two notes on this table. Policy and safety is weighted, but it also works as a hard gate. One serious violation should fail the whole call regardless of the other scores. Task success carries the most weight because a pleasant call that never solved the problem is still a failed call.

Each dimension needs a scale. A five-point Likert scale works well for graded dimensions like tone. Binary pass or fail works better for policy checks. Write a short descriptor for each point so two scorers reading the same call land in the same place. "3 = solved the stated problem but left a smaller one open" beats a bare number.

Weighting dimensions and setting pass bars

Once you have dimensions and scales, you combine them. The standard method is a weighted arithmetic mean. Each dimension score is multiplied by its weight, and the products are summed. A call scoring 4 out of 5 on task success at 30% contributes 24 points on a 100-point scale.

Weights encode what your business actually cares about. A healthcare intake line weights accuracy and policy above tone. A retention line weights empathy higher. There is no universal weighting. Set yours against the cost of each failure mode, not against a template.

Pass bars turn the score into a decision. You need at least two. First, a hard gate on policy and safety: any critical violation fails the call outright, no matter the total. Second, an overall threshold, say 80 out of 100, below which the call counts as a miss. Some teams add a per-dimension floor so a call cannot pass on a strong average while failing one thing that mattered.

Be explicit about what a passing call means. A pass is not "the agent tried." A pass is "this call met the bar we would defend to a customer." Set the bar where a reasonable caller would agree the call was good. Then hold it steady, so scores stay comparable across weeks and across agent versions.

Choosing your scorer: human, LLM, or hybrid

The rubric is only as good as who applies it. You have three options, and each has real trade-offs.

Human scorers are the gold standard for tone, empathy, and edge cases. A trained reviewer hears sarcasm, discomfort, and the moment a call went sideways. The cost is speed and scale. Humans are slow and expensive, and they drift. Two reviewers can score the same call differently, which is why you measure agreement between them.

An LLM judge scores fast and cheap, and it never gets tired. It can read a transcript and rate helpfulness or policy adherence in seconds. But it inherits the blind spots of whatever it reads. A text-only judge misses everything in the audio. It also brings its own biases and can be confidently wrong. The limits are real enough that we wrote a whole piece on where LLM-as-judge breaks down for voice.

A hybrid is what most serious programs run. The model scores every call as a wide, cheap first pass. Humans score a targeted sample: high-stakes scenarios, calls the model flagged, and a random slice for calibration. The human sample keeps the model honest, and the model keeps the humans from having to hear every call.

Whichever you choose, measure how consistent your scoring is. When two scorers disagree often, the rubric is vague, not the call. Inter-rater reliability, often reported as Cohen's kappa, tells you whether your rubric produces repeatable scores. Low agreement means you rewrite the descriptors before you trust any number.

How to build and apply a conversation scorecard

Here is the practical sequence for turning the ideas above into a working scorecard you can run on real calls.

1. Define the dimensions. Start from the six in the table. Cut any that do not apply to your use case, and add none you cannot describe in one sentence.

2. Write the scale descriptors. For each dimension, spell out what earns each point. Make the difference between a 3 and a 4 concrete enough that two people agree.

3. Set the weights. Assign a weight to each dimension so they sum to 100. Weight by the cost of failure, and mark policy and safety as a hard gate.

4. Set the pass bars. Fix an overall threshold and any per-dimension floors. Decide what a single critical violation does to the score.

5. Pick the scorer per dimension. Route delivery dimensions to audio scoring. Route content and policy to whichever scorer you can trust, human or model.

6. Calibrate on a shared set. Have two scorers grade the same 20 calls. Measure agreement, then rewrite any descriptor that produced disagreement.

7. Score, then audit. Run the scorecard across your calls. Re-score an audio sample of the transcript-passed calls to catch what text missed.

8. Report the score and the reasons. Store the number, the per-dimension breakdown, and a one-line rationale per call. A score with no reasons cannot be improved.

Run this on a batch and you get more than one number. You get a distribution, a set of failing calls with reasons, and a rubric you can defend. Feeding the scorecard your own recordings is the point of benchmarking voice agents on your own data rather than on someone else's demo.

Where an independent scorer fits

Scoring your own agent has a built-in conflict. The team that built the agent tends to write a rubric it passes. Bars drift down over time, and failing calls get explained away. That is the case for an independent evaluator applying a fixed rubric you did not tune to flatter yourself.

Evalgent scores voice agent conversations as a third party. We grade the audio, apply a consistent rubric across dimensions, and report the per-call reasons behind each number. Because the scoring is external, the pass bar does not move to match the agent. Independent scoring is the difference between a number you like and a number you can show a customer. Our overview of independent voice AI evaluation explains why the third-party angle changes what a score is worth.

The output feeds the same decisions your team already makes. A conversation score maps cleanly onto customer satisfaction targets, and it connects to the metrics in our voice agent metrics scorecard. One is the per-call judgment, the other is the program-level view. Together they tell you whether the agent is ready and where it is weak.

Frequently asked questions

How do I score a voice agent conversation?

Score a voice agent conversation by grading the whole call against a weighted rubric. Rate task success, accuracy, policy and safety, resolution, tone, and efficiency on defined scales. Multiply each score by its weight, sum them into one number, and check it against your pass bars. Grade delivery dimensions on the audio, not the transcript.

What dimensions belong in a conversation scorecard?

A solid conversation scorecard uses six dimensions: task success, accuracy, policy and safety, resolution, tone and empathy, and efficiency. Task success carries the most weight, since a pleasant call that solves nothing still fails. Policy and safety doubles as a hard gate. Cut dimensions that do not apply, and never add one you cannot describe in a sentence.

Should I score the transcript or the audio?

Score both, for different dimensions. Content and policy can be graded from the transcript, which is cheap and fast. Tone, pacing, pronunciation, and interruption handling need the audio, because a transcript hides them. Most teams score every call from text, then re-score an audio sample to catch delivery failures the transcript never recorded.

How do I weight the rubric dimensions?

Weight rubric dimensions by the cost of each failure mode, not by a template. Assign a percentage to each so they sum to 100, then combine scores with a weighted arithmetic mean. A healthcare line weights accuracy and policy highest. A retention line weights empathy higher. Review the weights whenever your priorities or risks change.

What pass bar should a scored conversation meet?

Set at least two bars. First, a hard gate: any critical policy or safety violation fails the call outright, whatever the total. Second, an overall threshold, often around 80 out of 100. Add per-dimension floors so a call cannot pass on a strong average while failing something that mattered. Hold the bars steady across versions.

Can an LLM score a voice agent conversation reliably?

An LLM judge can score fast and cheaply, but it inherits the blind spots of whatever it reads. A text-only judge misses everything in the audio and can be confidently wrong. Use it for a wide first pass on content and policy, then have humans score a sample. Measure agreement to know when the model can be trusted.

How do I know my scoring is consistent?

Measure inter-rater reliability. Have two scorers grade the same shared set of calls, then compute agreement, often reported as Cohen's kappa. High agreement means the rubric produces repeatable scores. Low agreement means the descriptors are vague, not the calls. Rewrite the ambiguous descriptors and re-test before you trust any number the scorecard produces.

How is this different from multi-turn evaluation?

Scoring a conversation grades one finished call as a whole, against a rubric, to produce a single judgment. Multi-turn evaluation is a separate concern that tracks context and state across many turns within a call. This article covers the end-to-end score of the complete conversation, not how well the agent maintained memory turn by turn.

The bottom line

Scoring a voice agent conversation means grading the whole call against a weighted rubric with clear pass bars. Score the audio, choose the right scorer for each dimension, and measure whether your scoring is consistent enough to trust.

Want an independent score on your own conversations, graded on audio against a rubric you can inspect? Book a demo and we will score a sample of your real calls.