How to Measure Sentiment Score for Voice Agents

# How to measure sentiment score for voice agents

Quick answer

A sentiment score for voice agents rates how a caller feels during a call. Measure it from two sources: the words in the transcript and the tone in the audio. Track it per turn and across the whole call. Rising negativity is an early warning to escalate.

Most teams measure sentiment the easy way. They run the transcript through a text model and read back a number. It looks like sentiment. It is only half of it. A caller can say "that's fine" while sounding furious, and the words score neutral.

Voice is not text. A voice agent talks to a person whose mood lives in pitch, pace, and volume, not just in vocabulary. If you score only the words, you miss the angry sigh, the clipped reply, the rising strain. This piece shows how to measure a real sentiment score for voice agents, from words, from tone, and across time.

It builds on our broader work on how to score a voice agent conversation and where a sentiment metric fits on a full voice agent metrics scorecard.

What a sentiment score for a voice agent actually measures

A sentiment score estimates the caller's emotional state during the call. It is usually a number on a scale, from strongly negative to strongly positive. The goal is simple. You want to know how the caller felt, turn by turn, without listening to every call yourself.

> Sentiment score: a rating of the caller's emotional state during a call, derived from what they said and how they said it. It can be per turn, per call, or tracked as a curve over time.

Sentiment draws on two fields. One is sentiment analysis, the branch of natural language processing that classifies text as positive, negative, or neutral. The other is the study of affect), or how emotion shows up in behavior and, for voice, in sound.

A voice agent gives you both inputs at once. It produces a transcript, and it produces audio. The score you trust reads both. The score most teams settle for reads only the first.

Text sentiment vs acoustic sentiment: the polite-words problem

Here is the core issue. Text sentiment and acoustic sentiment can disagree on the same turn, and the disagreement is the point.

Text sentiment reads the words. "I already told you that twice" scores mildly negative on wording alone. Acoustic sentiment reads the sound. The same line, said with a hard edge and a rising pitch, is a caller about to hang up. The words undersell the risk.

The reverse happens too. A caller says "no, this is a nightmare, I can't believe it" while laughing. The words look alarming. The tone is playful. Words-only sentiment flags a crisis that is not there.

This gap is why transcript-only sentiment is unreliable for voice. It flattens delivery into neutral text. The tone that carried the real feeling never made it into the words. We cover the wider version of this problem in our comparison of transcript versus audio evaluation, and it applies directly to sentiment.

Why tone escapes the transcript. Emotion in speech lives in prosody) — the rhythm, stress, and intonation of how words are said. It also lives in paralanguage: sighs, pauses, and emphasis. A transcript strips all of it. Two very different calls can produce the exact same text.

The three sentiment signals: words, tone, and trajectory

A complete sentiment score reads three signals, not one. Each catches something the others miss. The table below is the mental model to keep.

Words are cheap and scalable. You can run thousands of transcripts fast. But words alone are the weakest of the three for voice.

Tone is the signal that separates voice from chat. It comes from acoustic features: fundamental frequency, which drives perceived pitch, plus loudness and speaking rate. The academic field here is emotion recognition, which infers affect from the audio signal. This is what catches the angry tone under polite words.

Trajectory is the signal most teams skip. A single average per call hides the story. A call can start bad, recover, and end well. Another can start fine and collapse in the last minute. Only the curve tells you which.

Per-turn vs whole-call sentiment

You can score sentiment at two grains. Both matter, and they answer different questions.

Per-turn sentiment scores each caller turn on its own. It shows exactly when the mood moved, and what the agent said right before it moved. This is the grain you need to debug a specific failure. If sentiment dropped at turn seven, you go read turn six.

Whole-call sentiment rolls the turns into one number for the call. It is what you report on a dashboard and trend across thousands of calls. It answers "are calls getting better or worse this week" without opening any single call.

Use both. Per-turn sentiment is for diagnosis. Whole-call sentiment is for monitoring. A whole-call average alone is a trap. It smooths over the exact moments that made the caller angry, which are the moments worth fixing.

Tracking sentiment trajectory: did the call recover

Trajectory is where a sentiment score earns its keep. A call is a sequence, not a snapshot. The shape of that sequence tells you whether the agent handled a hard moment or made it worse.

Picture three calls that all end at a neutral score. The first stayed neutral throughout, an easy call. The second started angry and climbed to neutral, a recovery, the agent did well. The third started warm and fell to neutral, a slow decline, something went wrong. Three identical end scores, three completely different calls.

To read trajectory, plot per-turn sentiment from the first turn to the last. Then look at the slope and the low point. A recovering call has a rising slope after its low point. A spiraling call keeps falling. The depth of the low point tells you how close you came to losing the caller.

This is also how you separate an agent problem from a caller problem. Some callers start angry no matter what. What you grade is whether the agent moved the curve up. An agent that consistently recovers angry openings is doing its job, even if the average sentiment on those calls looks low.

Sentiment as an early-warning and escalation signal

The most useful thing a sentiment score does is fire early. A falling curve is a warning before the caller ever says "let me speak to a human."

Set a threshold. When per-turn sentiment drops below it, or falls sharply over two or three turns, treat it as a trigger. The agent can soften its approach, slow down, or hand off to a person. A sharp negative slope is often a better escalation signal than any single keyword, because it catches frustration before it becomes an explicit demand.

This connects sentiment to real outcomes. A missed escalation is one of the most expensive failures a voice agent has. Our guide on escalation for voice agents covers the handoff design in depth. Sentiment is the sensor that should drive it.

Two cautions. First, do not escalate on a single noisy turn. One clipped word is not a crisis. Use a short trend, not one point. Second, calibrate the threshold to the use case. A collections call runs hotter than a billing question. The same raw score means different things in each.

How to measure sentiment for a voice agent

Here is a concrete, repeatable process. It works whether you run it in-house or through an independent evaluator.

1. Collect both artifacts. For every call, keep the transcript and the call audio, aligned by turn. If you only keep the transcript, you can never score tone later.

2. Score the words. Run text sentiment analysis on each caller turn. This gives you the explicit, stated layer of feeling.

3. Score the tone. Extract acoustic features from the audio for each turn: pitch, loudness, and pace. Map them to an emotional estimate. This is the layer the transcript cannot give you.

4. Combine per turn. Merge the word score and the tone score into one per-turn sentiment value. When they disagree, weight tone more for anger and stress, since that is where words lie most.

5. Plot the trajectory. Chart per-turn sentiment from the first turn to the last. Record the low point, the end point, and the slope out of the low point.

6. Set escalation thresholds. Define the score and the slope that trigger a warning or a handoff. Tune them per use case, not once for everything.

7. Validate against human ratings. Have people rate a sample of calls. Compare their ratings to the automated score. Measure agreement and fix the gaps before you trust the number.

That last step is not optional. A sentiment score is a claim about how people felt, so people have to check it. Measure agreement with a statistic like inter-rater reliability, and judge the model with precision and recall on the calls you most needed to catch.

Why an independent evaluator should score sentiment

There is a conflict of interest baked into self-reported sentiment. The team that built the agent, or the vendor that sold it, has a reason to report the flattering number. Words-only sentiment is easy to game, and it tends to look good. Tone is harder to fake and less flattering.

Evalgent is an independent, third-party evaluator built for exactly this. It scores sentiment from the audio, not just the transcript, so the angry tone under polite words shows up in the number. Because the scoring is external, the result is defensible to a buyer, a regulator, or an executive who wants the real picture. We make the case for this arm's-length approach in our piece on independent voice AI evaluation.

Independent scoring also runs on your data, not a canned demo. Sentiment is context-heavy, so the calls have to be your calls. Our approach to benchmarking voice agents on your own data applies here, and the wider method sits inside our voice agent evaluation framework.

An illustrative example makes the value concrete. Suppose text-only sentiment rates an agent's billing calls at 82 out of 100. Add acoustic scoring, and the same calls come back at 61, because callers sound strained while reading out card numbers. That 21-point gap is the difference between a demo you trust and one that hides the problem. The figure here is illustrative, not measured, but the direction is the whole reason to score tone.

Frequently asked questions

What is a sentiment score for a voice agent?

A sentiment score for a voice agent is a rating of how the caller felt during the call, usually on a negative-to-positive scale. It is derived from the transcript words and, ideally, the audio tone. It can be reported per turn, per call, or as a curve tracking mood over the whole conversation.

How do you measure caller sentiment from audio?

Caller sentiment from audio is measured by extracting acoustic features and mapping them to emotion. The main features are pitch, driven by fundamental frequency, plus loudness and speaking rate. A rising, strained pitch signals stress or anger. This acoustic layer catches feeling that never reaches the transcript, which is why audio scoring beats words alone for voice.

Why does transcript-only sentiment miss an angry tone?

Transcript-only sentiment misses an angry tone because a transcript strips prosody and paralanguage. The words "that's fine" score neutral, even when said through clenched teeth. Emotion in speech lives in how words are said, not just which words. Reading text alone flattens delivery, so a furious caller and a calm one can produce identical, misleading scores.

Should sentiment be scored per turn or per call?

Sentiment should be scored both per turn and per call, because they answer different questions. Per-turn sentiment shows exactly when the mood moved and what the agent said right before, which is what you need to debug. Whole-call sentiment gives one number for dashboards and week-over-week trends. A per-call average alone hides the moments that mattered.

What is sentiment trajectory in a voice agent call?

Sentiment trajectory is the shape of sentiment across a call, plotted turn by turn from start to finish. It shows whether a rough patch recovered or the call kept sliding. Two calls can share the same ending score but tell opposite stories. The slope out of the low point reveals whether the agent handled the hard moment or worsened it.

Can a sentiment score trigger escalation to a human?

A sentiment score can trigger escalation to a human, and a falling curve is often a better trigger than any keyword. Set a threshold on the score and its slope. When sentiment drops sharply over a few turns, hand off. Use a short trend, not one noisy turn, and tune the threshold per use case, since some calls run hotter.

How accurate is voice agent sentiment analysis?

Voice agent sentiment analysis is accurate enough to act on when it is validated against human ratings and scored from audio, not words alone. Measure agreement with inter-rater reliability, and judge the model with precision and recall on the calls you most needed to catch. An unvalidated score is a guess, so always calibrate against people before you trust it.

Who should measure a voice agent's sentiment score?

An independent, third-party evaluator should measure a voice agent's sentiment score, because the builder or vendor has a reason to report the flattering number. Words-only sentiment is easy to game and tends to look good. An external party like Evalgent scores tone from the audio and runs on your own calls, so the result is defensible to buyers and regulators.

The bottom line

A sentiment score is only trustworthy when it reads tone from the audio, not words alone. Track it turn by turn so you catch a call going wrong while you can still save it.

Sentiment is not a vanity metric. Done right, it is an early-warning system that tells you when to escalate before a caller ever asks. The catch is that the easy version, transcript-only scoring, is the one most likely to lie to you.

Want a sentiment score you can defend, scored from audio and validated against people? Book a demo and see how Evalgent scores your agents as an independent third party.