Evaluating Multilingual Accuracy for Voice Agents

# Evaluating multilingual accuracy for voice agents

Quick answer

Multilingual accuracy for voice agents is how well an agent transcribes, understands, and acts across each language it serves. A single blended score hides the truth. Measure word error rate and character error rate per language, test code-switching and language detection, and set a separate pass bar for every language on your real calls.

Your vendor says the agent is "95% accurate." Accurate in which language? On a clean English demo, most agents look strong. Put a caller who switches between English and Spanish mid-sentence on a cell phone, and the same agent can fall apart. A single accuracy number averages away exactly the failures your multilingual callers will hit.

This guide explains why one number lies, which dimensions actually break, and how to build a per-language and code-switching test set. It closes with a step-by-step method for running the evaluation on your own traffic. For the broader discipline, see our overview of voice agent evaluation.

Why one accuracy number lies when you serve many languages

A blended accuracy figure is a weighted average. If 80% of your calls are English and the agent scores well there, the number stays high even if Spanish accuracy is poor. The languages your smaller caller groups speak get buried in the mean.

This matters at scale in the US. The Census Bureau reports that nearly 68 million people spoke a language other than English at home in 2019, with Spanish the most common at 62%. A US voice agent is a multilingual service whether or not it was designed as one. Serving those callers well is a real requirement, and multilingualism is the norm for a large share of them.

> Multilingual accuracy: the correctness of a voice agent measured separately for each language and language mix it handles, rather than as a single pooled score. It covers recognition, understanding, and action in every language served.

The fix is not a better average. It is refusing to average at all. You measure each language on its own, report the results side by side, and hold each to its own bar. The rest of this guide shows how.

Where multilingual accuracy actually breaks

Four distinct problems hide under a blended score. Each has a different cause and a different test. Understanding them is the difference between a real evaluation and a demo.

The first is per-language recognition quality. The second is code-switching, where a caller mixes languages in one utterance. The third is language identification, where the agent picks the wrong language and misroutes the call. The fourth is accent variation inside a single language. The table below is the core of the evaluation scope.

Every row is scored on your calls, using human-labeled ground truth. Vendor benchmarks rarely cover any of them well. We explain why vendor numbers are not evidence in independent voice AI evaluation.

Per-language WER and CER, measured separately

Recognition is the foundation. If the agent mishears the words, everything downstream inherits the error. The standard recognition metric is word error rate, or WER. It counts substitutions, insertions, and deletions against a human reference transcript, then divides by the reference word count. Lower is better.

WER is computed from an edit distance, the same Levenshtein distance used to compare any two strings. Character error rate, or CER, applies the same idea at the character level. CER matters for spelled-out names, account numbers, and languages where word boundaries are less clean. We compare the two in our guide to WER vs CER.

The key move is to report these per language, never pooled. A WER of 0.06 in English and 0.19 in Spanish is not a WER of 0.09. It is a passing grade in one language and a failing grade in another. Pooling hides the second number and the callers behind it. For the recognition layer specifically, see STT evaluation for voice agents.

Error rates differ by language for a structural reason. Recognition engines are trained on uneven data. Well-resourced languages get lower error rates. Less-resourced languages get higher ones. You cannot assume a provider that is strong in English is strong in every language you serve.

Code-switching breaks models built for one language

Code-switching is when a speaker alternates languages within a single conversation or even a single sentence. Code-switching is common among bilingual US callers. A caller might say, "I need to cancel mi cita for Thursday." One clause is English, the next is Spanish, and the sentence is normal speech for millions of people.

Many recognition and understanding models assume one language per turn. When a caller mixes languages, these models can transcribe the foreign-language words as garbled English, drop them, or misclassify the whole intent. The failure is invisible in a monolingual test set because a monolingual set never contains a mixed utterance.

So you have to build for it directly. Collect real mixed-language utterances from your call logs. Label them with a human transcript and the true intent. Score both transcript accuracy and intent accuracy on this set as its own category. A code-switching WER of 0.30 next to an English WER of 0.05 tells you exactly where the agent is weak.

Do not settle for a synthetic script read by one bilingual speaker. Natural code-switching has its own rhythm and vocabulary. Your own recordings capture it; a scripted read does not. This is one reason evaluation on real audio beats evaluation on clean text, a point we develop in transcript vs audio voice agent evaluation.

Language identification errors misroute the whole call

Before an agent can respond in the right language, it has to detect the language. This step is language identification, and it is a frequent, quiet failure point. On short greetings or noisy lines, the detector has little signal and guesses wrong.

A wrong guess is expensive. The agent may answer a Spanish caller in English, route to the wrong prompt set, or hand off to the wrong queue. The caller experiences a broken call before a single word of content is exchanged. Because language ID happens first, its errors cascade into every later metric.

Test it as its own dimension. Feed the detector clips whose language you already know, across all the languages you serve, and measure detection accuracy per language. Include short utterances and noisy audio, because those are where detection breaks. Tagging clips with standard ISO 639 language codes keeps your labels consistent.

Then test the fallback. When the detector is not confident, what does the agent do? A good fallback asks the caller to confirm the language or offers a menu. A bad one silently commits to a guess. Your evaluation should score the low-confidence path, not just the confident one.

Accents within a language still matter

One language is not one accent. Spanish in the US spans many regional varieties, and English callers bring a wide range of accents too. Training data skews toward a few dominant accents, so recognition quality can drop sharply for the rest. A single per-language number can still hide an accent-level failure.

Segment your test set by accent and region where you have enough data. Compare WER for each segment against the same pass bar. If one accent group scores far worse, you have found a real gap that a language-level average would have smoothed over. Our guide to accent handling for voice agents covers this in depth.

Pronunciation on the output side deserves the same scrutiny. An agent that mispronounces Spanish names or place names reads as careless to the caller. We cover measuring this in pronunciation for voice agents.

How to run a multilingual accuracy evaluation

Here is a repeatable method. It works whether you are choosing a vendor or auditing one already in production. The principle throughout is per-language measurement on your own audio.

1. List every language and mix you serve. Pull your call logs and identify the actual distribution. Include code-switching as its own category, not an afterthought. This list defines your test scope.

2. Sample real calls per language. Draw a representative sample for each language and for code-switched calls. Include hard conditions: background noise, cell audio, rare names, and short utterances.

3. Create human-labeled ground truth. For each clip, have a fluent human produce a verified transcript, the true intent, and the correct entities. This is the reference every metric is scored against.

4. Measure recognition per language. Compute WER and CER for each language and for the code-switching set separately. Never pool the results into one figure.

5. Test language identification and fallback. Feed known-language clips, measure detection accuracy per language, and score what the agent does when confidence is low.

6. Score understanding and action per language. Measure intent and entity accuracy, then task completion, for each language. A correct transcript does not guarantee a correct action.

7. Set a pass bar for each language. Define the maximum acceptable WER and minimum acceptable task success per language, based on the stakes of those calls. Hold every language to its own bar.

8. Re-run on a schedule. Providers update models, and accuracy shifts. Repeat the evaluation on fresh samples so a silent regression in one language does not reach production unnoticed.

This method is the multilingual case of a general accuracy audit. For the full audit scope across all dimensions, see voice agent accuracy audit, and for running it on your own traffic, benchmark voice agents on your own data.

Setting per-language pass bars

A pass bar is the threshold a language must clear to ship. It is not one global number. High-stakes calls in any language, like collections or healthcare intake, need a stricter bar than low-stakes ones. The bar should reflect the cost of an error on those specific calls.

Write the bars down before you test. Otherwise the temptation is to accept whatever number the agent produces. A concrete rule, such as "Spanish task success must reach the same threshold as English," turns a vague goal into a testable gate. When a language misses its bar, that is a finding to fix, not a number to average away. For choosing a vendor against explicit criteria, see how to evaluate voice agent vendors.

How Evalgent measures per-language accuracy

Evalgent is an independent, third-party evaluator for AI voice agents. We do not build the agents we test, so we have no stake in the result. That independence is the point. A provider that picks its own test set, metric, and conditions can produce almost any headline number. An outside measurement on your real calls cannot be gamed the same way.

For multilingual accuracy, we score each language separately on your own audio against human-labeled ground truth. We report WER and CER per language, build a code-switching set from your logs, test language identification and its fallback, and segment by accent where the data supports it. You get a per-language scorecard with pass bars, not a single number that hides your weakest language. That is the difference between knowing your agent works for every caller and hoping it does.

Frequently asked questions

What is multilingual accuracy for voice agents?

Multilingual accuracy is how correctly a voice agent transcribes, understands, and acts in each language it serves. It is measured per language, not as one pooled score. A strong multilingual evaluation reports word error rate, intent accuracy, and task success separately for every language and for code-switched calls on your own real audio.

Why does a single accuracy number mislead for multilingual agents?

A single number is a weighted average dominated by your most common language. If most calls are English and English scores well, the blended figure stays high even when Spanish accuracy is poor. The languages your smaller caller groups speak get buried in the mean, so real failures stay hidden until production.

How do you measure accuracy per language?

Sample real calls for each language, create human-labeled ground truth transcripts and intents, then compute word error rate and character error rate for each language separately. Add intent accuracy and task success per language. Report the results side by side and never pool them into one figure that hides the weakest language.

What is code-switching and why does it break voice agents?

Code-switching is mixing two languages in one conversation or sentence, common among bilingual US callers. Many recognition models assume one language per turn, so they garble or drop the switched words and misclassify intent. Because monolingual test sets contain no mixed utterances, this failure stays invisible unless you build a code-switching test set from real calls.

How do you test language identification in a voice agent?

Feed the language detector audio clips whose language you already know, across every language you serve, and measure detection accuracy per language. Include short greetings and noisy audio, where detection fails most. Then test the fallback: check what the agent does when confidence is low, since a wrong guess misroutes the entire call.

Do accents matter within the same language?

Yes. One language spans many accents, and training data skews toward a few dominant ones, so recognition quality can drop for the rest. A language-level average can hide an accent-level failure. Segment your test set by accent and region where you have enough data, then compare each segment's word error rate against the same pass bar.

What is a per-language pass bar?

A per-language pass bar is the threshold a language must clear before the agent ships, such as a maximum word error rate or minimum task success. It is set per language based on call stakes, not as one global number. Writing bars down before testing turns a vague accuracy goal into a concrete, testable gate.

Should multilingual accuracy be tested on real calls or synthetic audio?

Real calls. Synthetic scripts read by one speaker miss the noise, rhythm, and natural code-switching of actual traffic. Your own recordings contain the hard cases: cell audio, crosstalk, rare names, and mixed-language sentences. Evaluating on real audio with human-labeled ground truth gives numbers that predict production behavior, not demo-condition results.

The bottom line

A single accuracy number hides your weakest language and the callers who speak it. Measure word error rate, code-switching, and language detection separately for every language on your own real calls, and hold each to its own pass bar.

Want to see per-language accuracy on your actual traffic? Book a demo and we will show you where your agent passes and where it fails.