Evaluating STT for Names and Entities

# Evaluating STT for names and entities

Quick answer

STT entity accuracy voice agents rely on is the share of names, numbers, and dates the speech engine transcribes correctly, not overall word error rate. A voice agent can post low WER and still mishear one account number. Measure entity error rate separately, on your own domain vocabulary, with an independent evaluator.

A voice agent lives or dies on a few words per call. The caller's name. Their street. Their account number. The dollar amount they owe. Miss any one of those, and the whole task breaks.

Yet most teams grade speech-to-text on a single averaged score. That score can look great while the words that matter go wrong. This post explains why, and how to test the right way.

Why aggregate word error rate hides the errors that matter

Word error rate is the standard metric for speech-to-text. It counts substitutions, deletions, and insertions across a whole transcript. Then it divides by the number of words. You can read the formal definition of word error rate for the math.

The problem is the average. A call has many words. Most of them are filler. "Um," "yeah," "okay," "so." Getting those right pads the score.

Entities are rare by comparison. A five-minute call might carry one name and one account number. If the engine nails 297 of 300 words, WER looks near perfect. But if the three misses are the name, the ZIP, and the last digit of the card, the call fails.

That is the trap. Low WER, broken task. The average rewards the easy words and hides the expensive ones. Our own read of transcripts backs this up: the words that break calls are almost never the common ones.

Character-level metrics help a little. They catch near-misses inside a word. We compare the two in our guide on WER versus CER. But neither one, on its own, tells you whether the name was right.

What counts as an entity in a voice call

An entity is a specific, meaningful chunk of information. It refers to one real thing. A person, a place, a number, a code. The field of named-entity recognition studies how to find these spans in text.

For a voice agent, the list is practical, not academic. These are the entities that drive an action:

• Names of people and companies
• Street addresses, cities, and ZIP codes
• Account numbers, member IDs, and case numbers
• Dates, times, and appointment slots
• Dollar amounts and quantities
• Product names, SKUs, and part codes
• Email addresses and confirmation codes

Every one of these feeds a lookup, a booking, or a payment. A single wrong character routes the whole call to the wrong record. That is why entity accuracy is its own problem, separate from general transcription.

Entity types, why they are hard, and how to test them

Not all entities fail the same way. Each type has its own trap. The table below maps the common ones. Use it to build a test set that stresses the parts your agent actually depends on.

Why entity accuracy must be measured on your domain vocabulary

Generic benchmarks use generic words. Your callers do not. A pharmacy hears drug names. A lender hears loan products. A dealership hears trim levels and part codes.

An engine that scores well on news audio can still fumble your vocabulary. It never saw those words in training. The rarer and more specialized the term, the higher the risk.

This is why aggregate benchmarks mislead buyers. The right test uses your words, your accents, and your call conditions. We make the case for testing on real calls in our piece on benchmarking voice agents on your own data.

Domain vocabulary also shifts. New products launch. New plan names ship. A test set built once and left alone goes stale. Refresh it as the business changes.

Homophones, spelling, and the phonetic trap

Many entity errors are not random. They are phonetic. Two words sound alike, so the engine picks the wrong one. These are homophones, and they are a top source of entity failure.

"Reed" and "Reid." "Steven" and "Stephen." "Sales" and "Seals." The audio is the same. The correct spelling depends on context the engine may not have. The mapping from sound to written letters, or graphemes, is not one-to-one.

Numbers have their own confusions. "Fifteen" and "fifty." "Fifty" and "fifty-five" clipped in fast speech. A single vowel decides a payment.

To catch these, score with more than exact-match. A phonetic algorithm like Soundex groups words that sound alike. Pair it with edit distance, or Levenshtein distance, to measure how far a transcript sits from the truth. Together they separate a true miss from a spelling variant.

Spelling matters even when the sound is right. "Jon" is not "John" in a record lookup. For entities, the written form is the answer, not an approximation. Our guide on pronunciation in voice agents goes deeper on the sound-to-text gap.

How to run an entity-accuracy evaluation

Here is the process we use to grade entity accuracy on real calls. Follow it in order.

1. Define your entity types. List the entities that drive an action in your flow. Names, IDs, amounts, dates. Ignore words that do not change the outcome.

2. Build an entity-focused test set. Pull real calls, or script realistic ones. Weight the set toward hard cases: rare names, long numbers, homophones, accents. Cover both clean and noisy audio.

3. Label the ground truth. For each clip, record the correct entity value, character by character. This is your answer key. Get it right; a bad key poisons every score.

4. Run the audio through the engine. Transcribe each clip with the STT setup you actually ship. Same model, same settings, same audio pipeline.

5. Extract the entities from each transcript. Pull the spans that should hold the entity. Align them to the labeled positions so you compare like with like.

6. Score each entity separately. Mark exact matches. Then add a phonetic check and an edit-distance check to sort near-misses from true errors. Score IDs and numbers character by character.

7. Compute entity error rate per type. Report a separate number for names, addresses, IDs, and amounts. One blended score hides the weak spot.

8. Trace failures to a cause. Group misses by pattern: accent, noise, homophone, fast speech. Fix the biggest cluster first, then re-run the set.

Run this on a schedule, not once. Models update. Vocabulary drifts. A monthly pass catches regressions before callers do. The same discipline applies to a full voice agent accuracy audit.

Entity error rate: the metric to report separately

Entity error rate is the fraction of target entities the engine gets wrong. Compute it per entity type. Names get one number. Account numbers get another.

> Entity error rate: the share of a specific entity type that the speech engine transcribes incorrectly, scored on its own rather than blended into overall word error rate.

Report it beside WER, never instead of it. WER tells you the general quality. Entity error rate tells you whether the call can succeed.

For entities that must be exact, like IDs, treat any character miss as a full failure. There is no partial credit on an account number. For names, you can track exact spelling and phonetic match as two figures.

Borrow the framing of precision and recall here. Precision asks how many extracted entities were right. Recall asks how many true entities you caught. A good report shows both, per type.

Test on the transcript alone and you can miss why an entity failed. Sometimes the audio was clear and the model still slipped. We cover that split in transcript versus audio evaluation.

Where an independent evaluator fits

Vendors report the number that flatters them. That is usually aggregate WER on clean audio. It says little about your callers saying your product names in a noisy car.

An independent evaluator measures what you cannot see from the outside. Evalgent scores entity accuracy on your real calls, by entity type, on your domain vocabulary. Not a generic benchmark. Your names, your IDs, your accents.

This matters most in regulated and high-stakes flows. Healthcare intake. Collections. Insurance claims. One wrong digit is a compliance event, not a rounding error. Independent, third-party measurement gives you evidence, not a vendor's word.

• For teams building an agent: entity error rate tells you which STT engine to ship, and where the flow needs a confirmation step.
• For teams buying an agent: it tells you whether a vendor's claim survives contact with your callers.
• When to skip it: if your flow captures no names, numbers, or IDs, general WER may be enough. Most real flows are not that simple.

Entity accuracy sits inside the wider practice of STT evaluation for voice agents and full voice agent evaluation. Accents deserve their own stress test too, covered in our guide on accent handling in voice agents. And if the terms trip you up, start with STT versus ASR.

Frequently asked questions

What is entity error rate for a voice agent?

Entity error rate is the share of target entities a speech engine transcribes incorrectly, scored per type. Names, addresses, account numbers, and amounts each get their own figure. It is reported beside word error rate, not instead of it. Unlike a blended average, it shows whether the specific values that drive an action came through right.

Why does low word error rate still miss names?

Word error rate averages over every word in a call. Most words are common filler that engines get right, which pads the score. Names and numbers are rare by comparison. An engine can miss the one name and one account number and still post a strong average, because a few misses barely move a whole-transcript number.

How do you test a voice agent on names and addresses?

Collect real caller names and full addresses, including diverse and non-English names and unit numbers. Label the correct spelling as ground truth. Transcribe with your shipped STT setup, extract the entity spans, and compare character by character. Add a phonetic check to separate true errors from spelling variants, and test both clean and noisy audio.

How do you measure STT accuracy on account numbers?

Score account numbers character by character, not as whole words. Treat any single-digit miss as a full failure, because there is no partial credit on an ID. Build clips with long digit strings, spoken fast and in noise. Compute a separate entity error rate for IDs so a strong overall score cannot hide a transposed digit.

Why do voice agents mishear homophones?

Homophones sound identical but spell differently, like "Reed" and "Reid." The audio gives the engine no way to pick the right spelling without context it may lack. Numbers have similar traps, such as "fifteen" versus "fifty." Because the written form is the answer for an entity, a correct sound with the wrong spelling still fails the lookup.

Should you measure entity accuracy on your own data?

Yes. Generic benchmarks use generic words, but your callers say your product names, plan names, and part codes. An engine that scores well on news audio can fumble specialized vocabulary it never saw in training. Testing on your own calls, accents, and noise conditions is the only way to know how the agent performs where it matters.

What entities matter most for voice agent accuracy?

The entities that trigger an action matter most: names, street addresses, account and member IDs, dates and times, dollar amounts, and product codes. Each feeds a lookup, a booking, or a payment. A single wrong character routes the call to the wrong record. Prioritize the entities that carry the highest cost when they fail.

How is entity accuracy different from intent accuracy?

Entity accuracy measures whether the specific values in a call, like a name or number, are transcribed correctly. Intent accuracy measures whether the agent understood what the caller wanted to do. An agent can read the intent right and still fail because it misheard the account number. Both need separate measurement, since one does not predict the other.

The bottom line

Aggregate word error rate can look excellent while the names, IDs, and amounts that decide a call go wrong. Measure entity error rate separately, on your own domain vocabulary, and confirm it with independent evaluation.

Want to know your true entity accuracy on real calls? Book a demo and we will score names, numbers, and IDs on your own data.