How to Monitor ASR Drift and Speech-to-Text Accuracy in Production

On this page
Here is a case teams will recognise. Nobody on your side ships a release. A week later, callers who spell their last names are asked to repeat themselves twice as often. Task completion barely moves, so no dashboard turns red. The speech-to-text layer changed underneath you, and the first sign is a pile of wrong names in your CRM.
That is ASR drift. This post covers ASR drift monitoring voice agent teams can run on live traffic. It is a different job from the neighbouring posts. The metric drift guide covers drift in general. The accuracy audit is a one-off measurement. The STT cost and accuracy post helps you choose an engine, and the background noise test is a pre-release test. Here the question is narrower: how do you notice, this week, that speech-to-text accuracy in production has moved, and what do you do next?
All numbers in worked examples are illustrative. Provider facts are dated as of November 19, 2026.
What ASR drift is and why it hides
ASR drift: a sustained change in how well your speech recognition turns real caller speech into text, caused by something other than a change you made to the agent.
Two things make it hard to see. First, the transcript is the input to everything downstream. The language model reads it, the tool call uses it, and the write-back stores it. A wrong word does not raise an error. It becomes a wrong record.
Second, you rarely have a reference. Word error rate needs the true words, as the word error rate guide explains. In production nobody has typed up the true words for every call. So you monitor proxies, and you check them against a small labelled sample.
If you want the offline side, the STT evaluation guide covers harnesses and normalization. This post starts where that one stops: after launch, with live calls and no ground truth.
What changes underneath your ASR
Drift has causes. Knowing them tells you which slice to check first.
| Cause | What moves | First place to look |
|---|---|---|
| Provider model update or retirement | Errors on names, numbers, accents | STT model label per call |
| Shift in caller mix | Blended error rate | Cohorts: language, region, device |
| Channel or carrier change | Clipped or muffled input | Channel and carrier per call |
| New vocabulary | Product names, places, jargon | Entity failures by type |
| Change in what the agent asks | Kind of speech heard | Question or slot per turn |
Provider updates. Version behaviour differs by vendor, so read your own provider's page. Deepgram's version parameter defaults to latest, and you can send an earlier version number instead. If you never set it, your traffic follows the provider's releases. AssemblyAI's model versions page says it gives at least 90 days' notice before retiring a model. It also says requests naming a retired model "are automatically routed to its replacement model." That page recommends testing the replacement on your own audio before the date. Those are both documented, planned changes. They can still shift your error profile.
Caller mix. A well-known study, Koenecke et al. (PNAS, 2020), tested five commercial ASR systems. The average word error rate was 0.35 for Black speakers and 0.19 for white speakers, across interviews with 73 and 42 speakers. The lesson for monitoring is simple. Error rates differ by speaker group. If a campaign changes who calls, your blended rate moves with no model change at all.
Channel. If you stream phone audio through Twilio, Media Streams carries it as mu-law at 8,000 Hz. Other paths may deliver different formats. Record the channel on every call, so a routing change shows up as a cohort and not a mystery. The phone audio quality post goes deeper on that leg.
Vocabulary and questions. A new product line adds words your recognizer never met. A prompt edit that asks callers to spell a name changes the speech you receive. That second cause is why drift monitoring and prompt regression monitoring must share call tags.
Five signals you can compute without audio
You do not need to re-process audio to monitor ASR drift. These five signals come from transcripts, provider fields and call outcomes you already log. Each has a blind spot, listed in the table.

1. Low-confidence share
Most providers return a score per word. Deepgram describes it as a value between 0 and 1, the model's estimated probability that the word was transcribed correctly. AssemblyAI says its scores come from model probabilities and that the transcript-level score is a plain average of word scores. Google documents word-level confidence as a request option, and on November 19, 2026 that page labelled the feature Preview.
Track the share of words below a cut you choose, per call and per slot. Do not track the transcript average. An average can hide a few bad words, and the bad words are the names and numbers.
Treat the score as a relative signal. A 2017 ICML paper, On Calibration of Modern Neural Networks, found that modern networks are poorly calibrated. It studied classifiers, not speech recognizers, so read it as a caution, not a measurement. Compare your confidence against your own past, never against another provider's score.
2. Re-ask rate
Count agent turns that ask the caller to repeat, such as "sorry, could you say that again." Detect them from the agent's own text. Your prompt controls that wording, so keep it stable or the signal breaks.
3. Read-back rejection rate
When the agent reads a value back and the caller says no, the caller has labelled an error for you. This is the closest thing to free ground truth in a live call. Count the rejections per entity type: names, dates, phone numbers, addresses.
4. Empty and short transcripts
Track turns with no words, or far fewer words than usual for that question. A rise can mean clipped audio, a bad endpointing setting or a recognizer that stopped hearing a cohort.
5. Entity validation failures
Count values that fail a format check or a lookup. A phone number with nine digits, a date in the past, or an account number with no match are all examples. The entity accuracy guide explains why these errors matter more than the average.
Then add one outcome signal: how the call ended after a re-ask. Hangups right after a repeat request are a cost of drift you can price.
Slice before you chart
A blended number hides a localised problem. Slice every signal by five keys:
- STT model label. The provider, model name and any version string.
- Question or slot. What the agent was asking when the caller spoke.
- Language or locale. The multilingual STT post covers why this matters.
- Channel and carrier. Where the audio came from.
- Prompt version. So you can rule out your own release.
Here is an illustrative case. A slot where callers spell a last name makes up 10% of turns. Re-ask rate on that slot rises from 12% to 18%. On every other slot it stays at 3.1%.

The blended rate moves from 4.0% to 4.6%. With 4,000 turns in the window, a three-sigma limit sits near 4.9%. The blended number stays inside it. The slot has 400 turns, and its limit sits near 16.9%. The slot's 18% is outside. Only the slice shows the problem.
Baselines and alerts that fit ASR signals
You do not need new statistics. Use a control chart you already know, and keep the alert rules boring. The prompt regression post builds the baseline and limit method step by step, and the metric thresholds guide covers where limits come from.
For a rate, the NIST/SEMATECH handbook gives the p-chart: center line p, limits p plus or minus 3 times the square root of p(1 minus p) over n. It assumes a stable process and independent units. Calls are not perfectly independent, so treat limits as a guide.
A p-chart reacts to steps. ASR drift often arrives as a slope. The handbook's EWMA chart is built for that. It notes a Shewhart chart "can only react when the last data point is outside a control limit," while EWMA "can be made sensitive to a small or gradual drift." It also says lambda is usually set between 0.2 and 0.3, though it calls the choice "somewhat arbitrary."
# Simplified, illustrative. One rate per day for one slice.
from math import sqrt
def ewma_alarms(daily_rates, baseline, daily_sd, lam=0.2, k=3):
limit = k * daily_sd * sqrt(lam / (2 - lam))
ewma, alarms = baseline, []
for day, rate in enumerate(daily_rates):
ewma = lam * rate + (1 - lam) * ewma
alarms.append(day if ewma - baseline > limit else None)
return [d for d in alarms if d is not None]Three rules keep the alerts honest:
- Gate on volume. The Google SRE Workbook says "It's harder to automatically distinguish unimportant events in low-traffic services." Thin slices need a minimum number of turns before any alert.
- Compare like with like. Weekday and weekend callers differ. Compare the same weekday to the same weekday.
- Tier the response. A slow rise in re-asks is a ticket. A rise that comes with falling task success is a page. The alerting guide explains the tiers.
How to monitor ASR drift in production
Run these seven steps once, and your ASR drift monitoring voice agent setup runs quietly. Each step is cheap on its own.
1. Tag every call - Store the STT provider, model name and version string, language, channel and carrier on each call record, plus your prompt version.
2. Keep word-level detail - Save the final transcript, per-word confidence where the provider returns it, and the question or slot for each caller turn.
3. Compute the five signals per slice - Low-confidence share, re-ask rate, read-back rejections, empty transcripts and entity failures, daily by slice, with hourly for high-volume slices.
4. Set baselines from a stable window - Use at least two full weeks without incidents, and merge thin slices rather than set limits on a few dozen turns.
5. Alert on steps and slopes - Use a p-chart for jumps and an EWMA for gradual change, gated by minimum volume and tiered into ticket or page.
6. Label a weekly sample - Have a person write the true words for a random set of turns and a set of flagged turns, then compute error rates with intervals.
7. Write the response playbook - Decide in advance who checks the model label, who can pin or roll back a version, and who reruns your golden set.
Step 6 needs stored recordings, and recording carries legal duties that vary by place. See the recording consent guide. This is not legal advice.
Size the labelled sample
The signals tell you where to look. A labelled sample tells you whether accuracy really moved. The sample size decides how much you can conclude.
For a proportion, such as "share of last names transcribed wrongly," the handbook gives the Wilson interval. At 9% wrong, the 95% interval narrows with sample size as the chart shows.

Take 400 labelled names with 36 wrong. The interval runs from 6.6% to 12.2%. Suppose next month 52 of 400 are wrong, which is 13%, with an interval of 10.1% to 16.7%. The two intervals overlap. Four points of change is real in your data, yet 400 labels cannot confirm it at that confidence.
Two practical consequences follow. Pool several weeks when a slice is small. And use signals 1 to 5 to pick which turns to label, since a random sample alone will rarely hit a rare slot.
Word error rate is not a simple proportion, so the interval is different. Bisani and Ney's ICASSP 2004 paper, Bootstrap Estimates for Confidence Intervals in ASR Performance Evaluation, gives a bootstrap approach for it. Resample whole utterances, not single words, because words within one utterance are not independent. Normalise text the same way each week, as the STT evaluation guide describes, or your trend will chase formatting changes.
Where this monitoring misleads you
Proxies fail in known ways. Check these limits of ASR drift monitoring voice agent dashboards before you blame the recognizer.
- Confidence is not accuracy. It is the model's own certainty. A version change can shift scores without changing errors, or the reverse.
- Deletions are invisible. A word the model never output has no confidence score.
- Re-asks have other causes. The language model may misread a clear transcript. A label sample separates the two.
- Mix shifts look like model shifts. Compare the same cohort before and after. If only the mix changed, the model did not.
- Your own edits move the signals. A reworded question changes re-ask text and the speech you receive.
- Transcripts hide the caller's real words. Only a person with the recording can say what was spoken.
That last limit is why step 6 exists. Treat every alarm as a lead until a label confirms it.
Where Evalgent fits
Evalgent Monitor takes your production calls, maps them to scenarios and judges them. It flags problems such as PII leakage, tool-call failures, latency spikes and policy failures, with fix recommendations. A speech error that corrupts a captured name tends to surface there as a failed step or a wrong tool payload. Monitor does not compute word error rate or measure speech-to-text accuracy from the audio, so keep your own labelled sample for that.
Before you switch STT versions, you can test the change first. Evalgent test calls use caller Profiles with seven parameters: interruption level, speech pace, language, response delay, behaviour, accent and background noise. Reviews replay each test call with audio and transcript together and show the exact tool payloads. For the offline comparison itself, the how to improve WER guide lists the levers worth trying. If you want to set this up for your first agent, you can book a demo.
Frequently asked questions
What is ASR drift in a voice agent?
ASR drift is a sustained change in how accurately your speech recognition transcribes real callers, caused by something outside your agent's code. Common causes are a provider model update, a different caller mix, a new carrier path or new vocabulary. It shows up as wrong names, numbers and repeated questions.
How do you detect ASR drift in production?
Track five signals from stored data: low-confidence share, re-ask rate, read-back rejections, empty transcripts and entity validation failures. Slice them by model label, question, language and carrier. Alert against your own baseline with a p-chart and an EWMA chart. Then confirm with a human-labelled sample.
Does an STT confidence score show real accuracy?
No. A confidence score is the model's own estimate, not a measured error rate. Deepgram describes it as an estimated probability, and AssemblyAI derives it from model probabilities. Neural networks can be poorly calibrated. Use scores to compare your calls over time, not to certify accuracy.
Can you measure word error rate in production without audio?
Not from transcripts alone. Word error rate compares a transcript with the true words, and the true words come from a person listening to a recording. Without that reference, monitor proxies such as re-asks and read-back rejections, and label a small sample each week.
How many calls do you need to label to estimate WER?
There is no fixed number. The interval width depends on how many utterances you label and how varied they are. For a 9% error rate, 400 labelled items give roughly a 6.6% to 12.2% interval. Pool weeks for small slices, and bootstrap over utterances.
How often should you check speech-to-text accuracy?
Compute the proxy signals daily, hourly for high-volume slices. Label a sample weekly, and add an extra sample right after a provider change or a campaign that changes who calls. Match the cadence to call volume, since thin slices need longer windows.
What should you do when drift follows a provider update?
Check the model label on affected calls first. If your provider lets you pin a version, as Deepgram's version parameter does, pin the earlier one while you investigate. Rerun your golden set on both versions, compare per cohort, and add custom vocabulary only after the cause is clear.
Is ASR drift monitoring different from testing STT before release?
Yes. Pre-release testing runs fixed audio under controlled conditions, such as noise and accents. Drift monitoring watches live calls whose mix, channel and vocabulary keep changing. You need both, and the same labelled cases can serve each job.
The bottom line
Monitor ASR drift by tracking proxy signals by slice against your own baseline, and check every alarm with a small labelled sample. Speech-to-text accuracy in production is something you estimate with intervals, never something a single confidence score certifies.
Related Articles

How to automate voice agent testing: synthetic callers vs manual QA
Learn how ai test automation replaces manual QA for voice agents. Compare synthetic callers vs human testers, with a 5-step framework to scale without hiring.
Read more
AI Agent Testing vs Voice Agent Testing: What General Tools Miss for Voice
AI agent testing measures text outputs. Voice agent testing measures behaviour through an acoustic pipeline. Five failure categories general tools miss.
Read more