Voice Agent Alphanumeric Capture: Where Codes and VINs Break

Your voice agent confirmation code flow looks healthy. The caller said "four eight two one three." The agent stored 48230, read back "four eight two three zero," heard "yes," and wrote it to the record. No error was raised. The record now points at someone else's order. Voice agent order number errors like this rarely raise an alarm.

That is the failure this post is about. A code is the key to the next system: the POS ticket, the CRM row, the claim file. One wrong character does not degrade the call a little. It breaks the join. We cover where strings get corrupted, the math, and a test design you can copy. Names and free-text entities are a different problem, covered in Evaluating STT for names and entities.

Why voice agent alphanumeric capture differs from names

A name has context. "Jon" and "John" both lead somewhere, and a fuzzy search can recover. A code has none. Alphanumeric speech recognition gets no help from the language model when it must pick between "B" and "D."

Three properties make codes hard.

1. No linguistic prior. In "K7B0Q2" the neighbours say nothing about the next character.

2. All-or-nothing correctness. A name that is 90% right often still matches. A code that is 90% right matches nothing or a different record.

3. Callers speak codes in odd ways. They chunk, pause, say "double five," say "oh" for zero, and correct themselves mid-string.

The Sheffield team built SNuC, which the authors call the first published corpus of spoken alphanumeric identifiers. It holds 13,125 identifiers from 52 speakers, recorded in a quiet booth (Barker et al., LREC 2022). Even there, a LibriSpeech-trained baseline scored 98.4% at word level but only 81.7% on whole identifiers. On partner-company data, whole-identifier accuracy fell to 77%. The authors report that nearly one in four identifiers contained an error. Adapting the model to identifiers lifted the partner figure to 91.7%. Those are lab numbers, not your production rates.

The paper also reports that 6.2% of identifiers contained a speaker mishap, and 68.2% of those mishaps went unnoticed by the speaker. Callers misread codes too.

Six places voice agent alphanumeric capture goes wrong

The agent hears a code, converts it to text, validates it, looks it up and reads it back. Each hop can change a character. Most teams only watch the transcript, which is the middle of the path.

Stages 3 to 6 are yours, not the STT vendor's. You can fix them in code and test them without touching the model. The voice agent testing guide covers the wider setup.

Letter confusion is predictable, so test it on purpose

Some letter pairs fail far more often than others. The best-known hard group is the E-set: B, C, D, E, G, P, T, V and Z. A 1990 ICSLP paper on spelled-letter recognition called it "a very difficult subset of nine letters." Its E-set-specific features cut the E-set error rate by 20% against a full-alphabet baseline (ISCA archive).

The confusable sets are small and known. You need a deliberate stress set, not a huge corpus.

Teams also add M/N and F/S, and the digit-letter pairs 0/O, 1/I, 5/S and 8/B. We found no primary source for those. Treat them as hypotheses and measure them.

Phone audio makes this worse. Narrowband legs carry less high-frequency detail, which hurts fricatives such as S and F. Our post on voice call quality metrics covers codecs and packet loss. Run the stress set on the codec path callers use, not on clean WAV files.

The compounding math

For voice agent alphanumeric capture, whole-string accuracy is roughly per-character accuracy raised to the string length, if errors are independent. They are not fully independent, but the formula is a useful upper-bound sanity check.

Worked example (illustrative, not measured):

A system reporting "99% character accuracy" sounds excellent. On a VIN, it means about 16 calls in 100 end with a wrong string. If your agent handles 5,000 VIN calls a month, that is roughly 790 bad lookups. The number is illustrative, but the shape is real. A healthy word error rate can hide wrong downstream records.

The SNuC gap has this shape: 98.4% compounded over about twelve to thirteen tokens is near 82%, and the paper reports 81.7% for identifiers of roughly 6 to 14 words. We are not claiming the paper used this formula. Expect the gap, and report both numbers.

Sample size for a whole-string metric

To estimate whole-string accuracy near 85% within plus or minus 3 points at 95% confidence, you need about 545 strings. For plus or minus 5 points, about 196. The formula is n = 1.96² × p(1−p) ÷ margin².

Speech-to-text normalisation is a separate failure layer

Even a perfect transcript has to become a string. Spoken digits are messy. Callers say "oh" for zero. They say "double five" and "triple zero." Pilots say "niner" and "tree"; the FAA's Aeronautical Information Manual lists those pronunciations and tells controllers and pilots to pronounce each digit separately for most numbers (FAA AIM 4-2-7 and 4-2-8). Some callers are former pilots or radio operators. Test for them.

STT vendors help partly. Deepgram, for example, documents `numerals` and `smart_format` options that convert written-out numbers to digits, across many languages for both pre-recorded and streaming audio (Deepgram numerals docs). It converts each spoken digit separately, so "four eight two" stays "4 8 2". That handles "twenty-three." It does not decide whether "oh" is the letter or zero, and it does not handle "B as in boy." Google's speech adaptation docs note that class tokens exist but that availability varies by model and language (Google Cloud STT). Check what your model supports before you rely on it.

So write a deterministic normaliser and unit-test it. A simplified version:

import re

DIGITS = {"zero": "0", "oh": "0", "one": "1", "two": "2", "three": "3",
          "tree": "3", "four": "4", "five": "5", "fife": "5", "six": "6",
          "seven": "7", "eight": "8", "nine": "9", "niner": "9"}
MULT = {"double": 2, "triple": 3}
NATO = {"bravo": "B", "charlie": "C", "delta": "D", "kilo": "K",
        "mike": "M", "november": "N", "papa": "P", "sierra": "S",
        "tango": "T"}          # extend to the full alphabet
FILLER = {"dash", "hyphen", "space", "and", "then"}

def normalise(text: str) -> str:
    toks = re.findall(r"[a-z0-9]+", text.lower())
    out, mult, i = [], 1, 0
    while i < len(toks):
        t = toks[i]
        if t in MULT:
            mult = MULT[t]
        elif t == "as" and i + 2 < len(toks) and toks[i + 1] == "in":
            if out:
                out.pop()                      # "B as in boy": keyword wins
            word = toks[i + 2]
            out.append(NATO.get(word, word[0].upper()))
            i += 2
        elif t in FILLER:
            pass
        else:
            ch = NATO.get(t) or DIGITS.get(t) or t.upper()
            out.append(ch * mult)
            mult = 1
        i += 1
    return "".join(out)

We ran it on four phrases. "K seven bravo oh Q two" gives K7B0Q2. "double five tree niner" gives 5539. "B as in boy 4 4" gives B44. "P as in Paul triple zero" gives P000. Notice what it cannot do. A bare "oh" always becomes zero, so a code with the letter O needs another rule, such as the field's known alphabet.

Validation gates: use the format the code already has

Many codes carry their own error detection. A validation gate should use it.

For voice agent VIN capture, the rules are public. Under 49 CFR 565.13, every VIN has 17 characters, and each character is a digit or one of the letters ABCDEFGHJKLMNPRSTUVWXYZ (49 CFR 565.13, via Cornell LII). The letters I, O and Q are not in the set. That is why a VIN gate can safely turn a heard "O" into zero. For an order number with no such rule, the same swap would be a bug.

Position 9 is a check digit. The regulation says its purpose is "to provide a means for verifying the accuracy of any VIN transcription" (49 CFR 565.15, via Cornell LII). The full mechanics, plus a longer validator with a confusion-aware candidate generator, are in our dealership VIN lookup QA guide. The gate below is the short version, and it reproduces the rule's own worked example, 1G4AH59H45G118341.

VIN_ALPHABET = set("ABCDEFGHJKLMNPRSTUVWXYZ0123456789")   # 565.13(g)
LETTER_VAL = dict(zip("ABCDEFGHJKLMNPRSTUVWXYZ",
    [1,2,3,4,5,6,7,8,1,2,3,4,5,7,9,2,3,4,5,6,7,8,9]))
WEIGHTS = [8,7,6,5,4,3,2,10,0,9,8,7,6,5,4,3,2]            # 565.15

def vin_gate(raw: str) -> tuple[bool, str]:
    s = raw.upper().replace("O", "0").replace("Q", "0").replace("I", "1")
    if len(s) != 17:
        return False, "length"
    if not set(s) <= VIN_ALPHABET:
        return False, "alphabet"
    total = sum((int(c) if c.isdigit() else LETTER_VAL[c]) * w
                for c, w in zip(s, WEIGHTS))
    check = "X" if total % 11 == 10 else str(total % 11)
    return s[8] == check, "ok" if s[8] == check else "check_digit"

The check digit has blind spots that match speech confusions

Here is the non-obvious part. The transliteration table gives several letters the same value. B, K and S are all 2. C, L and T are all 3. D, M and U are 4. E, N and V are 5. G, P and X are 7. If the agent swaps one letter for another with the same value, in the same position, the check digit still passes.

We tested it on the regulation's example VIN. Changing the second character from G to P gives 1P4AH59H45G118341, and the gate returns "ok." G and P both sit in the E-set. So do C/T and E/V. The check digit catches many substitutions, but it cannot catch the acoustic pairs that happen to share a value.

The lesson applies beyond VINs. A checksum is a filter, not proof. For any code with a checksum, find which confusions it cannot see, then cover those pairs with a read-back. A random wrong 17-character string passes a mod-11 check about one time in eleven, so never treat a pass as confirmation.

Gate rules to copy

1. Length and alphabet first. Reject early, and re-ask only for the missing part: "I heard fifteen characters. Please say the last two again."

2. Apply confusable repair only where the alphabet forbids a character. O to zero is safe for VINs. It is not safe for order numbers.

3. Run the checksum when one exists. On failure, do not guess. Ask for the string again in chunks.

4. Do not write on a pass alone. A passing checksum plus a caller-confirmed read-back is the write condition.

5. Log the raw transcript, the normalised string and the gate result. You cannot debug what you did not keep.

Lookup: exact first, fuzzy only inside a small candidate set

Fuzzy matching feels helpful and is dangerous. A nearest-order match turns a mishearing into a confident wrong answer.

A safer pattern: look up the exact string first. If that fails, search only the candidate set the caller could plausibly own, such as their open orders by caller ID, and accept an edit distance of 1 only when exactly one candidate qualifies. Then read back the candidate and confirm it. Never write or release information on a fuzzy hit alone.

def match_open_orders(heard: str, open_ids: list[str]):
    close = [c for c in open_ids if edit_distance(heard, c) <= 1]
    return close[0] if len(close) == 1 else None   # None: ask again

Note the limit. In a test run, "48220" matched "48230" among the candidates "48213" and "48230," because only one was within one edit. The code returned a candidate, and the read-back must still catch it. The worked cases in our posts on pizza "where's my order" calls, WISMO calls and pharmacy refill Rx numbers show what a wrong lookup costs in each trade. For vehicle lookups, see dealership VIN lookups.

Voice agent read-back design: the last gate is the caller

Voice agent alphanumeric capture ends with a read-back, and it only works if the caller can compare. Three rules help.

Chunk it. Read in groups of three or four with a short pause between groups: "four eight two, one three." The caller can stop you at "two" and fix the first chunk. Long unbroken strings force a yes or no on 17 characters at once.

Use the phonetic alphabet on the confirmation pass. On the first capture, accept whatever the caller says. On read-back, say "B as in bravo." The FAA AIM tells pilots to use phonetic equivalents when conditions are poor or call signs sound alike, and the same logic holds on a noisy phone leg. This is design opinion, not a measured result.

Let the caller correct one chunk. "No, the second group" should work. If the only recovery is repeating the whole string, callers will say "yes" to get through. Whether callers accept wrong read-backs is something to measure in your own recordings, not assume.

How to test voice agent alphanumeric capture

Use a seeded corpus, score at two levels, and break the system on purpose with confusable pairs.

1. Define the code types you capture. List each one with its format: length, alphabet, checksum, where it is written. A VIN, a six-character confirmation code and a numeric order ID each need their own rows.

2. Build a seeded corpus. Generate strings with a fixed random seed so reruns are comparable. Cover each length. Include codes with repeated characters, leading zeros, and double or triple runs. Use the sample sizes above.

3. Add confusable stress sets. Build strings dense in the E-set, M/N, F/S and 0/O pairs. Include near-duplicate pairs that differ by one character, so a fuzzy match has something to get wrong.

4. Vary the callers. Run each string across accents, speech pace, interruption behaviour, noise and line quality. Evalgent's caller profiles cover accent, speech pace, interruption level, response delay, behaviour, background noise and language, and its simulator reaches the agent over PSTN, SIP or WebRTC. That lets the same corpus run on the path real callers use.

5. Score two levels. Per-character accuracy tells you which pairs hurt. Whole-string exact match tells you what the record sees. Also record the silent wrong rate: strings written that do not match what the caller said.

6. Judge from the audio. A transcript can look right while the audio shows a clipped final digit. Evalgent evaluates from the audio, not only the transcript, and custom metrics can run on every call, such as "stored code equals spoken code."

7. Check the write, not just the call. Compare what the agent said, what it stored and what the downstream record holds. The post-call write-back testing guide covers that check, and silent tool failures covers failures that never raise an error.

8. Replay production. Evalgent's production monitoring takes call recordings, transcribes them, maps them to scenarios and judges them, so new failures become new corpus entries.

A copyable scenario list

The schema below is illustrative. Adapt the field names to your harness.

scenarios:
  - id: vin_clean_chunked
    code_type: vin
    seed: 3201
    spoken_as: "one G four A H five nine H four five G one one eight three four one"
    expect_stored: "1G4AH59H45G118341"
    expect_gate: ok
    caller: {accent: us_general, pace: normal, noise: none}

  - id: vin_gp_swap
    code_type: vin
    spoken_as: "one P four A H five nine H four five G one one eight three four one"
    expect_stored: "1P4AH59H45G118341"
    expect_gate: ok            # checksum cannot see G/P
    expect_readback: phonetic_confirm_required

  - id: order_double_run
    code_type: order_id
    spoken_as: "double five tree niner"
    expect_stored: "5539"

  - id: order_near_miss
    code_type: order_id
    spoken_as: "four eight two two zero"
    open_orders: ["48213", "48230"]
    expect_stored: null        # must ask again, not write 48230
    expect_readback: required

  - id: code_o_vs_zero
    code_type: confirmation_code
    spoken_as: "K seven B oh Q two"
    caller: {accent: south_asian_english, noise: street, pace: fast}
    expect_stored: "K7B0Q2"
    pass_if: whole_string_match

A scorecard to start from

For voice agent alphanumeric capture, these thresholds are starting points from our own judgement, not published standards. Set yours from the cost of a wrong write. The voice agent metric thresholds post shows how to tie numbers to consequences.

Re-run the corpus on every prompt, STT or TTS change, since each can move a letter pair. The tool-call accuracy and tool-calling test cases posts show where this fits in a regression suite. For live calls, see monitoring voice agents in production.

Frequently asked questions

Why does my voice agent get letters and numbers wrong?

Codes have no linguistic context, so the language model cannot guess a missing character. Confusable letters such as B, D and P, noisy phone audio and spoken forms like "oh" or "double five" add errors at each stage. Errors compound across the string, so a small per-character miss rate becomes a large whole-string failure.

How accurate does voice agent alphanumeric capture need to be?

For voice agent alphanumeric capture, it depends on what a wrong code breaks. Per-character accuracy of 99% still means about 16 in 100 wrong 17-character strings, by the compounding formula. Set a whole-string target per code type from the cost of a bad write, then add gates and read-backs to cover the gap.

How do you test a voice agent on VINs and confirmation codes?

Build a seeded corpus per code type, run it across caller profiles and noise, and score whole-string exact match plus silent wrong rate. Add confusable-pair stress strings and near-duplicate records. Then compare the stored value in the downstream system to what the caller said, not just the transcript.

Should the agent use the NATO phonetic alphabet?

Use it on the read-back, not as a demand on the caller. Most callers will not know it. The FAA AIM prescribes it in poor conditions and for similar-sounding call signs, and the logic carries over. Accept natural spelling on capture, then confirm ambiguous letters phonetically, such as "B as in bravo."

Does a VIN check digit prevent wrong VINs?

It catches many errors, but not all. Under 49 CFR 565.15, letters with equal transliteration values, such as G and P, can swap without changing the result. A random wrong VIN also passes about one time in eleven. Treat the check digit as a filter and keep the caller read-back.

Why do I and O cause problems in codes?

They look like 1 and 0, and callers say "oh" for zero. VINs exclude the letters I, O and Q under 49 CFR 565.13, so a gate can safely repair them to digits there. For order numbers or confirmation codes that allow those letters, the same swap would corrupt valid strings. Repair only where the alphabet forbids the letter.

Is fuzzy matching safe for order numbers?

Only inside a small candidate set, such as the caller's open orders. Accept a match within one edit only when exactly one candidate fits, then read it back for confirmation. Never write or disclose account data on a fuzzy hit alone, because a near neighbour is a different customer's record.

Should callers key in codes with the keypad instead?

For long strings, offering touch-tone entry is a sound fallback, since DTMF tones do not suffer letter confusion, but they carry digits only unless you design a letter scheme. Our guide to testing DTMF navigation covers the path. Test the switch itself, including callers who start speaking, give up and press keys mid-capture.

The bottom line

Voice agent alphanumeric capture fails quietly because errors compound across every character and the record looks valid. Test whole strings, stress the confusable pairs, and let nothing write until format, checksum and a chunked caller read-back all agree.