AI Receptionist for Auto Repair Shops: Is It Accurate?

An AI receptionist can sound perfect and still be wrong. The caller says "my F-150 grinds when I brake." The agent says "got it, we'll see you at nine." Then the shop system holds an F-250, a generic "brake inspection" job and a two-hour slot. The call was polite. The record is wrong. Nobody finds out until the tech pulls the wrong pads.

This post is an audit, not a tutorial. Auto repair AI receptionist accuracy is measurable, and it gives you the 12 accuracy checks that matter for repair-shop calls, a scoring rubric with sample-size math, and a one-week plan to run it on your live calls. It is written for teams that built and run the agent themselves, often inside shop management software used by many independent shops. Each shop has its own hours, bays, labor rates and rules. That is why one average accuracy number hides the failures.

The build guide lives in AI Receptionist for Auto Repair Shops: Step-by-Step QA Guide. This post is narrower. It tells you whether your agent is accurate today, and how to prove it.

What accuracy means on a shop call

A repair-shop call is the front door to a workflow. The caller speaks. The agent captures fields. The agent then writes to the shop system: a customer, a vehicle, an appointment, sometimes a repair order note. The service advisor reads that record the next morning and trusts it.

So accuracy has three layers. Each can fail on its own.

1. Heard. Did speech recognition get the year, model, VIN, name and callback number right?

2. Decided. Did the agent map the symptom to the right job, pick a safe slot, and say only what it may say about price?

3. Written. Did the right values land in the right fields on the right customer record, once?

A transcript audit sees only layer one. A record audit sees layer three but not why it broke. The audit below checks all three, from call audio to system record. For a general method, see Voice Agent Accuracy Audit: What It Measures.

The 12 accuracy checks

Here is the scorecard. Each check has a source of truth you can compare against. If you have no source of truth for a check, that is itself a finding.

The tier decides how you treat a miss. Gate checks have zero tolerance: one confirmed failure blocks the next release. Core checks have a pass-rate target. Watch checks you track for drift.

Vehicle identity: checks 1 and 2

The shop system keys everything off the vehicle. A wrong year changes the parts catalog. A wrong engine changes labor time. Callers say "twenty nineteen" or just "nineteen." They say "Sierra" and the agent hears "Sienna." The audit compares the captured year, make, model and engine to what the caller said, then to the VIN decode if one exists.

VINs are checkable by rule. Under 49 CFR 565.13, a VIN has 17 characters. The character set leaves out I, O and Q. Position 9 is a check digit. A captured VIN that fails the format or the check digit was misheard or mis-typed. NHTSA's free vPIC decode API returns the make, model and year for a VIN, so you can compare. Dealership flows go deeper in Dealership Service AI Scheduling: How to QA VIN Lookups. Spoken entity errors are covered in Evaluating STT for Names and Entities.

Job, safety and slot: checks 3, 4 and 5

Callers describe symptoms, not jobs. "It shakes at 60" can be tires, alignment or a driveshaft. The agent should map the symptom to a job class the shop actually sells. It should not invent a diagnosis.

Safety is the gate that matters most. "Brakes went to the floor" is not a normal booking. "Smells like gas" is not either. The agent needs a written list of phrases that mean stop driving or call a tow. The audit replays those phrases and checks the agent said so, plainly, and routed the call.

Slot accuracy is capacity accuracy, and ai appointment scheduling auto repair shops rely on is only as good as its duration table. A calendar slot is not a bay with the right lift and a qualified tech. Auto Repair AI Booking Errors: Test Before Bays Overbook covers duration errors and overbooking with worked math.

Money and consent: checks 6 and 7

Two things go wrong on price, including the diagnostic fee. The agent quotes a firm number it was never allowed to give, or it gives a ballpark that the caller hears as a promise. Both are gate failures when the shop's policy says estimates come from an advisor. The pricing mechanics live in How Accurate Are AI Repair Estimates Taken by Phone?.

Authorization is a legal topic as well as a quality one. California's Bureau of Automotive Repair says an estimate is required before repairs. It also says the customer's authorization must be obtained and recorded on the estimate before work begins, and that extra work needs a revised authorization. See the BAR's Write It Right guide. Other states differ. An AI receptionist should never imply that a booking is an approval for work. This is not legal advice.

Customer match, status and policy: checks 8, 9 and 10

One household often has three cars and two drivers. The agent finds "the customer" by phone number, then attaches the wrong car or books under the spouse. The audit checks that the customer, the vehicle and the contact on the record match the caller.

Status calls ("is my car ready?") read from live data. The risk is an invented answer when data is missing, such as a confident "this afternoon" when the repair order has no ETA. See Is My Car Ready? Testing AI Status Calls for Shops.

Policy answers come from per-shop config. Hours on a holiday, whether the shop works on diesels, whether a shuttle exists. These drift when a shop edits its settings and the agent's context does not follow. One Voice Agent, 500 Customer Setups explains how to test that.

Handoff and write-back: checks 11 and 12

When the agent transfers or takes a message, the callback number and the reason must survive. Digits are the weak spot. Escalation Accuracy: Measuring Good Handoffs covers the measurement.

Write-back is the check that catches everything the others missed. Take the call, list every field it should have set, and diff against the record. A common silent failure is a duplicate customer: the agent created a new profile instead of matching the old one, so the history is split.

Why errors start at the audio layer

Phone audio is thin. Twilio's Media Streams deliver audio as mono mu-law at 8,000 Hz. That is telephone bandwidth, with no room for the high-frequency detail that separates "F" from "S" or "M" from "N." Add a shop floor with an impact wrench, and the model hears less still.

Speech recognition also does not fail evenly. The 2020 PNAS study by Koenecke and colleagues tested five commercial systems. Stanford's summary reports average word error rates of 35% for Black speakers against 19% for white speakers. That study used interviews, not phone calls. For a shop, the lesson is that your error rate is a spread across caller groups, not one number. Measure by accent and noise. See Accent Robustness for Voice Agents and How to Test STT Under Background Noise.

Agents are also inconsistent. The tau-bench paper (Yao et al., arXiv 2406.12045) proposes pass^k, the chance an agent succeeds on all k repeated trials of the same task. Its abstract reports that even a top agent at the time succeeded on under 50% of tasks, and pass^8 fell below 25% in the retail domain. The takeaway for your audit: one good run proves little. A booking flow that works once may fail on the third try with the same words. Replay important cases more than once.

This is also why transcript-only scoring misleads. A transcript can read cleanly while the audio shows the caller said "four-thirty" and the agent heard "fourteen thirty." Evalgent's approach is to judge from the audio, not only the transcript, because many real failures sit between what was said and what was written.

The scoring rubric

This is the part that turns auto repair AI receptionist accuracy into numbers you can compare week to week.

Score every sampled call on every check it exercises. Mark each check pass, fail or not applicable. Do not average across checks. Report per check.

Each check then gets a color from its error interval, not its raw rate.

Targets are yours to set. A starting point, which is an assumption and not a standard: gate checks target 1% error, core checks 5%, watch checks 10%.

Gate checks add one more rule. A single confirmed gate failure with real customer impact is a release blocker, whatever the interval says. A failed safety triage is not a statistic.

Worked math: how many calls is enough

The interval is the point. It is a confidence interval, not a verdict. If you review 60 calls and find zero errors on a check, you have not shown the error rate is zero. The 95% Wilson upper bound is about 6.0%. Review 30 calls and it is 11.4%. This is an illustrative calculation using the standard Wilson score interval.

Read the last row twice. Five errors in 60 calls is an 8% raw rate, yet the interval runs from 3.6% to 18.1%. If your core target is 5%, that check is amber, not red. You need more calls, or you need to fix the likely cause and re-sample.

Most checks do not apply to every call. A status call never tests the VIN. If 25 of your 100 calls are status calls, check 9 has 25 observations, not 100. Plan the sample so each check gets at least 30 observations, or accept a wide interval.

Here is the scoring helper. It is simplified and runs as written.

from math import sqrt

def wilson(k, n, z=1.96):
    """95% Wilson interval for k errors in n observations."""
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, c - h), min(1.0, c + h)

def color(k, n, target):
    lo, hi = wilson(k, n)
    if lo > target:
        return "red"
    if hi < target:
        return "green"
    return "amber"

print(color(5, 60, 0.05))   # amber: need more calls
print(color(0, 200, 0.05))  # green

You can also validate captured VINs offline, before any decode call. This follows the transliteration and weight tables in 49 CFR 565.15.

TRANS = {**{str(i): i for i in range(10)},
         **dict(zip("ABCDEFGH", range(1, 9))),
         **dict(zip("JKLMN", range(1, 6))),
         "P": 7, "R": 9, **dict(zip("STUVWXYZ", range(2, 10)))}
WEIGHTS = [8, 7, 6, 5, 4, 3, 2, 10, 0, 9, 8, 7, 6, 5, 4, 3, 2]

def vin_ok(vin):
    vin = vin.upper()
    if len(vin) != 17 or any(c not in TRANS for c in vin):
        return False  # wrong length, or I, O, Q, or a symbol
    total = sum(TRANS[c] * w for c, w in zip(vin, WEIGHTS))
    check = "X" if total % 11 == 10 else str(total % 11)
    return vin[8] == check

A VIN that fails `vin_ok` is a capture error. A VIN that passes can still be the wrong car, so also decode it and compare the model year and model to what the caller said. Older vehicles and some non-North-American VINs may not follow the check-digit rule.

How to run a one-week accuracy audit on your live calls

This plan assumes you record calls and can export the shop system's records. The time figures are planning assumptions, not benchmarks. Adjust them to your team.

1. Define truth for each check (Monday). For each of the 12 checks, write down the source of truth from the table above. Write the target error rate and tier. If a check has no source of truth, add it to a gap list.

2. Pull and stratify the sample (Monday). Export last week's calls with their system records. Group them by intent: new booking, estimate question, status, reschedule, policy question, transfer, after-hours. Aim for at least 60 calls, with every intent represented.

3. Add the hard cases (Tuesday). Include calls with noise, accents, interruptions and callers who give a VIN or plate. Hard cases are where error clusters. Tag them so you can report them separately.

4. Label from the audio (Tuesday and Wednesday). A person listens to each recording, not just the transcript. They write down what the caller said for each field and mark each check pass, fail or N/A. Budget about eight minutes per call (a planning figure). Have a second person re-label 10 calls.

5. Diff against the record (Wednesday). For each call, compare the labelled truth to the system fields. This is check 12. Record the first point of failure: heard, decided or written.

6. Score and color (Thursday). Count errors per check. Compute the interval. Assign green, amber or red. Amber means "unknown," and that is useful.

7. Turn failures into tests (Friday). Every confirmed failure becomes a replayable scenario: the caller's words, the accent, the noise, the expected record. Rerun each at least three times, per the pass^k lesson above.

8. Set the next audit trigger (Friday). Re-run the scorecard after any prompt, model, speech-recognition or shop-config change. A weekly 30-call spot check keeps drift visible.

The labelling step is slow when done by hand across many shops. This is where tooling helps with an ai phone agent auto repair teams run across many shops. Evalgent runs a call simulator over phone, SIP or WebRTC with caller profiles for accent, pace, interruption and background noise. It generates scenarios from your agent's prompt and flows, and it judges from the audio with evidence and human review of verdicts. For live traffic, you push call recordings. They are transcribed, mapped to scenarios and judged, and it flags tool-call failures and policy failures with fix recommendations. Evalgent works with teams to set up evals for a first agent; it is sales-led, not self-serve.

Reading the results: three patterns

After a first audit, failures usually fall into one of three patterns.

Heard wrong, written wrong. Speech recognition produced the wrong entity and the agent trusted it. The fix is a confirmation step for high-stakes fields, plus rule-based validation (VIN check digit, year range, model in make). More prompt text rarely helps.

Heard right, decided wrong. The transcript is correct but the job, slot or statement is wrong. This points to the prompt, the job taxonomy or the shop config. Test the config first. A stale duration table looks like an AI error.

Decided right, written wrong. The agent said the right thing, but the record disagrees. This is a tool-call or integration bug: wrong field, wrong customer, a duplicate record, a timezone. Evaluating Tool-Argument Accuracy from Speech covers how to test the arguments.

Sorting failures this way shows who owns the fix: speech, prompt, config or integration.

Where to go deeper

Use this map after your first scorecard. Each neighbour post goes deeper than this audit does.

Frequently asked questions

How accurate is an AI receptionist for an auto repair shop?

There is no single number. Accuracy varies by check, shop and caller. Vehicle identity and write-back usually decide trust. Measure each of the 12 checks against the shop record, then report a range. With 60 scored calls and zero errors, the true error rate could still be about 6%.

What errors does an AI receptionist make most often at auto shops?

Expect wrong vehicle details, a symptom booked as the wrong job, a duration that overfills a bay, and a firm price quoted when policy forbids it. The quiet ones are duplicate customer records and wrong callback numbers. Your audit shows which dominate.

How do I audit my shop's AI phone agent?

Pick a source of truth for each of the 12 checks. Sample at least 60 calls across intents. Label each from the audio, diff against the shop record, and score each check with an error interval. Turn each confirmed failure into a replayable test. Repeat after every prompt, model or config change.

How many calls do I need to review to audit an AI receptionist?

At least 60 calls overall, and at least 30 observations per check you care about. Fewer than that leaves wide intervals. Zero errors in 30 observations still allows a true rate near 11%. Sample by intent so each check actually appears, and over-sample hard cases such as noise and accents.

Can an AI receptionist give repair quotes over the phone?

It can give ballparks if your policy allows. It should not give firm quotes or imply approval for work. Rules differ by state; California's BAR requires an estimate and recorded authorization before repairs begin. Treat price statements as a gate check, and have an advisor confirm firm numbers.

Does AI mishear VINs and license plates?

Yes, especially on 8 kHz phone audio with noise. Letters like B, D and P blur, and I, O and Q never appear in a VIN. Validate every captured VIN against the 17-character format and check digit. Then decode it and compare year and model to what the caller said.

How do I know if the AI booked the wrong vehicle?

Diff the call against the record. Compare the year, make, model and VIN the caller gave to the vehicle attached to the appointment. Check the customer too, since a household may own several cars. Flag any mismatch, and read the first failure point: heard, decided or written.

How often should I re-audit an AI phone agent after a change?

Re-run the full scorecard after any change to the prompt, model, speech recognition or a shop's configuration. Between changes, spot-check about 30 calls weekly. A change ships to every shop at once, so replay saved failure scenarios before release.

The bottom line

Auto repair AI receptionist accuracy is a set of per-check error rates measured against the shop record, with honest intervals. Run the 12-check audit on 60 or more live calls this week, and turn every failure into a test that must pass before the next release.