Monitoring AI Service Calls in Dealership BDCs

A service status call sounds harmless. The caller asks, "Is my car ready?" The agent looks up a repair order, reads a status and gives a time. Nothing gets booked or charged. Yet that call creates a promise. The customer plans a ride or leaves work early. If the promise is wrong, the dealer hears about it at the service counter, not in your logs.

This post is for the product team at a dealer-software company. You run an AI BDC or a service department AI phone agent inside a DMS or service-communication platform. It answers dealership service AI calls for hundreds of rooftops. myKaarma, for example, says its ServiceConcierge "can answer calls, book or modify appointments, and route customers who need human assistance" (myKaarma). You already passed launch. Dealership BDC AI monitoring is the production question. What do you sample and score? How do you catch drift at one rooftop and report quality to dealers? Still pre-launch? Start with our guide to QA for VIN lookups in dealership service scheduling and the go-live checklist for each new customer.

Why the service status call is a silent failure machine

Repair order status calls are high volume and look simple. That is why teams under-monitor them. The call runs a short agentic workflow. Each step can fail without an error.

1. Identify. The caller gives a name, phone number, RO number or the last digits of a VIN.

2. Look up. The agent calls a DMS or middleware tool to find the open repair order.

3. Read back. It turns DMS status codes into speech: "waiting on parts," "in the wash," "ready."

4. Promise. It states a time, or it doesn't.

5. Write back. It logs a note on the RO, texts the advisor or opens a callback task.

Here is how a correct-looking call goes wrong. The caller says "four oh five one eight." Speech recognition returns "40581." That RO exists and belongs to another customer whose car is ready. The agent says, "Good news, your vehicle is ready for pickup." The tool call succeeded, the transcript looks clean and every dashboard is green. No tool-call failure was logged. The customer arrives to a car still on a lift.

Names fail the same way. "Nguyen" can land on "Win" or "When." A household phone lookup can return the spouse's truck, not the sedan in for service. Our post on evaluating STT for names and entities covers why proper nouns and digit strings break first.

Research backs the pattern. On SpokenWOZ, 249 hours of spoken dialogue, the best state tracker reached only 25.65% joint goal accuracy (Si et al., 2023). Spoken slot values are hard to get exactly right. τ-bench tested agents that call tools while talking to simulated users. Top function-calling agents solved under 50% of tasks, and pass^8 fell below 25% in retail (Yao et al., 2024). A status agent across 300 rooftops runs that test thousands of times a day.

Stale DMS status: when the agent is right and the answer is wrong

The second silent failure has nothing to do with speech. The tech finished at 11:00, but the service advisor closed the RO at 2:00. For three hours the agent said "still being worked on." The reverse also happens: a 2:00 PM promised time stays in the DMS after parts were back-ordered.

Your agent read the record correctly, and the answer was still wrong. Read-back checks alone mark this call a pass. Complaint tracking alone blames your agent. You need both views, and a label for which one failed.

Status read-back fidelity: the share of status calls where the spoken status and time match the DMS record. Use the record captured at lookup time. It measures the agent, not the dealer's data hygiene.

What dealership BDC AI monitoring scores on every call

Score these on every call. They compare data you already hold: the tool log, the DMS snapshot and facts extracted from audio. Our post on scoring every call instead of a sample walks through the economics.

The thresholds are starting points, not benchmarks. Set your own from two weeks of baseline per rooftop.

Two metrics need care. RO lookup accuracy must be checked against the audio, not the transcript. If speech recognition misheard the digits, the transcript agrees with the wrong lookup, and a transcript judge passes it. Our piece on transcript vs audio evaluation shows other failures only audio exposes. Evalgent scores calls from the audio for this reason. It treats a misheard RO as a tool-argument failure, not a clean call.

Promised-time accuracy catches the most expensive error. A model filling a pause will say "should be done by end of day." The DMS may hold no promised time at all. Treat any time not grounded in the record as a hallucination. Our guide to measuring hallucination rate gives the labeling rules.

Warranty, recall and declined-work wording

NHTSA tells owners to "contact your local dealership to fix the recalled part for free" (NHTSA). Free does not mean parts are in stock. A safe agent says the recall repair is free and offers to book it. It never promises a same-day fix without a parts flag.

Warranty is riskier. The pay type on an RO line (customer, warranty, internal) decides who pays. "That should be covered" on a customer-pay line creates a dispute at the cashier. Score every coverage claim against the pay type.

Declined work is the upsell problem. Some rooftops want declined brake pads mentioned on status calls. Others forbid it. Store that policy per rooftop.

A minimal call checker (simplified)

This sketch compares facts extracted from one call with the DMS snapshot taken at lookup time. It tags each finding with an owner: your agent or the dealer's data.

from dataclasses import dataclass, field
from datetime import datetime, timedelta

@dataclass
class RoSnapshot:            # DMS state captured when the lookup tool ran
    ro_number: str
    status: str              # "WAITING_PARTS", "IN_PROGRESS", "READY"
    promised_at: datetime | None
    status_changed_at: datetime
    declined_ops: list = field(default_factory=list)

@dataclass
class AgentClaims:           # extracted from the call audio by your judge
    ro_caller_said: str      # from the audio, not the ASR transcript
    status_said: str
    promised_said: datetime | None
    mentioned_declined: bool

STALE = timedelta(hours=3)          # illustrative threshold
TIME_TOLERANCE = timedelta(minutes=30)

def check_status_call(snap, claims, call_at):
    out = []
    if claims.ro_caller_said != snap.ro_number:
        out.append(("looked_up_wrong_ro", "agent"))
    if claims.status_said != snap.status:
        out.append(("status_readback_wrong", "agent"))
    if claims.promised_said and snap.promised_at is None:
        out.append(("invented_promise_time", "agent"))
    elif claims.promised_said and abs(claims.promised_said - snap.promised_at) > TIME_TOLERANCE:
        out.append(("promise_time_mismatch", "agent"))
    if snap.promised_at and snap.promised_at < call_at and snap.status != "READY":
        out.append(("promise_already_missed", "dealer_data"))
    if snap.status == "IN_PROGRESS" and call_at - snap.status_changed_at > STALE:
        out.append(("status_possibly_stale", "dealer_data"))
    if snap.declined_ops and not claims.mentioned_declined:
        out.append(("declined_work_skipped", "agent"))
    return out

call_at = datetime(2026, 10, 16, 15, 10)
snap = RoSnapshot("40518", "IN_PROGRESS", datetime(2026, 10, 16, 14, 0),
                  datetime(2026, 10, 16, 9, 40), ["front brake pads"])
claims = AgentClaims("40518", "IN_PROGRESS", datetime(2026, 10, 16, 17, 0),
                     False)
for f in check_status_call(snap, claims, call_at):
    print(f)

Output:

('promise_time_mismatch', 'agent')
('promise_already_missed', 'dealer_data')
('status_possibly_stale', 'dealer_data')
('declined_work_skipped', 'agent')

The agent said 5:00 PM when the DMS said 2:00 PM. That one is yours. The 2:00 PM promise had already passed, and the status was five and a half hours old. Those two belong to the rooftop. Apply `declined_work_skipped` only where policy asks for reminders. Store the snapshot with the call, or you can't tell later what the agent saw.

Who owns a wrong promise: the attribution test

Every wrong promise needs an owner before anyone fixes it. Run four questions in order. Stop at the first "no."

1. Did the lookup arguments match what the caller said? Check the audio. A "no" is a speech or identity failure. Fix it with read-back confirmation of digits, VIN last-six checks or a second identifier.

2. Did the tool call return a valid record? A timeout, auth error or empty payload is a DMS integration failure. It often hits one DMS version or one rooftop's credentials.

3. Did the spoken answer match the snapshot? A "no" is reasoning or wording. Common causes: a status map missing a custom code, or the model guessing to fill a gap.

4. Was the snapshot itself current? If the status changed minutes after the call, the data was stale. The same holds if it sat untouched for hours.

The fourth bucket wins renewals. Dealers rarely see their own data hygiene. "38 of your 41 wrong promises came from ROs not updated after the tech clocked off" is useful to a service manager (illustrative numbers). It also protects your team from blame you don't own. Our post on detecting silent tool failures covers buckets one and two in depth.

What to sample for human review across hundreds of rooftops

Automatic checks cover structured facts. Tone and edge cases still need a person or a calibrated judge. Our guide to sampling live calls has the general method. For dealer fleets, stratify on three axes.

• Rooftop. Every rooftop gets a minimum sample, or small stores vanish behind big ones.
• Intent. Status, appointment change, recall question, loaner or shuttle request, and "talk to my advisor."
• Risk flag. Any call with an automatic finding, a transfer, a repeat caller within 24 hours or a long silence.

Now the math that sets the minimum. Say a rooftop's AI handles 40 status calls a week and two are wrong. That is a 5% error rate. The 95% Wilson interval runs from 1.4% to 16.5%. You can't tell a healthy store from a broken one. At 160 calls with eight errors, the interval narrows to 2.6% to 9.6%. Pool four weeks for small rooftops, or pool by DMS type.

Detecting a change needs more. Take the standard two-proportion formula at 80% power and 5% significance. A rise from 2% to 6% needs about 376 calls per period. A rise from 2% to 4% needs about 1,141. Small rooftops won't reach that in a week. Alert on fleet shifts weekly and on single rooftops monthly, unless a hard rule fires.

A workable start: review every flagged call plus 5 to 15 random calls per rooftop each week. Tune it to your reviewer hours.

Calibrate any LLM judge you use. Strong judges reached "over 80% agreement" with humans in MT-Bench, but showed position and verbosity biases (Zheng et al., 2023). See LLM-as-a-judge limits in voice AI.

Per-rooftop drift: why one store breaks while the fleet looks fine

Fleet averages hide rooftop failures. A 1% fleet error rate can mean 290 healthy stores and 10 broken ones. Track every metric per rooftop against its own baseline, then rank the outliers.

The causes repeat. A rooftop migrates DMS versions and status codes change overnight. A service manager adds a custom "QC" code your map can't phrase, so the model improvises. The store drops its shuttle or changes loaner rules, and the prompt still offers them. An OEM recall floods brand stores with recall calls. A new service advisor closes ROs late, which looks like agent drift.

Use a simple rule first. Flag a rooftop when a metric leaves its own four-week baseline. Require that it also sits in the fleet's worst decile. Two conditions cut noise from small samples. Our post on detecting metric drift covers control charts if you want more rigor.

Alerting on tool-call failures and latency

Some failures can't wait a week. A broken DMS connection turns every status call into "let me have your advisor call you back." Callers hear a polite answer. Meanwhile, the advisor's callback queue grows by dozens.

Alert on these in near real time. Thresholds are illustrative.

A status lookup often chains three calls: find the customer, list open ROs, fetch status lines. If the agent goes silent for four seconds, callers talk over it or hang up. Measure from the caller's last word to the first status word, from the audio. Our guide to monitoring voice agents in production covers dead air and latency spikes. Evalgent's production monitoring takes pushed call recordings, transcribes and judges them, and flags tool-call failures, latency spikes, PII leakage and policy failures with fix recommendations.

Weekly scorecards dealers will actually read

A service manager will read one page. The weekly AI call scorecard should cover volume, accuracy, escalation, "your data" findings and fixes shipped, as in the example above.

Show counts beside percentages. "2 invented times out of 212 calls" reads better than "99.1%." Link each count to its calls. Keep the "your data" section neutral. The voice agent KPI dashboard guide covers layout choices for internal dashboards.

Compliance checks the monitor should run

Dealer calls carry personal data and recordings. This is not legal advice; confirm your obligations with counsel.

FTC Safeguards Rule. The FTC's 2025 FAQ for dealers says the rule covers "most automobile dealers who finance or lease automobiles" (FTC). Service records alone are not "customer information," but they come in scope when combined with it. 16 CFR 314.4 requires dealers to oversee service providers "by contract" and to "monitor and log the activity of authorized users." A notification event involving 500 or more consumers must reach the FTC "no later than 30 days after discovery." Expect dealers to ask how your agent logs and protects data. Score PII disclosure on every call; see voice agent PII leakage detection.

Call recording consent. Federal law allows recording when "one of the parties to the communication has given prior consent" (18 U.S.C. 2511(2)(d)). Some states are stricter. California Penal Code 632 bars recording a confidential communication "without the consent of all parties" (Cal. Penal Code 632). Check that the recording disclosure plays on every call. Alert when an early barge-in cuts it off.

Outbound "your car is ready" calls. If your agent calls customers back, the FCC ruled in 2024 that TCPA limits "encompass current AI technologies that generate human voices" (FCC 24-17). Track consent status per number before any AI callback dials.

How to set up dealership BDC AI monitoring in production

1. Snapshot the DMS on every lookup. Store the RO, status code, promised time, pay types and declined ops, with a timestamp.

2. Keep dual-channel recordings. Separate caller and agent audio to check speech against tool arguments.

3. Extract claims from audio. Pull the RO or VIN spoken, status said, time promised and coverage wording.

4. Run automatic checks on every call. Tag each finding "agent" or "dealer data."

5. Wire real-time alerts. Start with lookup errors, not-found spikes, p95 latency and early disclosure.

6. Sample for human review. Stratify by rooftop, intent and risk flag.

7. Baseline each rooftop for two weeks. Then flag drift against its own baseline and the fleet.

8. Replay failures before you fix them. Turn each failure into a test with the same accent, pace and noise. Evalgent generates scenarios from your agent's prompt and flows and runs them with caller profiles over phone, SIP or WebRTC.

9. Send the weekly scorecard. One page per rooftop, with counts linked to calls.

Step 8 matters because a fix for one rooftop ships to all of them. See shipping prompt changes safely. For independent shops, see the AI receptionist QA guide for auto repair shops. For a similar fleet case, read monitoring AI check calls in trucking.

Frequently asked questions

How do you monitor AI calls in a dealership BDC?

Dealership BDC AI monitoring scores every AI service call against the DMS record the agent saw, and check the audio for what the caller actually said. Track RO lookup accuracy, status read-back fidelity, promised times, recall wording and escalations per rooftop. Alert in real time on lookup errors and latency. Send each dealer a weekly scorecard that separates agent errors from stale DMS data.

What should a dealership AI say when a car is not ready?

A dealership AI should read the current DMS status in plain words, give a promised time only if the DMS holds one, and offer an advisor callback when the time has passed. It should never guess "end of day." If parts are back-ordered, it should say so and log a note on the repair order for the advisor.

How accurate is AI at looking up repair orders?

AI repair-order lookup accuracy depends mostly on how it captures digits and names. A misheard RO can match another real RO, so the lookup succeeds and the answer is wrong. Measure it against the caller's audio, not the transcript. Confirming the number back, or asking for a second identifier like the VIN's last six, reduces wrong-record lookups.

How many AI service calls should you review each week?

Review every call with an automatic finding, plus a small random sample per rooftop. Small samples are noisy: two errors in 40 calls gives a 95% interval of 1.4% to 16.5%. Detecting a rise from 2% to 6% needs about 376 calls per period. Pool small rooftops over four weeks and review flagged calls first.

Does the FTC Safeguards Rule apply to dealership AI phone agents?

The FTC Safeguards Rule covers most dealers that finance or lease vehicles, and dealers must oversee service providers by contract under 16 CFR 314.4(f). A voice agent vendor that can access customer information is likely in that chain. Service records alone are not customer information, but combined data is. This is not legal advice; confirm with counsel.

Can a dealership AI give recall and warranty information?

A dealership AI can tell callers that recall repairs are free, which NHTSA states, and offer to book the repair. It should not promise same-day fixes without parts confirmation. Warranty answers should depend on the repair order's pay type. Saying "that's covered" on a customer-pay line creates a dispute at the cashier.

When should a service AI transfer to an advisor?

A service AI should transfer when a promised time has passed, when the customer disputes a charge or diagnosis, when coverage is unclear, or when the caller asks for a person. The transfer should carry the RO number and a short summary. Measure escalation accuracy and failed warm transfers per rooftop every week.

What should a weekly AI call scorecard for dealers include?

A weekly AI call scorecard for dealers should show calls handled, transfers, RO lookup accuracy, status read-back fidelity and invented promise times, each with counts. Add a neutral section on stale DMS data with example ROs, and a list of agent fixes shipped. Link every number to the calls behind it so the service manager can check.

The bottom line

A dealership AI that reads the wrong repair order or a stale status makes a confident promise that no dashboard flags. Dealership BDC AI monitoring catches it by comparing what the caller said, what the agent looked up and what the DMS held at that moment, per rooftop, every week.