Why Your HVAC AI Books the Wrong Tech (and How to Test It)

Last verified October 16, 2026, against EPA's Section 608 and Technology Transitions pages and the vendor pages linked below.

The tech shows up. The customer has a heat pump. The tech handles gas furnaces only. The agent booked a job, the API returned 200, and the dashboard counted a win.

This post is for product teams at field service management software companies. Your AI books jobs for hundreds of HVAC contractors, and each one runs its own dispatch rules. You will see how HVAC AI scheduling errors start in the audio, pass through the booking payload and come out as a wasted truck roll. Then you get a test matrix, a sample-size rule and a production monitor you can build this sprint.

How an AI booking actually picks a technician

An AI receptionist for HVAC does not pick a technician. It fills fields, and the contractor's dispatch rules pick the technician from those fields. That split is why one wrong field is so expensive.

Public product pages describe this design. ServiceTitan says its AI Virtual Agent books jobs "according to job type, location, required skill sets, and more," straight to the dispatch board. Workiz says its Genius Answering offers only the job types a contractor configures. It can also refuse bookings outside the contractor's service areas. Neither page says anything about error rates, and this post makes no claim about either product's accuracy.

A typical HVAC booking runs in five steps:

1. Capture. Speech-to-text turns 8 kHz phone audio into text. The LLM pulls out symptoms, equipment, address and urgency.

2. Classify. The LLM picks one value from the contractor's job type list. This list is often injected into the tool schema as an enum.

3. Book. A `create_job` tool call writes job type, equipment, address, priority and customer ID.

4. Dispatch. The FSM engine maps job type to required skill tags. It filters techs by service zone, capacity and arrival window.

5. Confirm. The agent reads back a window, and the call ends as "booked."

People often call step 4 AI dispatch for HVAC. In most setups it is rule-based, and it does exactly what the data says. If step 2 wrote "Furnace repair," the engine finds a gas-heat tech. It will not second-guess the label.

Wrong-tech booking: a job whose captured fields are internally valid but describe the wrong work, so correct dispatch rules assign a technician who lacks the skill, certification, parts or zone for the real job.

The failure chain, from one lost word to a second visit

A word drops, a slot stays empty, and the model fills it with a prior.

Walk through the example in the diagram. It is illustrative, but every link is a known mechanism.

The caller says "heat pump" while wind hits the phone outside. She talks over the agent's question. Short, unstressed words are the first to go when speech overlaps or the line is noisy. The transcript reads "it's the heat, the outside unit." The LLM now sees "no heat" in January and no equipment type. Its prior says gas furnace. It writes "Furnace repair" and "Gas furnace." Nothing in the payload looks wrong to a schema check.

Research backs the two weak links here. The SLURP benchmark (Bastianelli et al., EMNLP 2020) has about 72,000 recordings across 18 scenarios. Its baseline action accuracy fell from 86.99% on gold text to 79.70% on ASR output. Entity F1 fell from 78.19 to 62.69. Slots degrade faster than intents. Equipment type, ZIP and street names are slots.

Caller population matters too. Koenecke et al. (PNAS, 2020) tested five commercial ASR systems. Average word error rate was 0.35 for Black speakers and 0.19 for white speakers. An Interspeech 2008 study by Vipperla et al. measured 47.8% WER on elderly voices versus 36.4% for adults. That is old ASR, so ignore the absolute numbers. The direction still holds a lesson. The callers you most need to get right on a no-heat night are often the ones your recognizer handles worst.

For more on how entity errors survive into tool calls, see our guide to tool argument accuracy and STT entity accuracy.

Five fields that pick the tech

Every wrong technician dispatched traces back to one of five fields. Use this as a failure taxonomy when you label calls.

Job type is the field that matters most

HVAC job types carry very different skill, time and revenue profiles. A no-cool repair, a maintenance tune-up, an install estimate and a commercial rooftop unit call are not neighbours. A tune-up might be a short slot for an apprentice. An install estimate often goes to a comfort advisor, not a tech. A rooftop unit needs commercial experience and maybe a lift.

Callers blur these. "Can someone come look at my AC before summer, it's making a noise" is part tune-up, part repair. "How much for a new system" can be an install estimate or a dying unit that needs a repair today. HVAC job type classification fails at these boundaries, not on clean requests.

Equipment and refrigerant now carry more weight

Equipment type decides skill tags and truck stock. In 2026 the refrigerant matters more than it did a few years ago.

EPA's Technology Transitions rule sets a 700 GWP limit for new residential and light commercial AC and heat pump systems, from January 1, 2025. Carrier describes R-454B as an A2L refrigerant with mild flammability and a GWP of 466. Carrier also says R-410A systems cannot be converted to R-454B. In May 2026, EPA removed the January 1, 2026 installation deadline for pre-2025 inventory above the limit. That inventory can now be installed until supply runs out.

The practical result: as of October 2026, a contractor's customer base mixes older R-410A systems, newly installed pre-2025 R-410A inventory, and new R-454B systems. Many contractors tag techs and trucks by refrigerant. A caller who says "it's the new one we put in last spring" is giving you dispatch data. The agent has to keep it.

Certification is the other layer. EPA's Section 608 rules require certification for technicians who service equipment that could release refrigerant. Type II covers high-pressure appliances, which includes typical residential split systems. Universal covers all types. Contractors encode this as skill tags. The AI never checks a certificate. It only picks the job type that pulls the right tag. This is not legal advice; contractors own their compliance.

Why this stays invisible on your dashboards

HVAC AI scheduling errors pass every check most teams run. The tool call succeeds. The caller hears a window and says thanks. Containment, booking rate and call duration all look healthy.

Transcript review misses it too. A reviewer reading "it's the heat, the outside unit is iced up" might not notice the dropped word. The transcript is self-consistent. The evidence is in the audio, where "pump" was spoken and lost. This is the core argument behind how Evalgent judges calls: the audio, not only the transcript, is graded, because a transcript-only referee is deaf to exactly this failure.

The signal shows up days later in another system. The tech closes the job with a different job code. Or the dispatcher reassigns it, or a second visit appears on the same address. None of those events link back to the call unless you build the join.

The multi-contractor problem

In HVAC AI scheduling you don't run one agent. You run one agent against hundreds of dispatch configs.

Each contractor names job types differently. One has "AC Repair - No Cool." Another has "Cooling Diagnostic" and a separate "Heat Pump Service." A third does commercial work only. Some split zones by county, some by ZIP, some by drive time. Membership tiers differ too.

A prompt change that fixes heat-pump detection for one contractor can break another. Suppose your fix adds "if the caller mentions an outdoor unit, prefer heat pump job types." A heat-pump-only contractor has no gas job types at all. There, the fix now pushes AC no-cool calls into heat mode job types. You will not see this on the demo contractor you test by hand.

The matrix above is the unit of testing. Rows are caller scenarios. Columns are real dispatch configs, cloned from production contractors. A release gates on every column, not on an average.

How to test HVAC AI scheduling before a release

These steps turn the five-field taxonomy into repeatable HVAC voice agent testing. They assume you can clone contractor configs into a staging tenant.

1. Cluster your contractors by dispatch config. Group them by job type list shape, equipment mix (gas plus heat pump, heat pump only, commercial), zone model and membership rules. Most platforms end up with 6 to 15 clusters. Pick one real contractor config per cluster.

2. Write golden-path scenarios per job type. One clean caller per job type in each config. "My AC is blowing warm air." "I'd like my spring tune-up." "I want a quote on a new system." These must pass at 100%.

3. Add edge callers at the boundaries. Caller unsure of equipment. Tune-up that is really a repair. Install question that is really a breakdown. ZIP on a zone border. Commercial RTU calling the residential line. Member asking for priority.

4. Run each scenario through caller profiles. Vary accent, speech pace, interruptions, background noise and age. Wind, a TV or a running furnace blower are realistic noise. See our guide to background noise testing.

5. Assert on the payload, not the transcript. For each call, check job type, equipment, ZIP, zone, priority and member flag against the expected values. Then replay the payload through the contractor's dispatch rules in staging. Assert that the assigned tech holds the required skill tags and serves that zone.

6. Test the "unknown" path. When the caller can't say furnace or heat pump, the right outcome is a clarifying question, then an "HVAC diagnostic" job type or a dispatcher review flag. A confident guess is a fail.

7. Gate the release per column. Block the change if any config cluster drops on any golden-path scenario, or on more than one edge row. Then roll out by cluster, as in our guide to shipping prompt changes safely.

A test case can be this small. This is simplified YAML; adapt field names to your schema.

# Simplified scenario: unsure no-heat caller, heat-pump-only contractor
id: hs03-noheat-unsure-hp-only
contractor_config: cluster_b_hp_only
caller_profile:
  age_band: 75_plus
  pace: slow
  interruptions: high
  noise: outdoor_wind
script:
  - "The heat's out. The thing outside is covered in ice."
  - (if asked about system type) "I don't know, my son set it up."
expect:
  agent_asks_clarifying_question: true
  payload:
    job_type_in: ["Heat Pump Repair", "HVAC Diagnostic"]
    priority: urgent
  dispatch_replay:
    assigned_tech_has_skills: ["heat_pump"]
    assigned_tech_serves_zip: true

Evalgent can generate scenarios like this from your agent's prompt and flows, including branches and policy constraints your manual tests skip. It runs them over phone, SIP or WebRTC with caller profiles for accent, pace, interruptions and noise. For the general scheduling test plan behind this, see voice agent testing for appointment scheduling.

How many test calls per contractor cluster

Use the rule of three. If you run n calls and see zero failures, the 95% upper bound on the true failure rate is about 3/n.

• 20 clean runs: the failure rate could still be 15%.
• 60 clean runs: about 5%.
• 100 clean runs: about 3%.

So 20 calls per cell catches gross breaks. It cannot prove a cell is safe. Put 60 or more runs on the cells that cost the most when wrong, such as no-heat priority and heat pump versus furnace. Fewer runs are fine for low-cost rows. Re-run failures several times, because LLM output varies run to run.

What a wrong truck roll costs

Contractors know their own cost per truck roll. You can build the model and let them plug it in.

Worked example with illustrative assumptions, not measured data:

• A contractor's AI books 1,200 jobs a month.
• 3% of those go to a tech who can't complete the real job.
• That is 36 extra truck rolls a month.
• Assume a fully loaded cost of $150 per roll for drive time, labour and vehicle.
• 36 × $150 = $5,400 a month for one contractor.

Across 250 contractors at the same rate, that is $1.35 million a month in field cost your customers absorb. The cost also lands in places this model ignores. Think lost first-time fix rate, a refund or a bad review, and a no-heat night for someone at risk. A small error rate times a big booking volume is a big bill.

Weight each misclassification by its cost. A tune-up booked as a repair wastes some time. A no-heat repair booked as a next-week tune-up is the expensive cell.

Monitoring job-type accuracy against closed job codes

Pre-release HVAC AI scheduling tests catch regressions you can imagine. Production tells you what you missed. The best ground truth you already own is the job code the technician closes the job with.

Build the join: call ID, booked job type, booked equipment, then the final closed job code and any reassignment. Map each contractor's closed codes back to job families. Then build a confusion matrix per contractor.

In this illustrative month, overall accuracy is 91.7%. That number hides two problems. Fourteen "tune-ups" were really no-cool repairs, which is a wrong-tech risk in peak season. Install estimates sit at 85%.

A simplified version of the monitor in Python:

# Simplified: booked job type vs tech-closed job code, per contractor.
import math
from collections import defaultdict

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (c - h, c + h)

def job_type_report(jobs, code_map, row_floor=0.90, cell_cap=0.04):
    """jobs: dicts with contractor_id, booked_type, closed_code.
    code_map: {contractor_id: {closed_code: job_family}}."""
    m = defaultdict(lambda: defaultdict(int))
    for j in jobs:
        fam = code_map[j["contractor_id"]].get(j["closed_code"], "unmapped")
        m[(j["contractor_id"], j["booked_type"])][fam] += 1
    alerts = []
    for (cid, booked), cols in m.items():
        n = sum(cols.values())
        k = cols.get(booked, 0)
        lo, hi = wilson(k, n)
        if hi < row_floor:  # confidently below the floor
            alerts.append((cid, booked, "row_below_floor", k, n, round(lo, 3)))
        for fam, v in cols.items():
            if fam != booked and v / n > cell_cap:
                alerts.append((cid, booked, f"confused_with:{fam}", v, n, None))
    return alerts

Read the confidence interval before you alert. With 51 correct out of 60 install estimates, the 95% Wilson interval runs from about 74% to 92%. You can't yet say that row is below 90%. Raise a "watch" flag, not an incident. At 268 of 290 tune-ups, the interval is roughly 89% to 95%. That's tight enough to act on its off-diagonal cell.

Three practical notes on the join:

• Map codes per contractor. Closed codes are free-form across contractors. Keep an explicit `code_map` and count "unmapped" as its own column.
• Track reassignment separately. A dispatcher who moves the job before the tech leaves saved the truck roll but not the labour. Count it as a near miss.
• Pull the audio for every off-diagonal call. The confusion matrix says where. The recording says why: lost word, missing question or bad prompt rule.

Evalgent's production monitoring fits this loop. Teams push call recordings, and each call is transcribed, mapped to a scenario and judged. It flags tool-call failures, policy failures and latency spikes with fix recommendations. Job-type accuracy can run as a custom metric on every call, not a sample. Feed the confirmed misses back into the pre-release matrix as new edge rows. For why full coverage beats sampling, see score every call, not a sample. For the wider metric set, see appointment scheduling voice agent metrics.

Priority, membership and capacity: the quieter wrong-tech errors

Not every wrong booking sends the wrong skill. Some send the right tech at the wrong time.

No heat with a vulnerable occupant. Contractors often set their own rules for no-heat calls with elderly people, infants or medical needs at home. The agent must catch the cue ("my mom is 84," "my daughter has asthma") and apply the contractor's priority. Test it with the cue spoken once, early, with an interruption over it.

Membership. Customers on a maintenance agreement are often promised priority windows. If the agent books under a new customer record because the phone number didn't match, the member loses that slot. Test with a caller on a different phone than the one on file.

Capacity. When same-day capacity is full, the agent should not invent a window. It should offer the next real slot or route to the on-call path the contractor defines. Assert the confirmed window against the dispatch board after the call.

These checks sit alongside the standard go-live checks in our per-customer go-live checklist. The same pattern of one captured identifier driving the whole booking shows up in dealership service scheduling by VIN and in lawn and cleaning quote accuracy. For the full home services QA program, start with the AI receptionist for home services QA guide.

Frequently asked questions

Why does my AI book the wrong technician?

An AI booking agent books the wrong technician when it writes a wrong job type, equipment type, ZIP or priority. The contractor's dispatch rules then correctly match skills and zone to that wrong data. The tool call succeeds, so the call counts as booked. The error only appears when the tech arrives or closes the job under a different code.

How do I test AI job type classification before a release?

To test AI job type classification, clone one real dispatch config per contractor cluster. Run golden-path and boundary scenarios through varied caller profiles. Assert on the booking payload, then replay it through the dispatch rules and check the assigned tech's skills and zone. Block the release if any config cluster regresses.

Can an AI receptionist tell a heat pump from a furnace?

An AI receptionist can tell a heat pump from a furnace only if the caller says so clearly or the agent asks. Many callers don't know their system type. The safe design asks one discriminating question, such as whether the outdoor unit runs when the heat is on. If still unclear, it books a diagnostic job type rather than guessing.

How should an AI prioritize no heat calls?

An AI should prioritize no heat calls by the contractor's own rules, which often raise priority for elderly occupants, infants or medical needs. The agent must catch the cue even when spoken once, then apply the priority flag. Test this with interruptions and noise, and verify the booked window against same-day capacity.

What does a wrong truck roll cost?

The cost of a wrong truck roll depends on each contractor's drive time, labour and vehicle costs. As an illustrative model, 36 wasted rolls a month at an assumed $150 each is $5,400 for one contractor. Add lost first-time fix rate, refunds and reviews. Build the model and let each contractor enter its real cost.

How do you monitor AI booking accuracy in production?

To monitor AI booking accuracy in production, join each AI-booked job to the technician's closed job code and any reassignment. Build a confusion matrix per contractor and job type. Alert when a row's confidence interval sits below your floor, or when one wrong cell grows. Pull the call audio for every mismatch.

Does an HVAC tech need EPA 608 certification for heat pumps?

EPA Section 608 requires certification for technicians who service equipment that could release refrigerant. Type II covers high-pressure appliances such as typical residential split AC and heat pumps, and Universal covers all types. Supervised apprentices are exempt. Contractors usually encode this as skill tags, which the job type selects. This is not legal advice.

How many test calls per contractor do you need?

The number of test calls per contractor cluster follows the rule of three. Zero failures in n runs bounds the true failure rate near 3/n at 95% confidence. Twenty clean runs still allow 15% failure. Sixty allow about 5%. Spend the most runs on high-cost cells like no-heat priority and heat pump versus furnace.

The bottom line

HVAC AI scheduling books the wrong tech when one captured field is wrong and correct dispatch rules faithfully act on it. Test every contractor's dispatch config with edge callers before release, then measure booked job types against tech-closed job codes on every call.