Storm Follow-Up AI Calls: How to QA Them at Scale

A hailstorm gives a roofer a short window for storm lead follow-up. Thousands of homeowners need a call, and an outbound AI agent can place them in an afternoon. That same speed is the risk. One bad line in the script reaches every homeowner on the list before anyone listens to a single call.

This post is about roofing AI follow up calls QA for outbound storm follow-up calls, not inbound booking. For that, see testing roof inspection booking before storm season and why storm leads get lost at the front desk. Here the agent dials out after the storm. It must reach the right person, say true things, stay inside the rules and leave a clean record.

Why storm follow-up fails differently from inbound calls

An inbound caller picks the topic. An outbound agent picks it for a stranger who did not ask. Three things change.

First, the premise can be wrong. The agent opens with a claim about the caller's house. If it is false, everything after it is built on sand.

Second, consent is not implied. The caller never dialed you. Every call needs a lawful basis, and it differs by number type, voice type and state.

Third, volume hides errors. A manual tester checks ten calls. A storm campaign places ten thousand. A 1% defect is invisible in ten calls and a hundred angry homeowners in ten thousand.

The call is also the front door to a workflow: capture, book or decline, write to a system. A misheard answer or a false premise corrupts the record, and nobody sees it until a crew shows up at the wrong house. Our outbound voice agent testing guide covers the generic go-live plan. This post covers what is specific to storm roofing.

The failure classes in storm follow-up calls

Roofing outbound calls fail in five ways. Group defects before you sample. Each class needs a different check.

Most teams only test the last row, because only it shows up in the CRM. The first four live in the audio.

Wrong-address damage claims: the expensive failure

The most common opener in a storm campaign is some version of "hail came through your area on Tuesday." The risk is the next sentence: "and your roof may have damage."

The agent gets that from a list. Someone draws a storm polygon from a hail report, selects addresses inside it and loads them. Three things can go wrong between polygon and call.

1. The list is wider than the storm. Someone pads the polygon by a mile "to be safe." Addresses outside the swath get the same script.

2. The agent generalises. The prompt says "mention the hail size." The model states a size from its own memory, or from the last campaign.

3. The agent concludes. It tells the homeowner their roof is damaged. Nobody has been on the roof.

The third failure is the worst. Hail damage is not a function of the headline size alone. A 2025 study from the Insurance Institute for Business and Home Safety, reported by Insurance Journal, looked at sub-severe hail, meaning stones under 1 inch. Cumulative granule loss from those impacts exceeded the loss from a single 2-inch stone in the tests. Shingles that had weathered and taken sub-severe hits were about ten times more susceptible to later severe hail.

So the agent can't say "small hail is harmless." It also can't say "you have damage." The safe line is that a storm passed near the address and an inspector can check. The agent states the storm date and the report source, then offers an inspection. It never states a conclusion about the roof.

Gate it before launch. For every address, store the report date, the hail size and the distance to the nearest report. Pass those fields into the agent as variables. Then check two things on recordings:

• The size and date the agent said match the variables, not its own memory.
• Addresses beyond your cut-off distance never reached the dialer.

The lead list is the first thing a call review should check. A worked example, labelled illustrative: a list holds 10,000 addresses. A one-mile pad puts 4% outside the verified swath. That is 400 homeowners who hear a false premise. Fixing the list is cheaper than any call review.

Consent, calling hours and abandonment

This section covers the federal TCPA roofing rules your QA should encode. It is not legal advice. Ask counsel how they apply to your list sources and states.

Voice type. In February 2024 the FCC ruled that AI-generated human voices count as "artificial or prerecorded" under the TCPA, so such calls need the called party's prior express consent. Mayer Brown's summary cites FCC 24-17, adopted February 8, 2024. The FCC's own page states the same.

Telemarketing consent. Under 47 CFR 64.1200(a)(2), telemarketing or advertising calls to protected lines such as wireless numbers, made by autodialer or prerecorded voice, need prior express written consent. A storm-damage pitch is advertising for a service. A list bought from a data vendor is not, by itself, consent.

Hours. Section 64.1200(c)(1) bars telephone solicitations to residential subscribers before 8 a.m. or after 9 p.m., in the called party's local time. Storms cross time zones. Dial by the lead's zone.

Call abandonment. Section 64.1200(a)(7) caps abandoned telemarketing calls, answered live and dropped, at 3% over 30 days for one campaign. An agent that can't pick up fast enough counts against this. More on pacing below.

Do-not-call. Section 64.1200(c)(2) covers the national registry, and (d) requires an internal do-not-call list and a written policy. Requests must be honored within 10 business days. Test that the agent logs them at once.

Revocation. The FCC adopted new revocation rules on September 30, 2026, released October 1, 2026 as FCC 26-67. Per Kelley Drye and Nixon Peabody, it narrows "revoke-all" for informational calls and lets callers name exclusive opt-out methods. It takes effect 30 days after Federal Register publication, which had not happened as of October 7, 2026. Until then, treat "stop" in any wording as a stop.

State layers. Florida's telephone sales law, Fla. Stat. 501.059(8)(a), bars unsolicited automated or prerecorded sales calls without prior express written consent. Its private right of action starts at $500 per violation or actual damages, whichever is greater, and a court can treble that for willful violations. As of the 2025 statutes.

Turn each rule into a check that runs on audio, not on the dialer log.

State rules that shape what the agent says

Several states regulate how roofers talk about insurance after a storm. This is not a complete list. The two below are checked against official text.

Florida. Section 489.147 sets out prohibited property insurance practices. As of the 2025 statutes, a contractor may not:

• solicit with a "prohibited advertisement" that induces a consumer to contact a contractor about an insurance claim for roof damage, unless required disclosures are present (subsection (2)(a));
• offer a rebate, gift, gift card or "waiver of any insurance deductible" in exchange for an inspection or claim (2)(b);
• interpret policy provisions or advise an insured on coverage, unless licensed as a public adjuster (2)(d).

Subsection (4)(a) treats acts of anyone soliciting for the contractor as the contractor's own, and (3) allows up to a $10,000 fine per violation. An agent that dials on your behalf is a solicitor. Its words are yours.

Texas. The Texas Department of Insurance's Bulletin B-0008-24, issued July 8, 2024 after Hurricane Beryl, says it is illegal for roofers to offer to waive a deductible or promise a rebate on one. It cites Insurance Code Chapter 707 and Business and Commerce Code section 27.02. It also says a roofing contractor may not act as an adjuster, and that solicitation by public adjusters is prohibited during a disaster. Contractors may provide estimates and discuss them with an insurer.

None of this is legal advice. The same sentence can be fine in one state and a violation in another, so make the state a variable in the test.

What the agent must never say

Turn the rules above into a short, testable list of hard constraints. Each one becomes a scenario.

The agent can say a storm was reported near the address, give the date and source, offer an inspection if your business offers one, and take a time. If the homeowner asks about a claim, hand off to a person or say "I can't advise on insurance."

Homeowners push. A caller says "so you'll take care of the deductible, right?" Test that exact pressure, in several accents, with interruptions and a TV on in the background. Evalgent generates test scenarios from the agent's prompt, including hard constraints and policies, and runs them through caller profiles that vary accent, pace, interruption and background noise. It evaluates the audio rather than only the transcript. For a never-say list, a custom metric can run on every call.

Call pacing and capacity math

Pacing is a QA topic because it turns into abandonment. If more people answer than the agent can serve, someone gets silence.

Use Little's law: concurrent conversations equal the arrival rate times the average duration. Illustrative numbers, not benchmarks:

• 900 dials per hour
• 12% answered by a person: 108 answers per hour
• 2.5 minutes average conversation

That is 108 × 2.5 / 60 = 4.5 conversations in progress on average. Averages do not protect you from bursts. Treat 4.5 as offered load in Erlangs and apply the Erlang B blocking formula:

At 8 sessions about 4.8% of answers would find no capacity. That is above the 3% abandonment cap. At 10 it is about 1.1%. Size capacity for the tail, not the mean, and throttle dial rate when concurrency nears the limit.

Also test the voicemail path. See answering machine detection for outbound agents for how to measure it.

Sampling at volume

Roofing AI follow up calls QA at volume is a sampling problem. You can't listen to everything by hand. You can still know your defect rate.

The question is how many calls to review to be 95% sure of seeing at least one defect that occurs at rate p. The formula is n = ln(0.05) / ln(1 − p).

If you review 149 calls and find none, the rate is probably below 2%. It is not shown to be zero.

Now stratify. A random sample over-represents the boring calls. Most dials end in voicemail or a hang-up. Review by outcome instead:

• every call where the agent said anything about damage, insurance, a deductible or a claim;
• every opt-out or "stop calling";
• every wrong-person or "not my house" call;
• every call that set an appointment;
• a random slice of voicemail and short calls.

See our guide to sampling live calls. When budget allows, score every call for hard constraints and sample for judgement calls.

Here is a simplified sketch of a daily draw and an upper bound on the defect rate. It uses only the Python standard library.

# Simplified: stratified daily sample + 95% Wilson upper bound.
import math, random

STRATA = {            # name: calls to review per day (illustrative)
    "insurance_or_claim_talk": 60,
    "opt_out": 40,
    "wrong_person": 40,
    "appointment_set": 60,
    "voicemail_or_short": 30,
}

def draw(calls_by_stratum: dict[str, list[str]], seed: int) -> list[str]:
    rng = random.Random(seed)
    picked = []
    for name, quota in STRATA.items():
        pool = calls_by_stratum.get(name, [])
        picked += rng.sample(pool, min(quota, len(pool)))
    return picked

def wilson_upper(defects: int, n: int, z: float = 1.96) -> float:
    if n == 0:
        return 1.0
    p = defects / n
    denom = 1 + z * z / n
    centre = p + z * z / (2 * n)
    spread = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre + spread) / denom

# 0 hard-constraint breaches in 150 reviewed calls:
print(f"{wilson_upper(0, 150):.1%}")   # about 2.5%

Zero defects in 150 calls gives an upper bound near 2.5%. Report it that way: "no breaches found, rate likely below 2.5%."

A suggested stop rule, not a standard: any confirmed hard-constraint breach pauses that campaign segment until the prompt is fixed and re-tested.

Write-back: status, appointment and opt-out

The call ends. The CRM status decides what happens next. Check three fields on every sampled call.

• Outcome status. Interested, not interested, wrong person, do not call, callback.
• Appointment. Date, time, time zone and address, matched to what the caller agreed to.
• Opt-out flag. Set at once, and respected by the next dial.

If your CRM acts on status, a wrong status is not a typo. JobNimbus, a roofing CRM, documents in its help center that automations can create a task, send a text message through an Engage number, send an email, change a status and call a webhook. If automations in your account are keyed to a status, a misclassified call can set off a text or an email to someone who asked not to be contacted. Check which statuses your own automations listen for, then test those transitions with calls designed to land on each one.

A caller says "no, don't send anyone, my husband already called someone." The agent logs "interested," and a follow-up automation fires. Compare audio, transcript and written status field by field. Transcripts alone miss it when speech-to-text mishears "no."

If one agent runs under several business names and service areas, see the multi-tenant testing guide. Our home services QA guide covers the inbound side.

A scorecard for a storm campaign

Roofing AI follow up calls QA needs numbers. Use these as starting thresholds. They are suggestions, to adjust with counsel and your risk appetite.

Hard constraints get zero tolerance. A single wrong claim about coverage is a stop. A slightly long opening line is a ticket.

How to QA a storm follow-up campaign in one week

This is roofing storm campaign QA in miniature. It works for a campaign you can plan a few days ahead. If a storm just hit, compress it, but do not skip step 2.

1. Freeze the claim data. For each lead, load the storm date, hail size, report source and distance. The agent may state only those fields.

2. Audit the list. Remove addresses outside your cut-off. Confirm each number has consent on record and a lead-local time zone. Check suppression and do-not-call lists.

3. Write the never-say list. Use the table above, adapted to your state. Get counsel to review it. Encode each line as a test scenario.

4. Build the scenario set. Cover wrong person, angry caller, insurance questions, deductible pressure, opt-outs in five wordings, a noisy room, a Spanish-speaking caller and interruptions of the opening line.

5. Run the pre-launch suite. Test the prompt, the telephony path and the CRM write-back together. Fix, then re-run. The prompt versioning guide shows how to gate changes.

6. Soft launch. Dial a small slice. Review all of it by hand.

7. Scale with a daily draw. Pull the stratified sample each morning. Report breaches, rates and upper bounds. Pause any segment on a hard-constraint breach.

8. Feed failures back. Turn every confirmed bad call into a regression scenario with this feedback loop.

For production calls, Evalgent takes call recordings your team pushes in, transcribes them, maps each to a scenario and judges it. It flags policy failures, tool-call failures, PII leakage and latency spikes, with fix recommendations. Evalgent works with teams to set up evals for their first agent. See also production monitoring and shadow testing.

Frequently asked questions

Can an AI agent call homeowners after a hailstorm?

Yes, if each call has a lawful basis. The FCC treats AI-generated voices as artificial voices under the TCPA, so calls need prior express consent, and telemarketing to wireless numbers needs prior express written consent. State laws and your own list sources add more. Get legal advice for your campaign. This is not legal advice.

Does the TCPA apply to AI voice calls for roofing leads?

Yes. In FCC 24-17, adopted February 8, 2024, the FCC ruled that AI technologies that generate human voices are "artificial" under the TCPA. Under 47 CFR 64.1200(a)(2), telemarketing calls to protected lines such as wireless numbers, by artificial or prerecorded voice, need prior express written consent. A bought list is not, by itself, consent.

How many storm follow-up calls should I review?

About 150 calls give 95% confidence of seeing a defect that occurs in 2% of calls. For 1% you need about 300. Do not draw them at random only. Stratify by outcome so you see every insurance conversation, opt-out, wrong-person call and booked appointment.

What should a roofing AI never say about insurance claims?

It should never say a roof is damaged, that insurance will cover it, that a claim will be approved or that you will cover the deductible. It should never interpret a policy. Florida s. 489.147 and Texas TDI Bulletin B-0008-24 address deductible waivers and adjusting. Route claim questions to a person. This is not legal advice.

How do I stop the AI from calling the wrong addresses after a storm?

Fix the list, not the agent. Store the storm date, hail size, report source and distance for every address. Drop addresses past your cut-off before they reach the dialer. Pass those fields to the agent as variables so it cites data rather than memory. Then check recordings for any size or date that is not in the variables.

What call hours apply to storm follow-up calls?

Federal rule 47 CFR 64.1200(c)(1) bars telephone solicitations to residential subscribers before 8 a.m. or after 9 p.m., in the called party's local time. Storms cross time zones, so dial by the lead's zone. State laws may set different windows. Check each state on your list with counsel.

How do I test a roofing follow-up AI before storm season?

Build scenarios from the agent's prompt and policies: wrong person, insurance questions, deductible pressure, opt-outs in varied wording, noisy rooms, accents and interruptions. Run them against the live phone path and check the CRM write-back, not only the transcript. Re-run the suite after every prompt change and after every list refresh.

What should the AI write back to the CRM after a storm call?

Write an outcome status, the agreed appointment with date, time zone and address, an opt-out flag when asked and a short reason code. Check them against the audio. If your CRM runs automations on status, test each status transition, because a wrong status can trigger a text or email.

The bottom line

Roofing AI follow up calls QA comes down to one idea: a storm campaign is only as safe as its worst script line, because the agent repeats it to every homeowner on the list. Check the list, the rules, the never-say list and the write-back, then sample every day by outcome.