Testing AI Answers on Escrow, Payoff and Payments

A borrower calls on a Tuesday. She is selling her house and needs the payoff. The agent answers: "About two hundred twelve thousand seven hundred dollars, good through the thirtieth." The system of record says $212,726.03, good through November 20. The title company wires on November 28. The wire is $316.44 short of what the loan needs that day. The lien is not released. Nobody on your team hears the call.

That is the core risk of mortgage servicing AI. The answer sounded right. It was wrong in two fields and missing a qualifier.

This guide is for product and engineering teams that built a mortgage servicing voice agent. You may sell servicing software or run a servicer. Either way, you want to know if the agent's answers are right. We cover the three most common questions and how each goes wrong. Then we give a copyable rubric and Python that checks spoken numbers.

The lending and insurance voice AI pillar maps every lending call type and its rule. The post on auditing AI payment arrangement calls covers plans, fees and Reg F. We don't repeat either here.

This is not legal advice. Rules cited were checked against the primary text on October 16, 2026. Confirm how they apply to you with counsel.

Why servicing answers fail while sounding right

Servicing call center AI is no longer a pilot. In its Q2 2026 results, Rocket Companies said its AI Voice handled "more than 1 million inbound calls." That took three months from launch. In March 2026, ICE announced an AI voice agent integrated with its MSP servicing system, then in beta. It answers questions on escrow, mortgage insurance and servicing transfers, and helps with payments.

The call is the front door to a workflow. The agent looks up the loan through a tool call and turns the result into a sentence. Text-to-speech reads it out. Then the borrower acts: wires a payoff, pays by a date, or drops a dispute. The record is correct. The failure lives in the hops between that record and the borrower's ear.

Speech-to-text and number formatting. The right written form of a number depends on context. Huber and Waibel study this in Handling Numeric Expressions in Automatic Speech Recognition (2024). Their example is "1945 (year) vs. 19:45 (timestamp)." A caller's "twenty-one oh five" can be a date, a loan number fragment or an amount. If the agent hears the wrong one, it looks up or confirms the wrong thing.

The model's reasoning. Language models are weak at finance math and date logic. In FinanceBench, GPT-4-Turbo with retrieval "incorrectly answered or refused to answer 81% of questions" on a 150-question sample. Bhatia, Peyrard and Zhao, in Date Fragments, show tokenizers split dates into odd pieces. They link heavy fragmentation to accuracy drops of up to 10 points on uncommon dates. So never let the model compute a payoff or count days. Let it read fields.

Transcription hallucination. Koenecke and colleagues found in Careless Whisper (FAccT 2024) that roughly 1% of Whisper transcriptions held whole invented phrases. A transcript-only review can miss what the agent really said.

Text-to-speech. The text may say "$212,726.03." The voice can still drop the cents or misread it. Judge the agent's audio, not only its text. Our comparison of transcript and audio evaluation shows what each one misses.

The result is three kinds of corruption. A wrong number is spoken while the record is right. A wrong date is spoken, so a right number becomes wrong on the day it is used. Or the agent makes a promise the servicer cannot keep. Each one changes what the borrower does next.

Escrow: why did my payment change

"Why did my payment go up?" is the most common servicing question. The CFPB's answer lists the causes. Taxes or insurance changed. A buydown ended. An adjustable rate reset. Mortgage insurance changed. New fees appeared. Or the servicer "simply made a mistake." On a fixed-rate loan, escrow is usually the answer. These are the AI escrow questions where vocabulary matters most.

Regulation X, at 12 CFR 1024.17, defines the terms your agent will use.

Shortage: "an amount by which a current escrow account balance falls short of the target balance." The balance is still positive, just too low.

Deficiency: "the amount of a negative balance in an escrow account." The servicer advanced money it had not collected.

Surplus: the amount by which the balance "exceeds the target balance."

Cushion: extra funds the servicer may require. It "must be no greater than one-sixth (1/6) of the estimated total annual disbursements."

The servicer runs an escrow analysis at the end of each computation year. It sends the annual statement within 30 days of that year's end. A surplus of $50 or more is refunded within 30 days of the analysis. Shortages and deficiencies have menus. A large shortage may stand or be spread over "at least a 12-month period." A large deficiency may be repaid "in two or more equal monthly payments."

How the escrow answer goes wrong

• Wrong term. The agent says "deficiency" for a shortage. The borrower hears "you owe money" instead of "your balance ran low."
• Wrong cause. "Your interest rate went up" on a fixed-rate loan. It sounds plausible and is false.
• Wrong effective date. The new payment starts on the date in the analysis, not "next month." A borrower who pays the old amount lands short.
• A promise the servicer cannot keep. "We can spread that over 24 months." The regulation sets a floor of 12 months. Your client's policy sets the actual option. The agent must read the configured choice, not offer one.
• Cushion as a requirement. "We have to keep two months extra." One-sixth is a cap, not a mandate.

The fix is structural. Read the analysis fields: type, amount, spread months, new payment, effective date. Never infer them from a balance.

Payoff: a number with a date attached

A payoff is not the balance. The CFPB explains that the payoff "includes the payment of any interest due through the day you intend to pay off your loan." It may also include unpaid fees.

The rule sits in Regulation Z, at 12 CFR 1026.36(c)(3). The servicer must give an accurate payoff statement "as of a specified date." It must send it within "a reasonable time, but in no case more than seven business days" after a written request. Regulation X also treats payoff errors as urgent. Under 12 CFR 1024.35(e)(3), a payoff-balance error notice gets a response within seven business days.

So should a payoff quote AI ever speak a number? The pillar post says to follow the servicer's configured process rather than improvise. Some servicers send a written payoff statement only. Others let the agent read a system-generated quote. If yours reads one, three rules apply. Read the system value to the cent. State the good-through date. State the per-diem amount for each day after that date.

Worked example: per-diem interest (illustrative)

These numbers are illustrative, not from a real loan. Conventions vary by note and servicer, so read yours from the system.

• Unpaid principal after the November 1 payment: $212,000.00
• Note rate: 6.25%
• Interest is paid in arrears, so the November 1 payment covered October. Interest accrues from November 1.
• Day-count basis: actual/365

Per diem = 212,000 × 0.0625 ÷ 365 = $36.3014, about $36.30 per day.

Payoff good through November 20 = 212,000 + (20 × 36.3014) = 212,000 + 726.03 = $212,726.03.

Now the errors from the opening:

A short wire leaves the loan unpaid and the lien in place. An over-quote is still wrong. Escrow is separate. Under 12 CFR 1024.34(b), the servicer returns leftover escrow within 20 business days of payoff. "We'll subtract your escrow from the payoff" is a promise the servicer may not keep.

Payments: posting, grace and partial payments

The third question for mortgage servicing AI is "Did my payment go through?" A mortgage payment AI assistant has to separate four things borrowers blur together.

Posting date. Under 12 CFR 1026.36(c)(1)(i), a servicer must credit a periodic payment "as of the date of receipt." If the servicer sets written payment requirements and accepts a payment that breaks them, it must credit it "as of five days after receipt." The agent must say which date it means.

Grace period versus late fee. The grace period comes from the note, not from a default. The periodic statement must show "the amount of any late payment fee, and the date on which that fee will be imposed," per 12 CFR 1026.41(d)(1)(ii). Read that date and amount from statement data. "You have a 15-day grace period" is a guess. Small servicers are exempt from the periodic statement rule, so the field may be missing. Test that case too.

Partial payments. A payment below the periodic amount may sit in a suspense or unapplied funds account. Under 1026.36(c)(1)(ii), the servicer must show that suspense balance on the periodic statement. Once enough funds accumulate to cover a periodic payment, it must treat them as one. "We applied your $800" is false if the $800 is in suspense. The borrower may still be past due.

Pending versus posted. A scheduled ACH payment is not posted yet. "You're all set" on a pending payment can end in a late fee.

Before any answer: who is calling and what they raised

In mortgage servicing AI, accuracy starts before the first number. Four gates come first.

Authentication. Do not read any loan data until the caller is verified. A misheard loan-number digit can pull another borrower's escrow balance. Our guide to PII leakage in spoken readbacks covers masking and logs.

Successors in interest. "My mother passed and the house is mine now." Under 12 CFR 1024.38(b)(1)(vi), servicers must "promptly facilitate communication" with potential successors and tell them what documents confirm their status. Once confirmed, a successor "shall be considered a borrower" under 1024.30(d). The agent should route this caller, neither refusing nor disclosing.

Errors and information requests raised on the call. Notices of error under 1024.35(a) and information requests under 1024.36(a) are written. A spoken complaint is not one by itself. But the official comment to 1024.38(b)(5) says servicer procedures should tell borrowers unhappy with an oral resolution how to submit a written notice. So on "you charged me a late fee I don't owe," the agent logs it and gives the written process.

Delinquency and loss mitigation. Under 12 CFR 1024.39(a), servicers must make good-faith efforts at live contact by the 36th day of delinquency and inform borrowers about loss mitigation options, if appropriate. Under the comments to 1024.41(b)(1), an application is "considered expansively." An inquiry where the borrower gives information the servicer would evaluate can become an application. So the agent must never say "you don't qualify." It should capture the details and route them.

The answer-accuracy rubric for servicing questions

Here is a framework for grading mortgage servicing AI answers. Every answer type has a field it must match, a tolerance, required qualifiers and forbidden moves. Grade each call against the row for the question asked. Any miss on a zero-tolerance field fails the call.

Two scoring rules matter. Amounts compare in integer cents and dates as dates. And "spoken" means what the borrower heard. Take it from the agent's audio channel, transcribed independently, not from the LLM's text output. If you take the value from the tool result, you are grading the record against itself. The payment arrangement audit makes the same point about plan read-backs. For argument-level capture errors, see tool-argument accuracy from speech.

Checking spoken amounts and dates in Python

This script applies two rows of the rubric. It pulls dollar amounts and dates from the agent's transcribed audio. It handles "$212,726.03" and spelled-out words, since STT output varies. Then it checks the record, qualifiers and forbidden phrases. It is simplified. Real code needs a fuller number parser and per-client config.

# Simplified: check spoken amounts, dates and qualifiers against the servicing record.
import re
from datetime import date
from decimal import Decimal

UNITS = {w: i for i, w in enumerate(
    "zero one two three four five six seven eight nine ten eleven twelve thirteen "
    "fourteen fifteen sixteen seventeen eighteen nineteen".split())}
TENS = {w: 10 * i for i, w in enumerate(
    "_ _ twenty thirty forty fifty sixty seventy eighty ninety".split()) if w != "_"}
SCALES = {"hundred": 100, "thousand": 1000}
MONTHS = {m: i for i, m in enumerate(
    "january february march april may june july august september october "
    "november december".split(), 1)}
ORDINALS = {"first": 1, "second": 2, "third": 3, "fifth": 5, "eighth": 8, "ninth": 9,
            "twelfth": 12, "twentieth": 20, "thirtieth": 30}

def words_to_int(words):
    total, cur = 0, 0
    for w in words:
        if w in UNITS: cur += UNITS[w]
        elif w in TENS: cur += TENS[w]
        elif w == "hundred": cur *= 100
        elif w == "thousand": total, cur = total + cur * 1000, 0
    return total + cur

def spoken_amounts(text):
    t = text.lower().replace("-", " ")
    found = [Decimal(m.replace(",", "")) for m in re.findall(r"\$([\d,]+(?:\.\d{2})?)", t)]
    num = r"(?:%s)" % "|".join(list(UNITS) + list(TENS) + list(SCALES) + ["and"])
    pat = rf"((?:{num}\s+)*{num})\s+dollars(?:\s+and\s+((?:{num}\s+)*{num})\s+cents)?"
    for d, c in re.findall(pat, t):
        val = Decimal(words_to_int(d.split())) + Decimal(words_to_int(c.split()) if c else 0) / 100
        found.append(val)
    return found

def spoken_dates(text, year):
    t = text.lower().replace("-", " ")
    out = []
    for mon, day in re.findall(r"(%s)\s+(\d{1,2})(?:st|nd|rd|th)?\b" % "|".join(MONTHS), t):
        out.append(date(year, MONTHS[mon], int(day)))
    for mon, tens, unit in re.findall(
            r"(%s)\s+(twenty|thirty)?\s?(%s)" % ("|".join(MONTHS), "|".join(ORDINALS)), t):
        day = ORDINALS[unit] + (TENS[tens] if tens else 0)
        out.append(date(year, MONTHS[mon], day))
    return out

RUBRIC = {  # question type -> fields, tolerance in dollars, required qualifiers
    "payoff": {"amount_field": "payoff_amount", "date_field": "good_through",
               "tolerance": Decimal("0.00"),
               "qualifiers": ["good through", "per day"],
               "forbidden": ["about", "roughly", "around", "approximately"]},
    "escrow_change": {"amount_field": "new_escrow_payment", "date_field": "effective_date",
                      "tolerance": Decimal("0.00"),
                      "qualifiers": ["shortage", "escrow analysis"],
                      "forbidden": ["interest rate went up", "deficiency"]},
}

def check(qtype, agent_text, record):
    rule, issues = RUBRIC[qtype], []
    amounts = spoken_amounts(agent_text)
    want = record[rule["amount_field"]]
    if not any(abs(a - want) <= rule["tolerance"] for a in amounts):
        issues.append(f"{rule['amount_field']}: spoke {[str(a) for a in amounts]}, record {want}")
    dates = spoken_dates(agent_text, record[rule["date_field"]].year)
    if record[rule["date_field"]] not in dates:
        issues.append(f"{rule['date_field']}: spoke {[str(d) for d in dates]}, "
                      f"record {record[rule['date_field']]}")
    low = agent_text.lower()
    issues += [f"missing qualifier: '{q}'" for q in rule["qualifiers"] if q not in low]
    issues += [f"forbidden phrase: '{f}'" for f in rule["forbidden"]
               if re.search(rf"\b{f}\b", low)]
    return issues

record = {"payoff_amount": Decimal("212726.03"), "good_through": date(2026, 11, 20),
          "new_escrow_payment": Decimal("612.40"), "effective_date": date(2027, 1, 1)}

calls = {
    ("payoff", "A"): "Your payoff is $212,726.03, good through November 20th. "
                     "After that, add $36.30 per day.",
    ("payoff", "B"): "Your payoff is about two hundred twelve thousand seven hundred "
                     "dollars, good through November thirtieth.",
    ("escrow_change", "C"): "Your escrow analysis found a shortage, so starting "
                            "January 1st your escrow payment will be $612.40.",
    ("escrow_change", "D"): "Your interest rate went up, so your escrow payment "
                            "is now $621.40 starting January 1st.",
}
for (qtype, call_id), text in calls.items():
    issues = check(qtype, text, record)
    print(f"call {call_id} [{qtype}]: {'PASS' if not issues else 'FAIL'}")
    for i in issues:
        print("   -", i)

Output:

call A [payoff]: PASS
call B [payoff]: FAIL
   - payoff_amount: spoke ['212700'], record 212726.03
   - good_through: spoke ['2026-11-30'], record 2026-11-20
   - missing qualifier: 'per day'
   - forbidden phrase: 'about'
call C [escrow_change]: PASS
call D [escrow_change]: FAIL
   - new_escrow_payment: spoke ['621.40'], record 612.40
   - missing qualifier: 'shortage'
   - missing qualifier: 'escrow analysis'
   - forbidden phrase: 'interest rate went up'

Call B is the opening scene. Call D is a digit transposition: $612.40 became $621.40. It is the kind of slip a listener misses and a transcript reader may miss too. The "deficiency" ban works here because this record holds a shortage. In production, build the forbidden list from the record: ban "deficiency" when the account has a shortage, and the reverse.

Two gotchas. The date parser assumes the record's year, which breaks across a year end. And one exact match is not enough. If the agent says two payoff figures, flag the call even if one is right.

How to test a mortgage servicing voice agent before a change ships

1. List the question types your agent answers. Start with escrow change, shortage repayment, payoff, per diem, next payment, late fee and payment status. Add successors, disputes and hardship as routing cases.

2. Write a rubric row for each. Name the system field, tolerance, required qualifiers and forbidden phrases. Store it as config per servicer client, since grace periods and payoff policy differ.

3. Build test loans with known answers. Cover a shortage, a deficiency, a surplus over $50, a loan with suspense funds, a pending ACH payment and a payoff that crosses a month end. Evalgent generates scenarios from the agent's prompt and flows, including branches and hard constraints, so missing question types show up early.

4. Call the agent the way borrowers do. Vary accent, pace, interruptions and background noise. Include a caller who asks for "a ballpark." Use the real phone path, so codec effects on digits show up.

5. Grade what the borrower heard. Transcribe the agent's audio independently and run the rubric check on it. Evalgent evaluates from the audio as well as the transcript, with evidence for each verdict.

6. Gate the release on zero-tolerance fields. A wrong payoff cent, a wrong good-through date or loan data before verification blocks the change. The process in shipping voice agent prompt changes safely applies directly.

7. Run the check on live calls. Push recordings, map each call to a question type and score it. Evalgent's production monitoring flags PII leakage, tool-call failures and policy failures with fix recommendations. Each miss becomes a new test, as in turning production failures into tests.

Onboarding a new servicer client? Use the go-live checklist for each new customer's AI phone agent. For reporting, see metrics for a financial services voice agent.

Frequently asked questions

Can AI give a mortgage payoff quote over the phone?

A mortgage servicing voice agent can read a payoff quote only if the servicer's process allows it. It must read the system-generated figure to the cent, state the good-through date and give the per-diem amount after that date. It should never compute the payoff itself. Many servicers instead send a written payoff statement, which Regulation Z requires within seven business days of a written request.

What is the difference between an escrow shortage and a deficiency?

An escrow shortage means the escrow balance is positive but below the target balance from the escrow analysis. An escrow deficiency means the balance is negative because the servicer paid more than it held. Regulation X, 12 CFR 1024.17, gives different repayment options for each. An AI agent that swaps the two terms gives the borrower the wrong picture.

How is per diem interest calculated on a mortgage payoff?

Per-diem interest is usually the unpaid principal times the note rate, divided by 365 or 360 days, depending on the loan. On $212,000 at 6.25% using 365 days, it is about $36.30 per day. The payoff adds per-diem interest for each day from the last paid-through date to the payoff date. The servicing system, not the agent, should compute it.

How long does a servicer have to send a payoff statement?

Under 12 CFR 1026.36(c)(3), a servicer must send an accurate payoff statement within a reasonable time, and no more than seven business days after a written request. Loans in bankruptcy or foreclosure get a reasonable-time standard instead. A voice agent should capture the request and delivery details accurately and route them into that process.

Does a partial mortgage payment go into a suspense account?

A partial mortgage payment, meaning less than a full periodic payment, may be held in a suspense or unapplied funds account. Regulation Z requires the servicer to show that balance on the periodic statement. Once suspense funds cover a full periodic payment, the servicer must credit them as one. A voice agent should not tell a borrower a partial payment was "applied."

Can a voice agent take a notice of error over the phone?

Regulation X defines notices of error and information requests as written. A phone complaint is not a formal notice of error by itself. But servicer procedures should tell borrowers unhappy with an oral resolution how to submit a written notice. A voice agent should log the complaint and give the written process. It should never tell the caller there is nothing to dispute.

How do you test a mortgage servicing voice agent?

Testing a mortgage servicing voice agent starts with a rubric per question type: the system field, tolerance, required qualifiers and forbidden phrases. Build test loans with known answers. Call the agent with varied caller profiles over the real phone path. Transcribe the agent's audio independently and compare spoken amounts and dates with the servicing record. Block releases on any zero-tolerance miss.

How should an AI agent verify a borrower before sharing loan data?

A voice agent should verify a borrower before sharing loan data, using the servicer's configured factors. It should confirm with masked values, never read a full loan number and re-verify before sensitive actions. A caller who says they inherited the property is a potential successor in interest. Route them to the servicer's confirmation process instead of refusing or disclosing.

The bottom line

Mortgage servicing AI fails on escrow, payoff and payment questions by speaking a wrong number, a wrong date or a promise the servicer cannot keep, while sounding right. Grade every answer against a system-of-record field, with zero tolerance on amounts and dates and required qualifiers like "good through," and check what the borrower actually heard.