Auditing AI Answers on HOA Balances and Board Questions

A homeowner calls on a Tuesday night. She asks when the next board meeting is. The agent says "November 21." The board moved it to November 12 two weeks ago. The new notice is posted at the clubhouse and on the portal. The agent retrieved the old PDF. Nothing errored, and the call summary says "resolved."

The same night, another owner asks for his balance. The agent says "$412.50." He paid $400 that morning by e-check. The payment is pending, so the ledger still shows the old amount. He hears a number with no date and no mention of his payment. Tomorrow he calls a human, upset.

This post is for product teams at community association management software companies. Your HOA voice agent answers status questions for many management companies and hundreds of associations. Your HOA call center AI is live, and it is mostly right. The question is how often it is wrong, where, and how you would prove it. You get a failure map, the statutes behind the dates, sampling math, a ledger-diff script, a rubric and an agreement check.

Pre-launch testing of lot lookups and violation answers is covered in our post on testing HOA AI phone calls. The wider program sits in the property management AI answering QA playbook. This post is about production: auditing answers that already went out.

What an HOA call center AI answers on status calls

On status calls, the caller wants a fact: a balance, a date, a document or a name. HOA call center AI for these calls is shipping now. Vantaca's HOAi Voice page says it answers "balances, payment due dates, and assessment details" plus "board meeting dates" after hours, from "live community data."

Each answer has a source of truth. Name it for every question type before you score anything.

Status answer: a spoken fact the caller will act on, read from a record at a specific moment. It is correct only if the record, the moment and the read-back all line up.

This is the front door of an agentic workflow. The agent captures the question, calls a tool, reads a record and speaks a number. Then a write-back puts a call summary in the platform. A wrong spoken balance becomes a wrong note in the owner's file, which shapes the next human call.

Five ways a balance answer goes wrong

Most balance errors are not hallucinations. The tool returned a real number. The number was stale, partial, misread or explained wrongly.

1. No as-of date

Ledgers are snapshots. Many integrations sync on a schedule, and the agent reads a cached copy. If the agent says "$412.50" with no date, the caller hears a live number. A correct answer sounds like "As of this morning, your balance is $412.50." The audit check is simple: does the answer carry an as-of time, and is the snapshot within your staleness limit?

2. Pending payments ignored

Between payment and posting there is a gap. An e-check, a lockbox deposit or a bank bill-pay item can sit in a pending state. If the agent reads only posted entries, a paid owner hears that he still owes. Worse, the agent may say a late fee "will apply." Your audit needs the processor's pending list at call time, not just the ledger.

3. Rounding and number read-back

Speech adds its own errors. "$1,250.50" can be voiced as "twelve fifty fifty." A text normalizer can turn "$400.00" into "four hundred dollars and zero cents," which is clumsy but safe. The dangerous cases are quieter. "$4,000" heard as "$400" on a special assessment. A rounded "about four hundred" when the ledger says $412.50. Store money as integer cents, format once, and audit the audio, not only the text. A transcript can show "$412.50" while the voice said something else.

4. Payment application order

This is the gotcha most teams miss. When an owner is delinquent, states disagree on where a payment goes first. Florida applies payments "first to any interest accrued, then to any administrative late fee, then to any costs and reasonable attorney fees incurred in collection, and then to the delinquent assessment" (Fla. Stat. 720.3085(3)(b)). California says payments "shall first be applied to the assessments owed" (Cal. Civ. Code 5655).

The orders are close to opposite. An agent that says "your $400 covered the assessment" may be right in Sacramento and wrong in Orlando. One prompt across all clients will get one of them wrong. The rule belongs in per-state config, and the audit should check it per state.

5. Disclosure before verification

An HOA assessment balance is personal financial data. Owner verification must pass before the agent reads one. Our posts on credit union member verification and loan servicing balance answers cover the verification design. For the audit, one rule: any balance spoken before a verification pass is a severity-1 failure, even if the number was right.

Board and governance answers fail differently

Balance errors come from data timing. Board errors come from document versions. The agent answers meeting questions by retrieving a notice, a calendar entry or minutes. When a meeting moves, two notices exist. Retrieval can return the old one, because it matches the question just as well.

Dates matter here because statutes set board meeting notice windows. In Florida, board meeting notices "must specifically identify agenda items" and be posted "at least 48 hours in advance of a meeting, except in an emergency." A meeting that will consider an HOA special assessment needs written notice "not less than 14 days before the meeting" (Fla. Stat. 720.303(2)(c)). California requires notice of a board meeting "at least four days before the meeting," or two days for an executive-session-only meeting (Cal. Civ. Code 4920).

A wrong spoken date does not break those rules; the posted notice does the legal work. But an owner who missed a special assessment vote because of it will complain. Your client then explains why its phone line contradicted its own notice.

Minutes and records have their own clocks. California says minutes, draft minutes or a summary "shall be available to members within 30 days of the meeting" (Cal. Civ. Code 4950). Florida requires official records to be available "within 10 business days" after a written request (Fla. Stat. 720.303(5)). An agent that says "minutes are posted the next day" invents a promise the client never made.

Date math is a known weak spot. The Test of Time benchmark built 1,850 time-arithmetic questions. Duration questions were the hardest. GPT-4 scored 16% on that category in the paper's results. "How many days until my late fee?" is a duration question. Do not let the model compute it. Compute it in code and pass the result as a field.

Budget and reserve questions need a different rule. The agent quotes the adopted budget or reserve study and says where to find it. Score any computed reserve figure as a failure, even if it is close.

Collections-related statements

Some balance calls drift into collections. "Am I going to be sent to an attorney?" "Will you put a lien on my house?" The Fair Debt Collection Practices Act covers "debt" for "personal, family, or household purposes" (15 U.S.C. 1692a). It applies to a "debt collector," meaning a business whose principal purpose is collecting debts, or one that regularly collects debts "owed or due another."

Whether that is your client depends on its role. An association collecting its own assessments is usually a creditor. A collection firm or law firm working for the association can be a debt collector. A management company may fall under the exclusion for collection "incidental to a bona fide fiduciary obligation." Configure the agent per client role. Audit every call that mentions liens, attorneys, collections or foreclosure. Our post on AI payment arrangement calls and FDCPA covers the call-level rules if your client is a collector.

This post describes how to audit agent answers. It is not legal advice; have counsel confirm the rules for each state and client role.

The audit design: stratify, diff, score, agree

A good audit answers three questions. How often is each answer type wrong? Where? Can you trust the scores?

Stratified sampling

This is AI call auditing, and a random sample of all calls hides the risky ones. Pool hours dominate volume. Special assessment calls are rare and high stakes. Split calls into strata by question type and risk, then sample each stratum on its own.

Size the high-volume strata for a target margin. The formula for estimating a proportion is n = z² × p(1 − p) / e². With z = 1.96, an expected error rate p = 0.05 and a margin e = 0.03, n = 3.8416 × 0.0475 / 0.0009 ≈ 203 calls per stratum.

Rare strata need a different tool. If you review n calls and find zero errors, the "rule of three" gives an approximate 95% upper bound of 3/n. Zero errors in 60 special assessment calls still allows a true rate near 5%. That tells you whether "no errors found" is reassuring or not.

Then stratify by client. One stale calendar can produce hundreds of wrong answers while the global rate looks fine. Our guides on sampling live calls and on when to score every call go deeper on the tradeoff.

Worked math: how one stale notice becomes 212 wrong dates

Take an illustrative month. The HOA call center AI handles 40,000 calls across all clients. Board meeting date questions are 6% of them, or 2,400. One large association moves its meeting, and its old notice stays in the retrieval index. That association produces 9% of meeting questions, or 216 calls. If retrieval returns the old notice on 98% of them, 212 callers hear the wrong date.

Across all clients, that is an 8.8% error rate on meeting dates. A 200-call random sample spread over every question type would include about 12 meeting calls. You would expect about one error, and you might see none. A stratified sample that breaks out meeting dates by client finds the cluster in the first week. These numbers are a worked example, not measured data.

Ledger diffs

Every balance answer gets diffed against the ledger as it stood at the call minute, plus pending payments. Pull the spoken amount from the transcript, then confirm it in the audio. The script below is simplified. It assumes you can query ledger entries by posting time and the processor's pending list.

# Simplified: classify one spoken balance against ledger state at call time.
import re
from decimal import Decimal

AMOUNT = re.compile(r"\$?(\d{1,3}(?:,\d{3})*|\d+)(?:\.(\d{2}))?")

def spoken_cents(text: str) -> int | None:
    m = AMOUNT.search(text)
    if not m:
        return None
    dollars = int(m.group(1).replace(",", ""))
    cents = int(m.group(2) or 0)
    return dollars * 100 + cents

def audit_balance(call, ledger, processor, snapshot_max_age_min=60):
    said = spoken_cents(call.agent_balance_utterance)
    posted = sum(e.cents for e in ledger.entries(call.owner_id)
                 if e.posted_at <= call.started_at)
    pending = sum(p.cents for p in processor.pending(call.owner_id)
                  if p.created_at <= call.started_at)
    findings = []
    if call.balance_spoken_before_verification:
        findings.append("SEV1_DISCLOSED_BEFORE_VERIFY")
    if said is None:
        findings.append("NO_AMOUNT_PARSED")
    elif said != posted:
        findings.append(f"LEDGER_MISMATCH said={said} posted={posted}")
    if pending and "pending" not in call.agent_text.lower():
        findings.append(f"PENDING_NOT_MENTIONED pending={pending}")
    if not call.as_of_stated:
        findings.append("NO_AS_OF_DATE")
    age = (call.started_at - call.snapshot_at).total_seconds() / 60
    if age > snapshot_max_age_min:
        findings.append(f"STALE_SNAPSHOT age_min={age:.0f}")
    return findings or ["PASS"]

Run the same pattern for meeting answers. Store a notice version ID on every notice. Log which version the retrieval step returned. A meeting answer fails if the version was superseded before the call started. That check needs no model at all.

A rubric with severities

The rubric turns findings into scores your client can read. Keep it fixed and versioned between audit periods.

Set thresholds per severity. A reasonable starting point: zero tolerance on Sev 1, under 2% on Sev 2 and under 5% on Sev 3 per stratum per month. Those are our suggested defaults, not industry standards.

Reviewer agreement

Scores only mean something if two reviewers would give the same one. Double-score 10–20% of the sample and compute Cohen's kappa. Kappa corrects raw agreement for the agreement you would expect by chance.

Here is a worked example. Two reviewers score 200 calls. Both pass 150 and both fail 30. Reviewer A fails 12 that B passes, and B fails 8 that A passes. Observed agreement is 180 / 200 = 0.90. A fails 21% of calls and B fails 19%. Chance agreement is 0.21 × 0.19 + 0.79 × 0.81 = 0.680. Kappa = (0.90 − 0.680) / (1 − 0.680) ≈ 0.69.

A kappa near 0.69 is usable but not tight. Read the 20 disagreements. They usually point to a vague rubric line, like whether "about four hundred" counts as a mismatch.

Model judges need the same check. The LLM-as-a-judge study by Zheng et al. found strong judges reached "over 80% agreement" with human preferences. It also named "position, verbosity, and self-enhancement biases." That was on chat answers, not spoken balances. Calibrate any judge against your own double-scored set. Our post on LLM-as-a-judge limits in voice AI lists where judges miss.

Evalgent fits this step. Its judge scores from the audio, not only the transcript, so a voiced "four hundred" against a written "$412.50" shows up. Verdicts come with evidence, and humans review them. Custom metrics, such as "as-of date stated" or "pending payment mentioned," can run on every call.

How to audit HOA balance and board answers in production

1. Name the source of truth for each question type. Use the table above. Record which system, which field and which timestamp define "correct."

2. Log what the agent read. Store the ledger snapshot time, the pending list, the notice version ID and the tool response with every call.

3. Define strata. Cross question type with risk, then add client. Oversample special assessments, meeting dates and anything mentioning collections.

4. Size the sample. Use n = z² × p(1 − p) / e² for high-volume strata and the 3/n bound for rare ones.

5. Run automatic diffs first. Ledger diff, pending check, notice version check and verification-order check. These need no human.

6. Score the rest with the rubric. Policy fidelity, collections language and handoffs need a judge or a person.

7. Double-score a slice. Compute kappa every period. Below about 0.6, rewrite the rubric line before trusting the scores.

8. Report by client and stratum. Show error rates with intervals, not single numbers. Flag clusters, like one association's stale notice.

9. Fix the cause, not the call. Re-index superseded notices, add as-of fields, move payment-order rules into per-state config.

10. Re-audit the fixed stratum. Confirm the error rate dropped before you close the finding.

For an HOA call center AI, Evalgent's production monitoring follows this shape. You push call recordings. Calls are transcribed, mapped to scenarios and judged. Flags cover PII leakage, tool-call failures, latency spikes and policy failures, each with a fix recommendation. For teams running one agent across many clients, our guide to multi-tenant voice agent testing explains how to keep per-client configs testable. For ongoing cadence, see continuous voice agent auditing.

Frequently asked questions

Can an HOA AI tell a homeowner their balance?

An HOA AI agent can tell a homeowner their balance after verifying the caller's identity. The answer should come from the ledger at call time, state an as-of date and mention pending payments. Speaking a balance before verification is a privacy failure, even when the amount is right. Audit verification order on every balance call.

What should an HOA AI say about a pending payment?

An HOA AI should say that a payment is pending, give its amount and date, and state the balance both before and after it posts. It should never say a late fee will apply when a pending payment covers the amount due. Audit this by comparing the processor's pending list at call time to the spoken answer.

How do you audit AI answers on HOA balances?

To audit AI answers on HOA balances, diff each spoken amount against the ledger as it stood at the call minute. Include pending payments. Confirm the amount in the audio, not only the transcript. Check the as-of date, verification order and state payment-application rules. Score findings with a fixed rubric and report error rates by client.

How many HOA calls should you sample for an audit?

For a high-volume question type, about 203 calls estimates a 5% error rate within plus or minus 3 points at 95% confidence. Rare, high-risk types need oversampling. Zero errors in n calls only bounds the true rate near 3/n. Sample by question type and client, so one association's stale data does not hide in the average.

Can AI give board meeting dates to homeowners?

AI can give board meeting dates if it reads the current posted notice, not a superseded one. Store a version ID on every notice and log which version the agent used. Meeting moves are the main risk, because old and new notices both match the question. Florida and California set minimum notice periods, so dates carry weight.

Does the FDCPA apply when an HOA AI discusses a balance?

The FDCPA applies to a "debt collector" as defined in 15 U.S.C. 1692a. An association collecting its own assessments is usually a creditor. A collection agency or law firm collecting for it can be a debt collector. Configure the agent per client role, and audit calls that mention liens or attorneys. Confirm specifics with counsel.

How do you measure reviewer agreement on call audits?

Reviewer agreement on call audits is measured by double-scoring 10–20% of sampled calls and computing Cohen's kappa. Kappa adjusts raw agreement for chance. Two reviewers who agree on 90% of calls, with fail rates of 21% and 19%, reach a kappa of about 0.69. Read every disagreement and tighten vague rubric lines.

What should an HOA AI say about special assessments?

An HOA AI should state a special assessment's amount, due dates and status from the board resolution or notice. It must say whether the assessment is proposed or adopted. It should not estimate amounts or payment plans. Florida requires 14 days' notice for meetings that consider special assessments, so the meeting date matters as much as the amount.

The bottom line

Most wrong HOA answers are real numbers read at the wrong moment or from the wrong document version. Audit them by diffing each answer against the record at the call minute, sampling by question type and client, and checking that your reviewers agree.