AI Legal Intake: Is It Losing You Cases?

An intake call can sound flawless and still lose a case. The caller says, "It happened on March thirteenth, and the other driver works for Whitfield Logistics." The agent says, "An attorney will call you back." The case file then holds March 30, "Widfield Logistics" and a deadline 17 days too generous. The conflict check finds nothing, because nothing is spelled right. Nobody notices until a clock runs out.

This post is an audit, not a build guide. AI legal intake accuracy can be measured. It gives 12 checks, a scoring rubric with sample-size math and a one-week live-call plan. It is written for teams that built the agent themselves, often as a law firm AI receptionist inside practice-management software used by many firms. Each firm has its own practice areas, conflict rules and script. That is why one average hides the failures.

The build and rules playbook is AI Legal Intake: The Complete QA Playbook for 2026. This post is narrower. It tells you whether your AI intake for law firms is accurate today, and how to prove it. It is not legal advice; check your own state's rules with counsel.

What accuracy means on an intake call

A legal intake call is the front door to a workflow. The caller speaks. The agent captures facts. The agent then writes to the case management system: a contact, a matter, a deadline, an assignment. An intake specialist reads that record later and trusts it.

Legal intake is stricter than most phone work. The record starts clocks, such as a statute of limitations. It feeds a conflict check. And the caller is a prospective client, so what they say is confidential even if the firm never takes the case.

So accuracy has three layers. Each can fail alone.

1. Heard. Did speech recognition get the names, dates, places and numbers right?

2. Decided. Did the agent pick the right matter type, urgency and next step, and stay inside what it may say?

3. Written. Did the right values land in the right fields on the right record, once?

A transcript audit sees only layer one. This audit checks all three, from call audio to case file. For the general method, see Voice Agent Accuracy Audit: What It Measures.

The 12 accuracy checks

Each check has a source of truth. If a check has none, that is itself a finding.

Group the checks by what a miss costs. Gate checks have zero tolerance: one confirmed failure with real impact blocks the next release. Core checks have an error-rate target. The Watch check you track for drift.

Names and roles: checks 1 and 2

A conflict check depends on spelling. A caller says "Whitfield" and the transcript says "Widfield." The search runs against the firm's client list and finds nothing. The firm then takes a case against its own client.

In legal intake AI, the first call is where confidentiality starts. Under ABA Model Rule 1.18, a person who consults a lawyer about forming a client relationship is a prospective client. The lawyer may not use or reveal what they learned, and may be barred from representing someone adverse in the same or a related matter. Your agent hears that information first. So a wrong adverse party name is not a typo. It is a failed screen.

The audit compares every captured name to what the caller said and spelled: the adverse party, employer, insurer and witnesses. Role errors count too. "My landlord sued me" and "I am the landlord" give opposite conflict results. Spoken-entity failure is covered in Evaluating STT for Names and Entities.

Dates, place and matter: checks 3, 4 and 5

Dates are the highest-leverage field. The incident date, the date of discovery, the date of a denial letter and the court date all start or stop something. Callers say "last Tuesday" or "about three months ago." Speech recognition confuses thirteen and thirty. Agents resolve relative dates against the wrong "today" or time zone, and a conflict of interest search needs the same care with names.

Two examples show how much the place matters. New York General Municipal Law 50-e(1)(a) requires a notice of claim "within ninety days after the claim arises" in the cases it covers. California's Code of Civil Procedure 335.1 gives two years for personal injury actions. A wrong state or a wrong date changes which clock applies. Rules vary by claim and state. Your audit checks the captured facts; attorneys decide the deadline.

Matter type (check 5) is a Watch check. A mislabel sends the file to the wrong team, but a human usually catches it. Track it for drift.

Prior lawyer, urgency and qualification: checks 6, 7 and 8

Check 6 asks whether the caller already has a lawyer or is already a client. Agents often miss the quiet version: "my lawyer isn't returning my calls."

Check 7 is triage. Some calls are not intake: a hearing tomorrow, a detention, a safety threat. The agent must route these before it fills any form. Test each written rule with spoken phrasings. Callers rarely say "emergency."

Check 8 is the qualification call. A false decline turns away a case the firm wanted. A false accept books a case outside its practice or territory. Count both against the attorney's final decision. The agent should decline gently and never say the case has no merit; see check 9.

Advice, disclosure, contact and write-back: checks 9 to 12

Check 9 covers what a legal intake AI says. It must not give legal advice, which risks unauthorized practice of law, predict outcomes, or imply an attorney-client relationship exists. Test it with callers who push: "Do I have a case?" "Should I sign this?" "How much is it worth?"

Check 10 covers disclosure and recording. ABA Formal Opinion 512, issued July 29, 2024, says lawyers must make reasonable efforts to prevent disclosure of client information. It says informed consent is needed before putting client information into self-learning generative AI tools. It also calls for supervising vendor use. Federal law, 18 U.S.C. 2511(2)(d), allows recording when one party consents, but state rules differ. Check that your required notice is spoken on every call, before any detail is taken.

Check 11 is contact data: phone digits, email spelling and language. A wrong digit means no callback. A caller who switches to Spanish mid-call should still get a correct record. See Evaluating Multilingual Accuracy for Voice Agents.

Check 12 is the diff: a person's labels against what the case management system holds. Errors here are often not speech errors: wrong field mappings, duplicate contacts, truncated notes, timezone shifts. Evaluating Tool-Argument Accuracy from Speech shows how to test the arguments.

Why speech errors become legal errors

Speech recognition does not fail evenly. The study by Koenecke et al. tested five commercial systems. It found an average word error rate of 0.35 for Black speakers and 0.19 for white speakers. A firm with a diverse caller base will see uneven rates by group. Slice your audit by accent and language, as in Voice Agent Accent Robustness.

Legal calls make this worse.

Surnames, company names and court names are rare words. Spelled-out letters are hard to capture. Callers under stress rush, restart and talk over the agent.

An error on a casual word is harmless. An error on a name or date is permanent, because later steps trust it. Nothing in the record says "this was a guess."

Worked math: one misheard date

This illustrative example uses a 90-day notice period. It states no real case's deadline.

The caller says the incident was on March 13, 2026. Speech recognition hears "March 30." The system writes March 30 and computes 90 days.

The firm believes it has 17 more days than it does. The error is silent if the agent read the wrong date back and the caller, half listening, said "yes."

The reverse error shortens the clock, which is safer. Both are failures, and only the audio tells you which one you got. A handoff note should say "date read back and confirmed" or "date unconfirmed."

The scoring rubric

This turns AI legal intake accuracy into numbers you can compare week to week.

Score every sampled call on every check it exercises. Mark each check pass, fail or not applicable. Do not average across checks. Report per check.

Each check gets a color from its error interval, not its raw rate.

Targets are yours to set. A starting point, which is an assumption and not a standard: gate checks target 1% error, core checks 5%, the watch check 10%.

Gate checks add one rule. A single confirmed gate failure with real impact is a release blocker, whatever the interval says. A missed urgent call is not a statistic.

Worked math: what a clean week proves

Say you score 80 calls and find no errors on a check. That does not show the rate is zero. The 95% Wilson upper bound is about 4.6%. With 40 calls it is 8.8%. With 300 calls it is 1.3%. This is an illustrative calculation.

Here is an example audit. The numbers are invented.

Three lessons sit in this table. First, a gate check with a 1% target almost never turns green from sampling alone. You would need hundreds of observations. So for gate checks, use two controls: a zero-tolerance rule on every confirmed failure, and automated scoring on every call, as argued in Score Every Call, Not a Sample. Second, urgent triage has 25 observations because most calls are not urgent. Aim for 30 or more per check by pulling urgent-sounding calls on purpose; see Sampling Live Calls for Voice Agents.

Third, failures compound. If 12 checks each passed 98% of the time, independently, a call applying all 12 would pass them all only about 78% of the time. Repeats are worse: the tau-bench authors report pass^8 under 25% for strong agents in their retail domain (arXiv 2406.12045). Different task, same lesson. Rerun failing scenarios several times.

Here is the scoring helper. It is simplified and runs as written.

from math import sqrt

def wilson(k, n, z=1.96):
    """95% Wilson interval for k errors in n observations."""
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, c - h), min(1.0, c + h)

def color(k, n, target):
    lo, hi = wilson(k, n)
    if lo > target:
        return "red"
    if hi < target:
        return "green"
    return "amber"

print(color(3, 80, 0.01))    # red: conflict names
print(color(0, 150, 0.05))   # green: jurisdiction

A write-back diff you can run

Check 12 needs a diff between labelled truth and the case file. This simplified helper checks four fields for ai legal intake accuracy. Extend it to your schema.

import re
from difflib import SequenceMatcher

def digits(s):
    d = re.sub(r"\D", "", s or "")
    return d[-10:]  # compare US 10-digit numbers

def norm(s):
    return re.sub(r"[^a-z0-9 ]", "", (s or "").lower()).strip()

def same_name(a, b, cutoff=0.97):
    return SequenceMatcher(None, norm(a), norm(b)).ratio() >= cutoff

def diff(truth, record):
    fails = []
    if truth["incident_date"] != record.get("incident_date"):
        fails.append("incident_date")
    if digits(truth["phone"]) != digits(record.get("phone")):
        fails.append("phone")
    if not same_name(truth["adverse_party"], record.get("adverse_party")):
        fails.append("adverse_party")
    if truth["state"].upper() != (record.get("state") or "").upper():
        fails.append("state")
    return fails

truth = {"incident_date": "2026-03-13", "phone": "(555) 010-4821",
         "adverse_party": "Whitfield Logistics", "state": "NY"}
record = {"incident_date": "2026-03-30", "phone": "555-010-4821",
          "adverse_party": "Widfield Logistics", "state": "NY"}
print(diff(truth, record))  # ['incident_date', 'adverse_party']

The 0.97 cutoff is deliberately strict, because a near-match on a name is exactly the bug.

How to run a one-week AI legal intake audit on your live calls

This plan assumes you record calls and can export case records. Time figures are planning assumptions.

1. Define truth for each check (Monday). For each of the 12 checks, write the source of truth from the table above. Write the target error rate and tier. A check with no source of truth goes on a gap list.

2. Pull and stratify the sample (Monday). Export last week's calls with their records. Group them by practice area and call type: new matter, existing client, wrong number, urgent, after-hours. Aim for at least 80 calls with every group present. Keep the audit set confidential.

3. Add the hard cases (Tuesday). Include calls with noise, accents, interruptions, spelled names and relative dates. Tag them.

4. Label from the audio (Tuesday and Wednesday). A person listens to each recording, not just the transcript. They write down what the caller said for each field and mark each check pass, fail or not applicable. Budget about ten minutes per call. A second person re-labels 10.

5. Diff against the record (Wednesday). Compare labelled truth to the case file. Record the first point of failure: heard, decided or written.

6. Score and color (Thursday). Count errors per check. Compute intervals. Assign green, amber or red.

7. Turn failures into tests (Friday). Each confirmed failure becomes a replayable scenario: the caller's words, accent, noise, and the expected record. Rerun each at least three times.

8. Set the next audit trigger (Friday). Re-run the scorecard after any prompt, model, speech-recognition or firm-config change. See Ship Voice Agent Prompt Changes Safely. Keep a weekly 30-call spot check.

Step 4 is slow by hand when many firms share one agent. Tooling helps here. Evalgent generates test scenarios from your agent's prompt and flows, and runs a call simulator over phone, SIP or WebRTC with caller profiles for accent, pace, interruption and background noise. It evaluates from the audio, not only the transcript, with evidence and human review of verdicts. For live traffic, you push call recordings. They are transcribed, mapped to scenarios and judged, and Evalgent flags PII leakage, tool-call failures and policy failures with fix recommendations. Evalgent works with teams to set up evals for a first agent; it is sales-led.

Reading the results: three patterns

After a first audit, failures usually fall into three patterns.

Heard wrong, written wrong. Speech recognition produced the wrong entity and the agent trusted it. Add a read-back for names and dates and a spelling capture for adverse parties. More prompt text rarely helps.

Heard right, decided wrong. The transcript is right but urgency, matter type or the acceptance call is wrong. Test the firm's configuration first. A stale practice-area list looks like an AI error.

Decided right, written wrong. The record disagrees with the call. This is an integration bug: wrong field, duplicate matter, timezone. Sorting failures this way shows who owns the fix: speech, prompt, config or integration.

Where to go deeper

Use this map after your first scorecard. For live monitoring, see How to Monitor Voice Agents in Production.

For outbound follow-up calls to leads, remember that the FCC's February 2024 declaratory ruling confirmed that AI-generated human voices count as an "artificial or prerecorded voice" under the TCPA, so prior express consent rules apply. The consent rules are in 47 CFR 64.1200. Have counsel review outbound flows.

Frequently asked questions

How accurate is AI legal intake?

It depends on the check. Summaries often look fine, while names, dates and write-back fail quietly. No honest single AI legal intake accuracy number exists. Score each of the 12 checks against the case file, with a confidence interval. Zero errors in 80 calls still allows a true error rate near 4.6%.

What errors does AI make most often at law firm intake?

Common errors are misheard names and dates, wrong party roles, missed prior-lawyer mentions and fields written to the wrong place. Your own audit will show your mix. A law firm AI receptionist and an intake-only agent will differ. Sort each failure into heard, decided or written to see who owns the fix.

How do I audit my law firm's AI intake agent?

Define a source of truth for each of the 12 checks. Pull at least 80 recorded calls, stratified by practice area. A person labels each call from the audio. Diff the labels against the case file. Score each check with an interval, then turn every failure into a replayable test.

How many intake calls should I review to audit AI?

At least 80 per week as a start, with 30 or more observations per check. Rare checks, like urgent triage, need targeted sampling. For gate checks with a 1% target, sampling alone rarely proves safety, so also score every call automatically. These figures are planning assumptions.

Can AI legal intake give legal advice?

It should not. The agent may collect facts and explain next steps, but it must not predict outcomes, say a case has merit, or imply a lawyer-client relationship exists. Test this with callers who push for an opinion. Treat one confirmed breach as a release blocker. This is not legal advice; check your state's rules.

Does AI mishear names and dates on intake calls?

Yes, it can. Rare surnames, company names and spelled-out letters are hard to capture, and error rates vary by speaker group. Research across five commercial systems found word error rates of 35% versus 19% between two speaker groups. Read-backs and spelling capture reduce the risk, but only an audit shows your rate.

How does AI legal intake handle conflict checks and prospective clients?

A good agent captures the adverse party, employer and insurer names with spelling, then runs the conflict check before taking detailed facts. Under ABA Model Rule 1.18, information from a prospective client is protected. Audit check 2 by comparing captured names to the caller's spoken and spelled names.

How often should I re-audit AI legal intake after a change?

Re-run the scorecard after any change to the prompt, model, speech recognition, telephony or a firm's configuration. Between changes, spot-check about 30 calls weekly. Run a full audit at least quarterly.

The bottom line

AI legal intake accuracy is a set of 12 checks scored against the case file with intervals, not one percentage from a transcript. Run the one-week audit on live calls, treat every gate failure as a release blocker, and re-run it after every change.