AI Phone Screening: Is It Scoring Candidates Fairly?

If your team built its own AI candidate screening flow, nobody gave it a performance review. It talks to hundreds of candidates a week. Each call ends in a score, a stage change or a rejection email. Recruiters see the output and rarely hear the audio.

This post is the audit: twelve checks, a scoring rubric and a one-week plan for your own calls. It also covers fairness, using only the law as it stands in October 2026. Our deeper guides handle the details. Each is linked in the cluster map near the end.

Why AI phone screening accuracy differs from receptionist accuracy

A receptionist agent books an appointment. If it errs, the customer notices on the day. A screening agent decides who moves on. If it errs, the candidate never learns why.

Three things make it harder.

1. The error is invisible to the person it harms. A rejected candidate does not call back to say "you misheard my license class."

2. The score is a judgment, not a fact. A booked slot is right or wrong. A 4 out of 10 is only right relative to a rubric and a human's score.

3. Fairness is measured across groups, not calls. One call can be perfect and the system still unfair.

The workflow is a chain: capture the answer, extract a field, score it, write the result to the applicant tracking system. A speech error at the start flows silently through every later step.

That trace is the mental model for the whole audit. At each layer you compare the output to the layer above it. The first layer where they disagree is where the bug lives.

This post assumes the agent is live and you own it. For the buying side, see our guide to evaluating a recruiting screening voice agent vendor.

What the research says about hearing candidates correctly

Everything downstream depends on the transcript. The most-cited study of speech recognition fairness is Koenecke et al. in PNAS (2020). The team tested speech recognizers from Amazon, Apple, Google, IBM and Microsoft. The sample was 2,141 matched snippets per group, from 73 Black speakers and 42 white speakers. Average word error rate was 0.35 for Black speakers and 0.19 for white speakers. The authors traced the gap mainly to the acoustic models. They wrote that the likely cause was insufficient audio data from Black speakers in training.

Three cautions apply before you quote that paper.

• The systems were tested in 2019, and models have changed. Treat the paper as evidence of a failure mode, not your vendor's score.
• The audio was long-form interview speech, not noisy 8 kHz phone calls.
• Word error rate counts every word equally. Screening cares about a few: a license class, a shift, a pay figure, a yes or no.

The practical lesson is to measure transcript accuracy on the fields your rubric uses, split by caller group, on your own call audio. Our guides to accent robustness and disfluency robustness show how to build those test sets.

The twelve AI phone screening accuracy checks

Here is the scorecard in one view. Each card names a check, what you measure and a proposed starting threshold. The thresholds are defaults I suggest for a first audit. They are not legal standards. Tighten them for high-stakes roles.

Checks 1 to 3: hearing and extraction

1. Hear it right. Compare the transcript to a human transcript for the answer-critical words only. Report the error rate per caller group. The gap between groups matters more than the absolute number.

2. Extract the fact. Compare each stored field (license class, years of experience, start date, pay expectation) to what a human reviewer heard. Fields fail quietly. "Thirteen" becomes "thirty." "Fifteen" becomes "fifty."

3. Knockout rules. Many screeners apply must-have rules: minimum age, a license, shift availability. Check that each rule fires only when the evidence supports it. One wrong knockout means a qualified person never reaches a recruiter. Count misfires against human review.

Checks 4 and 5: scoring quality

4. Score repeatability. AI interview scoring is only useful if it is stable. Replay the same call audio through the scoring step five times. If scores wander by two or more points on a ten-point rubric, the score is noise. Pin the model version and temperature, and re-run the check whenever either changes.

5. Score vs humans. Have two trained reviewers score a sample blind from the audio. Measure inter-rater agreement between the two humans first. If your reviewers agree only 70% of the time, an AI that agrees with them 85% of the time is already as consistent as your people. Then measure how often the AI lands within one point of the human average.

Checks 6 to 8: fairness

6. Group parity. Compute the impact ratio: each group's advance rate divided by the highest group's rate. The next section shows the math.

7. Off-limits topics. The agent should not ask about, or infer, protected traits such as age, pregnancy, disability, religion or national origin. Search the transcripts for the agent's own questions. Then check whether any free-text notes mention them.

8. Delivery bias. Read your rubric. If any item scores fluency, confidence, tone, pace or "communication style," you are scoring how a person sounds. That correlates with accent, disability and age, and it is a common source of AI screening bias. Strip those items from the automated score or move them to a human step.

Checks 9 to 12: the call and the record

9. Pauses and cut-offs. Count how often the agent talked over a candidate or moved on after a long pause. Candidates who pause to think, or speak a second language, get cut off more. Our guide to escalation and handoffs covers what should happen when the call goes sideways.

10. ATS write-back. Reconcile every field in the applicant tracking system against the call. Stage, disposition code, interview slot and notes all count. The interview scheduling errors guide shows how multi-location setups break here.

11. Candidate statements. The agent says things too: pay ranges, shift times, locations, benefits. Check each claim against the live job record. Staffing teams should read testing skills and pay rates.

12. Human review path. Confirm that every rejection can be pulled up with its audio, transcript, fields and score. Confirm a candidate can reach a person. If a decision cannot be reviewed, it cannot be audited.

The AI phone screening accuracy scoring rubric

Score each check 0, 1 or 2 after the audit. Total out of 24.

Read the total in three bands:

• 20 to 24: Run quarterly audits and monitor continuously.
• 13 to 19: Fix the 0s first, then re-audit in 30 days.
• 12 or below: Put a human in every rejection decision until the 0s are gone.

Two rules override the total. Any 0 on checks 3, 6, 7 or 12 is a stop-and-fix item no matter how well the other checks score. And a check you never measured is a 0, not a pass.

How the fairness math works

The first thing most teams get wrong is that fairness is a rate comparison, not a feeling. Federal selection guidelines give the standard test. Under 29 CFR 1607.4(D), a selection rate for any group that is less than four-fifths (80%) of the highest group's rate is generally treated as evidence of adverse impact. The same paragraph adds two limits. Smaller differences can still count if they are statistically and practically significant. And larger differences may not indicate adverse impact when they rest on small numbers.

Here is a worked example. The numbers are illustrative, not from any real system.

The impact ratio for B is 40 divided by 60, which is 0.67. That is below 0.80. A Fisher exact test on those counts gives a p-value near 0.0001, so the gap is not chance. This is a flag, not a verdict. It does not say why B advances less. It says you must find out.

Now keep the same rates but screen only 30 per group: A advances 18 and B advances 12. The ratio is still 0.67. The Fisher p-value is about 0.20, so the gap could be noise. The 95% Wilson intervals are 42% to 75% for A and 25% to 58% for B. They overlap heavily.

That is why I suggest 60 or more per group as a floor. This shows how the uncertainty on a 90% agreement rate shrinks as the sample grows (95% Wilson interval, computed for this post):

At 60 calls, a 90% result could really be 80%. That is fine for a first audit, not for a final defense. This code computes both numbers (simplified, pure Python):

import math

def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

def impact_ratios(advanced, screened):
    rates = {g: advanced[g] / screened[g] for g in screened}
    top = max(rates.values())
    return {g: (rates[g], rates[g] / top) for g in rates}

print(impact_ratios({"A": 120, "B": 80}, {"A": 200, "B": 200}))
# {'A': (0.6, 1.0), 'B': (0.4, 0.667)}
print(wilson(54, 60))  # about (0.80, 0.95)

A caution on "groups." Most voice screeners do not collect race, sex or age. You often cannot compute impact ratios on real candidates without a voluntary self-identification step, which the ATS may already hold. Where it does not, audit with a proxy you can defend: scripted test callers with known accent, speech rate and background noise. That tests whether the system treats the same answer the same way. It is not a substitute for the real-population analysis a regulator or court would expect.

Is it legal? What applies in October 2026

This section is a summary of public sources and is not legal advice. Ask counsel about your roles and states. Here is what I could verify.

• Federal guidance was withdrawn, the statutes were not. The EEOC removed its AI employment guidance from its website in January 2025, according to a National Law Review analysis. That analysis notes Title VII and the Uniform Guidelines (29 CFR Part 1607) remain in force. The four-fifths test above comes from those guidelines.
• Colorado replaced its AI Act. Governor Polis signed SB 26-189 on May 14, 2026. It repeals and reenacts SB 24-205 with a narrower, procedural framework effective January 1, 2027, per Epstein Becker Green. For employers it requires notice before a covered automated decision tool materially influences a decision, and a plain-language explanation within 30 days of an adverse outcome. It also requires human review and reconsideration where commercially reasonable, and three years of compliance records. The attorney general enforces it.
• Illinois has a statute but no rules yet. HB 3773 took effect January 1, 2026. It requires notice when AI is used in covered employment decisions and bars discriminatory use. The state's human rights department said on June 2, 2026 that it was postponing its rulemaking, including the June 10 hearing. The statute still applies. What notice must look like is unclear.
• New York City already audits. Under Local Law 144, an automated employment decision tool needs a bias audit within one year of use, a public summary of results, and candidate notice at least 10 business days before use, per the city's DCWP page. Enforcement began July 5, 2023.
• Litigation is testing vendor liability. In Mobley v. Workday, disparate impact claims survived a motion to dismiss in July 2024. The court granted preliminary collective certification in May 2025. The case summary I could open ran through December 2025 and showed the case ongoing with no merits ruling. Check the docket for anything newer.

Whatever the rulebook says this year, you will be asked three things: who was scored, how accurate was it, and can a human review it. Checks 5, 6 and 12 answer them. For a bias-focused test design, see our AI interviewer bias testing guide.

How to run a one-week accuracy audit on live calls

Run this on one role family at a time, such as warehouse hourly roles. Mixed rubrics muddy the numbers.

1. Day 1: pull and stratify a sample. Export at least 60 completed calls per group you can identify, plus 20 calls the agent rejected early. Record the rubric version, model version and prompt version for each call. If you cannot recover those, note that as a finding.

2. Day 2: build human gold labels. Two trained reviewers listen to the audio, not the transcript, and score each call blind to the AI output. They fill the same fields the agent filled. Measure how often the two humans agree before you compare anyone to the AI.

3. Day 3: diff transcript, fields and scores. Compare each layer to gold using the trace above. Log every disagreement with its first-failing layer. This yields checks 1, 2, 3, 4 and 5.

4. Day 4: run the parity math. Compute advance rates and impact ratios by group, with Wilson intervals and Fisher p-values. Where group labels do not exist, use the scripted-caller approach from the previous section.

5. Day 5: stress-replay. Call the agent, or replay audio, with the same answers delivered with different accents, speech rates, background noise and pauses. Hold the content constant and only vary delivery. Evalgent's caller profiles cover accent, pace, interruption level, response delay and background noise, and its evaluation reads the audio, not only the transcript.

6. Day 6: reconcile the ATS. For every sampled call, check stage, disposition, slot and notes in the ATS against the call. Log mismatches by field. This is check 10 and often finds the most embarrassing bugs.

7. Day 7: score and decide. Fill the twelve-check card. Choose fix, freeze or escalate for each 0. Freeze means a human approves every rejection until the fix ships.

The sample sheet, gold labels and parity table are your evidence if anyone asks. Our guide to a golden dataset for voice eval shows how to turn the gold labels into a reusable regression set. For the human side, see human-in-the-loop evaluation.

From audit to monitoring

A one-week audit is a photograph. The screener changes whenever someone edits the prompt, swaps a model or adds a role. Three things keep AI phone screening accuracy from drifting after the audit.

• Regression tests before every change. Replay your gold set against each new prompt or model version. Block the change if a check drops a band. The pattern is in our guide to shipping prompt changes safely.
• Production monitoring. Score live calls on the fields and flags that matter. Monitoring voice agents in production explains the signals. Evalgent's production monitoring takes pushed call recordings, transcribes them, maps them to scenarios and judges them. It flags PII leakage, tool-call failures, latency spikes and policy failures. Human reviewers can check the verdicts.
• A review queue for rejections. Sample a fixed share of automated rejections each week and have a person listen. Track how often the person disagrees. That rate is your live accuracy estimate, and it moves before complaints arrive.

Also watch hallucination rate when the agent answers pay and benefits questions. Watch PII handling too, because candidates read out addresses and license numbers.

Where to go deeper: the hiring cluster map

Each of these posts covers one slice in depth. Use this map to pick the next one.

Frequently asked questions

How accurate is AI phone screening?

There is no single number. AI phone screening accuracy depends on the speech recognizer, your rubric and your callers. Measure it as agreement with two trained human reviewers on your own calls, in three layers: transcript, extracted fields and final score. Report each layer by caller group. A strong average can hide a weak group.

Can AI phone screening be biased against accents?

Yes, it can. A 2020 PNAS study found five commercial speech recognizers averaged 0.35 word error rate for Black speakers against 0.19 for white speakers. Phone screening adds compression and noise. Test with the same scripted answers delivered in different accents, then compare field accuracy and scores across them.

How do you audit an AI phone screener?

Pull a stratified sample of live calls, have two humans score the audio blind, and compare transcript, fields and scores to that gold set. Compute impact ratios by group, reconcile ATS records, and replay calls with varied accents and noise. Score twelve checks from 0 to 2. Fix any zero on knockouts, parity, off-limits topics or human review first.

What is the four-fifths rule for AI hiring tools?

The four-fifths rule comes from the federal Uniform Guidelines at 29 CFR 1607.4(D). A group's selection rate below 80% of the highest group's rate is generally treated as evidence of adverse impact. It is a screening test, not a verdict. Small samples can mislead, and smaller gaps can still matter.

Do you need to tell candidates an AI is screening them?

It depends on where the candidates are. New York City requires candidate notice at least 10 business days before an automated employment decision tool is used. Illinois HB 3773 requires notice of AI use in covered decisions. Colorado's SB 26-189 adds notice from January 1, 2027. Check your states with counsel, and disclose the AI on the call regardless.

How many calls do you need to audit an AI phone screener?

Plan on at least 60 calls per group for a first audit. At 60 calls, a 90% agreement result has a 95% Wilson interval of roughly 80% to 95%. At 200 calls it narrows to about 85% to 93%. Small samples can show a large impact-ratio gap that is still consistent with chance.

Does the EEOC still regulate AI hiring tools?

The EEOC removed its AI hiring guidance from its website in January 2025. The underlying law did not change. Title VII and the Uniform Guidelines at 29 CFR Part 1607 still apply to selection procedures, including software. Employers should still check adverse impact, and state laws now carry much of the AI-specific detail.

Who is liable when an AI screener rejects a qualified candidate?

It is unsettled and varies by state. In Mobley v. Workday, a federal court let disparate impact claims proceed against a screening software vendor and granted preliminary collective certification in 2025. Colorado's SB 26-189 can reach both developers and deployers. Employers using a tool should assume they carry responsibility and keep records.

The bottom line

AI phone screening accuracy is not one number; it is twelve checks across hearing, scoring, fairness and the record, each measured by caller group. Run the one-week audit on your live calls, keep a human in every rejection until the zeros are fixed, and re-audit whenever the prompt or model changes.