AI Phone Screening and Interviews: A Step-by-Step QA Guide

Last verified October 16, 2026, against the NYC DCWP AEDT FAQ, Illinois Public Act 103-0804, Colorado SB 26-189, the EEOC's June 2026 National Enforcement Plan and 47 CFR 64.1200. Laws in this area are changing fast. Check them again before you rely on them.

This guide is for the product team at an ATS, recruiting-software or staffing-platform company that built its own AI phone screening feature. Your screener runs for hundreds of employer customers. Each one has its own knock-out questions, pay bands, shifts and stage rules. Every prompt change ships to all of them at once. Testing is a recruiter-turned-PM making test calls from a cell phone.

If you are buying a screener instead of building one, read our guide to evaluating a recruiting screening voice agent vendor. This post is the builder's view. It covers what each call writes, how a misheard word flips a candidate's outcome, how to test per-employer config, and which 2026 rules your tests must prove.

How AI phone screening works inside an ATS

AI phone screening is a voice agent that asks a candidate a fixed set of questions by phone. It then writes the answers and an outcome back into the applicant tracking system. Each AI phone screen is the front door. The ATS record is the product.

Automated phone screening has a measured upside. Jabarian and Henkel ran a natural field experiment on 70,884 applications for entry-level customer service jobs. Applicants interviewed by an AI voice agent received offers at 9.73% versus 8.70% with human recruiters. Job starts rose 18% and one-month retention rose 17%. AI interviews also covered more topics on average (6.78 vs 5.53). Applicants did rate them as less natural, so candidate experience still needs testing. That is a strong case for the feature. It is also a reason your customers will push volume through it fast.

Here is the agentic flow, step by step.

1. Call. Inbound (candidate calls the job line) or outbound (the screener dials an applicant). The agent identifies itself, discloses recording and AI use, and confirms identity.

2. Knock-out answers plus transcript. The agent asks the employer's knock-out questions. An LLM maps each spoken answer to a typed field: `weekend_ok`, `cna_license_active`, `pay_min`, `authorized_to_work`.

3. Score and disposition. Rules check the knock-outs. An AI interviewer may score open questions against a rubric. The system sets a disposition: advance, reject, or hold for review.

4. ATS stage change. The integration moves the application to a new stage. It may trigger a rejection email or a recruiter task.

5. Scheduling invite. For advanced candidates, AI interview scheduling books a slot on a recruiter or site manager calendar and texts a confirmation.

Knock-out question: a yes/no or threshold question that removes a candidate when the answer fails a hard requirement, such as "Are you 18 or older?" or "Do you hold an active CNA license in this state?"

Every step after the call trusts the step before it. Nobody listens to the audio again. A transcript review asks "did the agent sound right?" The business asks "did the ATS end up right?" Those are different questions. Our post on transcript versus audio evaluation explains why the first one misses so much.

What each hiring segment needs from the call

How a misheard answer silently rejects or advances the wrong candidate

A screening error does not crash anything. It writes a valid-looking field, and the ATS acts on it. The candidate gets a polite "we've decided not to move forward" email. Nobody on your team ever hears the call.

These are the five patterns that cause most of the damage.

Availability negations. The candidate says "I can't do weekends." Phone audio drops the final "t." The model hears "I can do weekends" and sets `weekend_ok = true`. The candidate advances to a Saturday shift interview they will never attend. The reverse also happens: "I can't see why not" turns into a refusal. Shift availability is a core knock-out in high-volume hiring. It is also the one most exposed to a single lost consonant.

Certification and license names. "CNA" and "CMA" differ by one consonant. "BLS" and "ACLS" are both life-support certifications with different requirements. A nurse says "my license lapsed in May but the renewal is in." The model needs a status, not a yes. Healthcare staffing makes this a patient-safety issue, not just a funnel issue.

Pay expectation numbers. "Fifteen" and "fifty" are a classic speech-to-text confusion. A candidate saying "eighteen to twenty" can become `pay_min = 80` on a sloppy parser. If the req's top rate is $22, the candidate fails the pay knock-out. They were inside the band. Spoken numbers need the same care as order numbers. Our guide to STT entity accuracy covers digit and number tests.

Work authorization question wording. The safe wording asks whether the candidate is authorized to work in the US and whether they will need sponsorship. Asking about citizenship or country of origin creates risk. Federal law, 8 U.S.C. 1324b(a)(1), bars hiring discrimination based on national origin, and for protected individuals, citizenship status. Two failures matter here. First, the LLM paraphrases a compliant question into a non-compliant one ("Are you a US citizen?"). Second, a "no" to "will you need sponsorship?" is stored as a "no" to "are you authorized?" The candidate is rejected for the opposite of what they said.

Truncated answers. Some candidates pause, restart or stutter. The agent's endpointing decides they finished and moves on. Lea et al. studied 91 people who stutter using speech recognition. Tuning cut truncated utterances by 79.1% and lowered word error rate from 25.4% to 9.9%. That tells you how much default settings clip disfluent speech. A clipped answer can turn into "no answer," then into a failed knock-out. Our disfluency robustness guide shows how to test it.

Why do these happen? Phone calls arrive as 8 kHz narrowband audio, often μ-law encoded. That strips high-frequency sounds like the "t" in "can't" and the "s" in "CMA's." Then the LLM fills a typed field from a noisy transcript. It is trained to be helpful, so it rarely writes "unclear." It picks the most likely value. Our post on tool-argument accuracy from speech walks through that last hop in detail.

The error asymmetry

Not all errors cost the same. Rank them before you set pass bars.

• False reject. A qualified candidate is removed. You lose a hire, and nobody notices. If the errors cluster by accent, dialect or disability, it becomes adverse impact.
• False advance. An unqualified candidate is advanced. A recruiter wastes a slot, or a site manager meets someone with no license. This one gets noticed, so teams tune for it. That tuning often pushes false rejects up.
• Wrong record. The answer is right, but it lands on the wrong application or requisition. Staffing agencies with many open reqs see this most.

Worked example (illustrative assumptions). Say your platform runs 50,000 screens a month across all employers. Each screen asks five knock-out questions. Assume each field is misread 1% of the time, independently. The chance a call has at least one wrong field is 1 − 0.99⁵ ≈ 4.9%. That is about 2,450 calls a month with a corrupted field. If half of those flip the disposition, about 1,225 candidates a month get the wrong outcome. None of them trigger an error log. Swap in your own measured field error rate. The shape of the math holds.

Per-employer configuration is the real test surface

Your prompt is shared. Your customers' configs are not. On a multi-tenant platform, the riskiest failures hide in config combinations nobody tested, not in the base prompt.

A typical employer config includes:

• Knock-out questions, their wording, and pass values per requisition
• Pay bands per req, sometimes per location
• Shift patterns and the words the employer uses for them ("swing," "PM," "seconds")
• Languages offered on the line
• Scoring rubric and cutoff for AI interviews
• Stage mapping: which disposition moves to which ATS stage
• Auto-reject on or off, and who reviews holds
• Calendar sources and booking windows for AI interview scheduling
• Disclosure text, recording behavior and state-specific notices

The combinations grow fast. Five knock-out types, three languages, two auto-reject modes, four stage maps and three calendar setups give 5 × 3 × 2 × 4 × 3 = 360 combinations. You cannot hand-test that on every release. You can cover every pair of values with a much smaller pairwise set, often a few dozen rows. Then add the exact config of every employer that changed this week.

Our go-live checklist for each new customer's AI phone agent covers the onboarding side. The code below shows the testing side. It turns one employer's config into scripted test cases with expected ATS writes.

# Simplified: generate knock-out test cases from one employer's screener config.
# The run_call and ats_read functions are placeholders for your own harness.
from dataclasses import dataclass

@dataclass
class Case:
    employer_id: str
    req_id: str
    question: str
    spoken_answer: str       # what the simulated candidate says
    expected_field: dict     # what the ATS should store
    expected_stage: str      # where the application should land

NEGATION_VARIANTS = {
    True:  ["yes, weekends are fine", "I can do weekends", "Saturdays work for me"],
    False: ["I can't do weekends", "no weekends, sorry", "I can't, I have my kids on weekends"],
}

def weekend_cases(cfg: dict) -> list[Case]:
    cases = []
    for req in cfg["reqs"]:
        if "weekend_ok" not in req["knockouts"]:
            continue
        must = req["knockouts"]["weekend_ok"]           # True if weekends are required
        for answer, phrases in NEGATION_VARIANTS.items():
            stage = req["stage_map"]["advance" if answer or not must else "reject"]
            for p in phrases:
                cases.append(Case(cfg["employer_id"], req["id"], "weekend_ok", p,
                                  {"weekend_ok": answer}, stage))
    return cases

def pay_cases(cfg: dict) -> list[Case]:
    cases = []
    for req in cfg["reqs"]:
        top = req["pay_max"]                              # e.g. 22.00
        for spoken, value in [("fifteen an hour", 15), ("fifty an hour", 50),
                              ("eighteen to twenty", 18), (f"{int(top)} is my minimum", top)]:
            ok = value <= top
            stage = req["stage_map"]["advance" if ok else "reject"]
            cases.append(Case(cfg["employer_id"], req["id"], "pay_min", spoken,
                              {"pay_min": value}, stage))
    return cases

def check(case: Case, run_call, ats_read) -> list[str]:
    call = run_call(case)                    # phone call with a scripted caller profile
    app = ats_read(call.application_id)      # read back the record, not the transcript
    errors = []
    for k, v in case.expected_field.items():
        if app.fields.get(k) != v:
            errors.append(f"{case.req_id} {k}: expected {v}, got {app.fields.get(k)}")
    if app.stage != case.expected_stage:
        errors.append(f"{case.req_id} stage: expected {case.expected_stage}, got {app.stage}")
    return errors

Two design choices matter here. First, the assertion reads the ATS record, not the agent's summary. The summary can be right while the write is wrong. Second, the test cases come from config, so a new employer gets coverage the day they onboard. Add the same generators for licenses, work authorization wording and scheduling.

Scoring and AI interviews: test the judge, not just the agent

An AI interviewer scores open answers such as "Tell me about a time you handled an upset customer." A structured interview uses the same questions and rubric for every candidate. That consistency is the main fairness argument for automating it. It only holds if the score is stable.

LLM scorers have known biases. Zheng et al.'s study of LLM-as-a-judge (NeurIPS 2023) documented position bias, verbosity bias and self-enhancement bias. Verbosity bias is a hiring problem. It rewards candidates who talk longer, not better. Non-native speakers and people with speech disabilities often give shorter answers. Our post on LLM-as-a-judge limits in voice AI lists nine more failure modes.

Errors also stack. The scorer reads a transcript, and the transcript is not equally accurate for everyone. Koenecke et al. tested five commercial ASR systems on 19.8 hours of interviews. Average word error rate was 0.35 for Black speakers and 0.19 for white speakers. A rubric that scores "clear, specific examples" will punish whatever the transcript garbled. Names matter too. Wilson and Caliskan ran resume screening tests with three embedding models. White-associated names were favored in 85.1% of race tests. Black male names were never preferred over White male names. Your screener may not use embeddings, but the lesson is the same. Run name-swap tests.

Score stability near the cutoff

Run the same candidate audio through the scorer several times. The score will move. What matters is how often it crosses the cutoff.

Worked example (illustrative). The cutoff is 70. Candidate B's mean score is 72, with a run-to-run standard deviation of 4 points. The chance one run lands below 70 is P(Z < −0.5) ≈ 31%. So nearly one time in three, the same answers get rejected. Candidate A at 82 and candidate C at 60 are stable. The flip zone is roughly the cutoff ± 2 standard deviations, here 62 to 78.

Three fixes follow from that chart. Route scores inside the flip zone to human review instead of auto-reject. Lower variance with a tighter rubric, fixed temperature and per-criterion scoring. And track flip rate as a release metric, not just mean score.

Adverse impact math you can run on test data

The federal four-fifths rule in 29 CFR 1607.4(D) treats a selection rate below 80% of the highest group's rate as general evidence of adverse impact. NYC's bias audit uses a similar impact ratio. Here is the math.

• Group A passes the screen at 60%. Group B passes at 42%.
• Impact ratio = 0.42 ÷ 0.60 = 0.70. That is below 0.8.

The same rule warns about small numbers that are not statistically significant. To estimate a pass rate within ±5 points at 95% confidence, you need about n = 1.96² × 0.25 ÷ 0.05² ≈ 385 calls per group. For synthetic test runs, that is cheap. For real call data, it can take months per employer. Our sibling post on AI interviewer bias testing covers matched-pair designs, accent sets and score parity in depth. For the speech side, see accent robustness for voice agents.

Scheduling, candidate FAQs and reference checks

The screen is not the only flow on the line. Three others write to systems too.

AI interview scheduling. The agent offers slots and books one. Errors here cause no-shows, which employers blame on the candidate. Test relative dates ("next Tuesday" said on a Sunday), "the 14th" vs "the 4th," time zones for remote roles, and multi-site employers. Assert the calendar event, the ATS interview record and the confirmation text all match.

Candidate FAQs. Candidates ask about pay, benefits, parking and dress code. The agent must answer only from the req and the employer's approved FAQ. The common failure is a confident pay range the employer never posted. Test questions whose answer is missing from the data. The pass condition is "I'll have the recruiter confirm that," not a guess.

Reference checks. Staffing and healthcare platforms use outbound calls to referees. Test that the agent confirms it reached the right person and states the purpose. It should record only structured answers. Referees sometimes volunteer health or family details. The agent should not ask for them, and the notes should not store them.

The 2026 compliance map for AI recruiting calls

AI candidate screening by phone sits under employment law, telephone law and recording law at once. The table shows where each rule stands as of October 2026. This is not legal advice. Work with employment counsel on how each rule applies to your product and customers.

A few notes that change how you test.

The vendor is in the frame. In Mobley v. Workday, a federal court granted preliminary certification of a nationwide ADEA collective in May 2025. The claim targets screening software, not one employer. California's regulations also reach an employer's agents. If you sell the screener, assume your test records may be asked for.

Federal retreat is not federal repeal. The EEOC's own enforcement plan dropped disparate impact. Title VII still prohibits it in statute, and private plaintiffs can still sue. State laws are filling the gap. Keep the adverse impact numbers regardless.

Enforcement has been light, but scrutiny is rising. The New York State Comptroller's December 2025 audit found DCWP received only two AEDT complaints in two years. DCWP reviewed 32 companies and found one issue. The auditors found at least 17 instances of potential non-compliance in the same group.

Consent for recruiting calls. A screening call about a job the candidate applied for is usually informational, not telemarketing. Prior express consent from the application is the common basis. A call that also sells something, like a paid certification course, can change that. Test that the agent never drifts into promotion. Our outbound voice agent testing guide covers voicemail, answer rates and opt-outs.

How to test an AI phone screener before every release

These steps assume a few engineers and no QA team. Run them on every prompt, model, STT or config change. Our guide to shipping voice agent prompt changes safely covers versioning and canaries.

1. Write the field contract. List every field the call writes, its type, and its allowed values. Include `unclear` as a legal value for every knock-out. A screener with no way to say "unclear" will guess.

2. Generate cases from each employer's config. Use generators like the code above for every knock-out type. Cover negations, numbers, ranges, license names and work authorization wording. Add pairwise rows across languages, stage maps and auto-reject modes.

3. Run real phone calls with varied caller profiles. Use PSTN or SIP, not a text harness. Vary accent, speech pace, pauses, interruptions and background noise. Include hourly-worker settings: a car, a kitchen, a bus stop. Evalgent's call simulator does this over phone, SIP or WebRTC, with caller profiles for accent, pace, interruption level, response delay, noise and language.

4. Assert the ATS record, not the transcript. Read back fields, disposition, stage and invite. Compare each against the scripted truth. Count a correct summary with a wrong field as a failure.

5. Probe the question wording. Check that the agent asks work authorization, age and availability questions as written. Flag any paraphrase that asks about citizenship, national origin, age beyond the gate, family status or health.

6. Measure score stability. Run each scored test candidate at least five times. Report the share of candidates whose runs cross the cutoff. Send flip-zone scores to human review.

7. Run swap tests for fairness. Keep answers fixed and swap names, accents and speech rates. Compute pass rates per group and the impact ratio. Use roughly 385 calls per group for a ±5-point estimate.

8. Test accommodation and opt-out paths. Script candidates who ask for a reasonable accommodation, ask for a human, say they have a speech disability, refuse recording, or say "stop calling me." Each must reach the right outcome and leave the right record.

9. Test scheduling end to end. Book across time zones and relative dates. Confirm calendar, ATS and SMS agree.

10. Set pass bars before you look at results. If you run 300 probes of a field and see zero failures, the rule of three gives a 95% upper bound of about 3 ÷ 300, or 1%. Decide in advance whether 1% is acceptable for that field.

Starting pass bars (assumptions to tune, not standards):

Evalgent also generates scenarios straight from the agent's prompt and flows, including branches, policies and hard constraints. That fills the coverage gaps hand-written cases miss. It judges calls from the audio, not only the transcript, which matters for the negations above.

Monitoring AI phone screening in production

Pre-release tests catch known risks. Production finds the rest. A new employer adds a knock-out with odd wording. A carrier changes a codec. A model update shifts scores by two points.

Monitor these signals per employer, per req and per language:

• Auto-reject rate by question. A sudden jump in one knock-out's fail rate usually means a parsing change, not a change in candidates.
• ATS reconciliation. Sample calls, re-judge them from the audio, and compare against the stored fields. Disagreements are your real field error rate.
• Disposition mix and stage changes. Compare to the employer's baseline week.
• Score drift. Track the mean and the share in the flip zone after every model or prompt change.
• Tool-call failures. A failed ATS or calendar write the agent covers with "you're all set" is the worst kind of silent error.
• Opt-outs and accommodation requests. Confirm each one wrote back.

Evalgent's production monitoring takes pushed call recordings, transcribes them, maps each to a scenario and judges it. It flags PII leakage, tool-call failures, latency spikes and policy failures, with fix recommendations. For a broader setup, see our guide to monitoring voice agents in production.

Frequently asked questions

How does AI phone screening work?

AI phone screening uses a voice agent to call or answer candidates and ask an employer's knock-out and screening questions. An LLM maps each spoken answer to a typed field. Rules or a scorer set a disposition. The integration then moves the application to an ATS stage and may book an interview. The ATS record, not the call, is what the business acts on.

Is AI phone screening legal?

AI phone screening is legal in the US, but several rules apply as of October 2026. NYC Local Law 144 requires bias audits and notices. Illinois HB 3773 bars discriminatory AI use and requires notice. Colorado SB 26-189 takes effect January 1, 2027. The ADA, Title VII, the TCPA and recording laws also apply. This is not legal advice.

Does NYC Local Law 144 apply to AI phone screening?

NYC Local Law 144 applies when an automated tool substantially assists or replaces discretionary hiring decisions for NYC candidates. DCWP's FAQ says screening counts as an employment decision. A screener that scores or auto-rejects candidates likely qualifies. Employers and employment agencies need an annual bias audit and must give notice 10 business days before use.

How do you test an AI phone screener?

Test an AI phone screener with scripted phone calls generated from each employer's config. Vary accents, pace, pauses and noise. Then read back the ATS record and compare every knock-out field, disposition, stage change and interview invite to what the caller said. Add score-stability runs, name and accent swap tests, and accommodation and opt-out scenarios.

Can an AI phone interview reject a candidate?

An AI phone interview can trigger a rejection if the platform auto-rejects on failed knock-outs or low scores. That is where silent errors do the most harm, because nobody reviews the call. Safer designs route unclear answers and scores near the cutoff to a human. Colorado's SB 26-189 will also give consumers a right to human review after adverse outcomes.

How accurate are AI interview scores?

AI interview score accuracy depends on transcript quality, rubric design and run-to-run variance. LLM judges show position and verbosity bias, and ASR error rates differ across speaker groups. Measure it yourself. Run each test candidate at least five times, report how often scores cross the cutoff, and compare pass rates across groups with the four-fifths rule.

Do AI recruiting calls need consent?

AI recruiting calls to cell phones generally need prior express consent under the TCPA. The FCC ruled in 2024 that AI-generated voices count as artificial voice. Calls must state the caller's identity and a callback number. Recording also needs consent, and California requires all parties to agree. The candidate's application is the usual consent source. Confirm with counsel.

How do you test AI phone screening across employer customers?

Test AI phone screening across employer customers by generating cases from each employer's config. That includes knock-outs, pay bands, shifts, languages, stage maps and calendars. Cover every pair of config values with a pairwise set. Then add the exact config of every employer that changed this week. Assert each case against the ATS record that call wrote.

The bottom line

AI phone screening fails silently in the ATS record, not loudly on the call. Test every release by asserting each employer's knock-out fields, scores, stage changes and invites against what candidates actually said.