AI Interviewer Bias Testing: Accents, Scores and Fairness

Regulation last checked October 16, 2026. This post is engineering guidance, not legal advice.

Your AI interviewer runs phone screens for dozens of employers and scores each candidate against a rubric. Most teams test fairness by feeding the scorer transcripts. That misses half the system. Bias can enter at the phone line, the turn detector and the speech recognizer. By the time the scorer reads the transcript, a strong answer may already look weak.

This post is for the product team that owns that pipeline. It covers where AI interview bias enters, a matched-pair test with sample-size math, Python for impact ratios, production monitoring, and the rules as of October 2026.

Where bias enters an AI interviewer before the scorer

An AI phone interview is a chain of six stages. Each one can lose information, and losses are not spread evenly across speakers. AI interviewer bias testing has to cover all six, not just the last two.

1. Phone line. Most PSTN calls arrive as 8 kHz narrowband audio, often G.711 μ-law. Mobile calls add codec compression, jitter and packet loss. Our guide to phone audio quality and STT covers the codec chain.

2. Turn detection. The agent's VAD and endpointing timer decide when the candidate is done. A candidate who pauses to think, or speaks slowly, can get cut off mid-answer. The transcript shows a short answer.

3. Speech recognition. ASR word error rate varies by accent, dialect and disfluency. Domain terms get swapped and repeated words get deleted.

4. Transcript scorer. An LLM or classifier grades the transcript against the rubric. It grades what the ASR wrote, not what the candidate said.

5. Advance decision. The employer sets a cutoff. A 4-point drop near the cutoff turns "advance" into "reject."

6. ATS record. The score and the decision get written to the applicant tracking system. Nobody re-listens.

This is the "deaf referee" problem: a consistent scorer fed damaged input. See transcript vs audio evaluation.

Three mechanisms that push scores down

Truncation. Endpointing timers in phone agents often sit in the hundreds of milliseconds. A thoughtful candidate who pauses 1.2 seconds mid-sentence loses the second half of the answer. The rubric item "gave a specific example" now fails. Our pause handling guide shows how to test this timer directly.

Substitution of key terms. A candidate says "I ran the forklift certification for my shift." The ASR writes "I ran the four-lift certification." The scorer misses the relevant experience.

Repair loops. When the agent misunderstands, it asks the candidate to repeat. That burns interview time. In a fixed-length interview, the candidate answers fewer questions and scores lower.

A rubric gotcha: style criteria grade the ASR, not the candidate

Rubric items like "communicates clearly" read ASR quality on a transcript. Garbled words look like rambling. If you keep a communication criterion, validate that it is job-related and score it from audio.

Filler handling is a second gotcha. Some ASR settings drop "um" and "uh," and some keep them. Flip a verbatim flag, and disfluent speakers' scores move overnight.

What the research says about ASR gaps

Speech recognition accent bias is well documented. Five studies are worth knowing, and each setup limits how far it generalizes.

The last row matters most. Hickman and colleagues, in the Journal of Applied Psychology, found real transcription bias, yet their scorer was mostly robust to it. A different scorer, rubric or channel could behave differently. Their audio came from video interviews, not 8 kHz phone lines. So the honest answer to "does speech recognition error affect AI interview scores" is: it can, and only a test on your pipeline tells you how much.

Caller profiles are audio conditions, not demographics

A test caller with a Nigerian English accent is an audio condition. It says nothing about any real candidate's national origin or race. Accent is not ethnicity, and speech pace is not age.

So keep two activities separate.

• Engineering tests vary audio conditions on purpose. They ask: does the pipeline score the same answer the same way under different audio? This is matched-pair testing.
• Bias audits measure outcomes for real applicants by protected category. Under NYC's rules, those categories come from the EEO-1 report, and the data is normally historical.

Do not try to infer race, sex or national origin from a candidate's voice. It is unreliable and creates records about protected traits. If an employer wants a demographic audit, use voluntary self-identification data that the employer collects.

Audio-condition tests have limits. Test voices cover only the accents you chose. A TTS "Indian English" voice is not the range of real Indian English speakers. Results show where the pipeline breaks, not that it is fair to every group. Our posts on accent robustness for US calls and the accent handling guide cover how to source accented audio.

Matched-pair testing: same answer, different caller

Matched-pair testing borrows from audit studies in hiring research. You hold the answer fixed and change only the audio. It is the core of AI interviewer bias testing.

Matched pair: two interview calls with the same scripted answer content, delivered under two different caller profiles, scored by the same pipeline version.

Build the answer bank

Write scripted answers to your real interview questions. Two human raters grade each one against the rubric first. Use three tiers:

• Strong: clearly above the employer's advance cutoff.
• Borderline: within a few points of the cutoff. Put half your answers here. This is where flips happen.
• Weak: clearly below the cutoff.

Human grades give you a reference score and expose tier inversions.

Define the profiles

Start with a clean baseline: General American accent, typical pace, quiet room, good line. Then change one dimension per profile.

In Evalgent, a caller profile bundles accent, speech pace, interruption level, response delay, behaviour, background noise and language. The call simulator places the call over PSTN, SIP or WebRTC. The same answer script reaches your interviewer through the real phone path.

Metrics to compute per profile

• Mean paired delta: average of (profile score minus baseline score) for the same answer. Report a 95% confidence interval.
• Decision flip rate: share of pairs where advance or reject changes.
• Tier inversion rate: share of weak-under-baseline answers that outscore strong answers under the profile.
• Mechanism metrics: WER against the script, truncation rate and repeat-request count. They locate the stage that failed.

Evalgent evaluates from the call audio, not only the transcript. That helps you see whether a low score came from the scorer or from words lost upstream.

Worked math: how many test calls you need

First, measure the noise floor. Run the baseline twice for every answer. LLM scorers are not fully deterministic, so the same audio can score differently. If repeat runs differ by several points, small deltas mean nothing.

Then size the test. For a paired design:

n = ((z₀.₉₇₅ + z₀.₈₀) × σd / δ)²

Assume the standard deviation of paired differences σd is 6 points on a 100-point scale. You want to detect a 3-point delta (δ) with 80% power at α = 0.05. Then n = ((1.96 + 0.84) × 6 / 3)² = 5.6² ≈ 31.4. Round up to 32 answers per profile.

With a baseline and six profiles, that is 32 × 7 = 224 calls. Run each three times for scorer noise: 672 calls per release. These are illustrative assumptions; plug in your own σd.

So, how many test calls do you need for a bias test? Hundreds per release, under these assumptions. Ten manual calls will not detect a 3-point gap.

Code: paired deltas, flips and impact ratios

This is simplified Python with no dependencies. Each row is one scored call.

import random
import statistics as st
from collections import defaultdict

CUTOFF = 70  # employer's advance threshold on a 0-100 scale

def paired_report(rows, baseline="baseline"):
    """rows: dicts with answer_id, profile, score. Repeat runs are averaged."""
    runs = defaultdict(list)
    for r in rows:
        runs[(r["answer_id"], r["profile"])].append(r["score"])
    mean = {k: st.mean(v) for k, v in runs.items()}
    profiles = {p for _, p in mean} - {baseline}
    out = {}
    for p in sorted(profiles):
        pairs = [(mean[(a, baseline)], mean[(a, q)])
                 for (a, q) in mean if q == p and (a, baseline) in mean]
        deltas = [x - b for b, x in pairs]
        flips = sum((b >= CUTOFF) != (x >= CUTOFF) for b, x in pairs)
        boots = sorted(st.mean(random.choices(deltas, k=len(deltas)))
                       for _ in range(2000))
        out[p] = {"n": len(pairs),
                  "mean_delta": round(st.mean(deltas), 2),
                  "ci95": (round(boots[50], 2), round(boots[1949], 2)),
                  "flip_rate": round(flips / len(pairs), 3)}
    return out

def scoring_impact_ratios(rows, key="segment"):
    """LL144-style: scoring rate = share above the full-sample median."""
    median = st.median(r["score"] for r in rows)
    above = defaultdict(list)
    for r in rows:
        above[r[key]].append(r["score"] > median)
    rate = {g: sum(v) / len(v) for g, v in above.items()}
    top = max(rate.values())
    return {g: {"n": len(above[g]),
                "scoring_rate": round(v, 3),
                "impact_ratio": round(v / top, 3)} for g, v in rate.items()}

An illustrative output for one profile: `{"n": 32, "mean_delta": -4.1, "ci95": (-5.6, -2.7), "flip_rate": 0.156}`. That is a real drop, with about one in six pairs changing decision.

Adverse impact analysis: selection rates and the four-fifths rule

Any AI hiring bias audit starts with this arithmetic.

So what is the four-fifths rule in AI hiring? The EEOC's Uniform Guidelines say a selection rate for any group that is "less than four-fifths (4/5) (or eighty percent) of the rate for the group with the highest rate" will generally be regarded as evidence of adverse impact. That is 29 CFR 1607.4(D).

The same section adds two caveats. Smaller differences can still count when they are "significant in both statistical and practical terms." Larger differences may not count when they rest on small numbers that are not statistically significant. So the 0.8 line is a screen, not a verdict.

NYC Local Law 144 uses a related measure. Its DCWP final rule defines the scoring rate as "the rate at which individuals in a category receive a score above the sample's median score." The impact ratio is that scoring rate divided by "the scoring rate for the highest scoring category." Tools that score must report it for sex, race/ethnicity and intersectional categories. LL144 itself does not set a pass mark; it requires calculation and publication.

You can apply the same arithmetic to audio segments in testing or production. Here is an illustrative example. Of 1,000 interviews, 540 had a clean line and typical pace. Fifty-six percent of those scored above the full-sample median. The "long pauses" segment had 120 interviews and a 38% scoring rate.

Impact ratio = 0.38 / 0.56 = 0.68. That is below 0.8. A two-proportion z-test gives z ≈ 3.6, so the gap is unlikely to be chance.

An audio-segment ratio is not a legal adverse impact finding. It is an early warning that tells you which stage to fix.

Monitor production scores by audio segment

Pre-release tests catch regressions you can predict. Production shows the rest. Segment live interviews by signals measured from the call itself:

• Background noise level or estimated SNR
• Channel: PSTN, mobile or WebRTC
• Mean ASR confidence on candidate turns
• Speech pace in words per minute
• Truncation events and repeat requests
• Language, employer and job family

Each week, compute advance rates and scoring-rate impact ratios per segment. Alert when a ratio drops below 0.8 with enough volume, or a segment mean moves more than your noise floor. Our metric drift guide covers alert thresholds.

Evalgent's production monitoring takes pushed call recordings, transcribes them, maps them to scenarios and judges each call. Custom metrics run on every call, so a per-segment truncation or repeat-request metric can sit next to the score.

Three kinds of fairness evidence

So, is a matched-pair test the same as a bias audit? No. The test finds causes; the audit measures outcomes for real people. You need both.

What the rules say as of October 2026

This is a summary for engineers, not legal advice. Employers should confirm obligations with counsel.

So, does NYC Local Law 144 apply to AI phone interviews? It can. Per the DCWP FAQ, it covers tools that substantially assist or replace discretionary decisions for jobs in the city. A phone interviewer that outputs a score used to screen candidates may qualify. The obligation sits with the employer or employment agency, but they will ask you, the vendor, for the data.

A December 2025 New York State Comptroller audit found DCWP had reviewed 32 companies and found one case of non-compliance. The auditors found at least 17 potential cases in the same set. Weak enforcement is not a safe harbor.

Illinois does not require bias audits, per Epstein Becker Green, and its notice rules are still being written. Colorado's draft attorney general rules were open for comment until October 26, 2026.

How to run AI interviewer bias testing before each release

1. Freeze the pipeline version. Record the telephony path, VAD and endpointing settings, STT model and flags, scorer prompt and rubric version. A bias result is only valid for that combination.

2. Build a graded answer bank. Write at least 32 answers per question set, with half near the advance cutoff. Have two human raters grade each one.

3. Pick profiles from your real candidate pool. Choose accents, channels and noise levels your employers' candidates actually bring. Add speech pace, disfluency and response delay.

4. Measure the noise floor. Score every answer twice under the baseline profile. Record the repeat-run standard deviation.

5. Run the matched pairs over the phone path. Place calls through PSTN or SIP, not file uploads. Run each pair three times.

6. Compute deltas, flips and ratios. Use paired deltas with confidence intervals, decision flip rates and scoring-rate impact ratios per profile.

7. Trace each gap to a stage. Compare WER, truncation and repeat requests across profiles. Fix the stage that broke, such as endpointing, STT vendor or a rubric style criterion.

8. Gate the release and keep monitoring. Block a release that raises any flip rate or drops any ratio below 0.8. Then watch the same segments in production.

Store results with the version, as our prompt versioning guide describes. Employers and auditors will ask for them. AI interviewer bias testing then becomes a release gate, not a yearly scramble.

For the full QA plan, start with the AI phone screening QA guide. Buyers can use the recruiting screening vendor checklist. See also disfluency robustness and LLM-as-a-judge limits.

Frequently asked questions

Are AI interviews biased against accents?

AI interview bias against accents happens when speech recognition makes more errors on some speakers' audio. Koenecke et al. found a WER of 0.35 for Black speakers vs 0.19 for white speakers across five ASR systems. Whether that becomes a score gap depends on the scorer. Hickman et al. found ASR bias without meaningful score bias in their model. Test your own pipeline.

How do you test an AI interviewer for bias?

Test an AI interviewer for bias with matched pairs. Write graded answers, then deliver each one under a clean baseline and under caller profiles that vary accent, pace, disfluency, noise and phone line. Compare scores for the same answer. Report mean paired deltas, decision flip rates and impact ratios, then monitor live scores by audio segment.

What is the four-fifths rule in AI hiring?

The four-fifths rule comes from the EEOC Uniform Guidelines at 29 CFR 1607.4(D). A group's selection rate below 80% of the highest group's rate is generally treated as evidence of adverse impact. It is a screen, not a final test. Statistical significance and sample size also matter, and smaller gaps can still count.

Does NYC Local Law 144 apply to AI phone interviews?

NYC Local Law 144 can apply to AI phone interviews that score candidates for NYC jobs and substantially assist hiring decisions. Employers must have a bias audit within one year of use, publish a summary and notify candidates. The audit reports scoring-rate impact ratios by sex, race/ethnicity and intersectional categories. Confirm coverage with counsel.

Can you test for bias without collecting demographic data?

Engineering bias tests do not need demographic data. Matched-pair tests vary audio conditions like accent, pace and line quality using scripted test callers. Production monitoring can segment by measurable call signals such as noise, channel and truncation. A legal bias audit of outcomes still needs category data, usually voluntary self-identification collected by the employer.

How many test calls do you need for a bias test?

A paired test needs about 32 answers per profile to detect a 3-point delta, assuming a 6-point standard deviation of differences, 80% power and α of 0.05. With a baseline and six profiles, that is 224 calls. Running each three times for scorer noise gives 672 calls per release. Use your own measured variance.

Does speech recognition error affect AI interview scores?

Speech recognition error can lower AI interview scores when the scorer reads a damaged transcript. Deleted words, swapped domain terms and truncated answers look like weak content. The effect is not guaranteed. One 2025 study found ASR bias with little score bias. Measure WER and score deltas together on your own pipeline to know.

Is a matched-pair test the same as a bias audit?

A matched-pair test is not a bias audit. The test holds answer content fixed and changes audio conditions, so it finds which pipeline stage causes a score gap. A bias audit, such as one under NYC Local Law 144, measures outcomes for real applicants by protected category using historical data. Teams need both.

The bottom line

Bias in an AI interviewer often enters through the audio pipeline, before the scorer reads a single word. Test it with matched pairs over the real phone path, measure deltas, flips and impact ratios, and keep watching live scores by audio segment.