How to QA Drive-Thru Voice AI Across 500 Stores

Your drive-thru agent passed at the pilot store. Now the plan says 500 stores by next quarter. This post is for the team that owns that rollout, usually inside a QSR technology or POS software company. It gives a copyable plan for drive thru voice AI testing: strata, test orders per stratum, gates, canary waves and rollback triggers.

The menu-modifier basics are covered in our restaurant phone ordering QA playbook. This post is about the multiplier.

Why one passing drive-thru store proves almost nothing

A single pilot store samples one menu zone, one lane layout, one noise profile and one crew. Most rollout failures live in the combinations you did not sample.

The public record shows the pattern. McDonald's ran its Automated Order Taker with IBM in about 100 restaurants. In June 2024 it told franchisees it would end that test, Nation's Restaurant News reported. The same report cites a 2022 BTIG estimate of accuracy "in the low 80% range" against a 95%-plus target.

Taco Bell had voice AI in more than 500 drive-thrus and over two million orders by 2025, per Silicon UK citing the Wall Street Journal. After viral clips, including an "18,000 water cups" order, its digital chief said the company would be more deliberate about where AI runs. He noted that busy stores with long lines may do better with a human, per NRN. That is a per-store conclusion, not a per-model one.

Wendy's took the staged rollout path with FreshAI. It started with Google Cloud at a handful of Ohio stores. Restaurant Business reported it passed 100 locations and targeted 500 to 600 by the end of 2025. At launch, both companies said "noise and order complexity" make the drive-thru hard, per Restaurant Dive.

Independent mystery shops show drive-thru order accuracy is improving but not solved. Intouch Insight's 2026 study made 120 visits to AI drive-thrus at Burger King, Taco Bell and Wendy's. Accuracy was 90.8%, up from 82.5% the year before. Nearly three in ten guests still repeated part of the order. The 2025 report found 62% of wrong AI orders came from customization. Orders the AI finished alone were 81% accurate. When a crew member stepped in, accuracy was 95%.

Now the math. Suppose 3% of your stores carry a store-specific defect, like a stale price file or a broken lane mapping. One pilot store finds it 3% of the time. Ten random stores find it about 26% of the time. You need 99 random stores for a 95% chance of seeing even one. Random sampling is the wrong tool. Defects cluster by store type, so use stratified sampling by store type.

What changes from one drive-thru store to the next

Store variance comes in three kinds. Data variance is the menu, prices and hours. Acoustic variance is the speaker post and its surroundings. Human variance is who orders and who interrupts. Each kind needs a different test method.

Data variance: test every store, not a sample

Menus, prices, limited-time offers and dayparts are data. You do not need to drive to a store to test them. Point your simulator at each store's real menu sync and run a short scripted suite against all 500.

A good per-store suite has about 20 orders. Cover every active LTO, one 86'd item, the breakfast cutoff and the most expensive combo. The pass check is mechanical. The total the agent reads back must match the total the POS computes.

A price drift is invisible at the speaker post. It shows up at the window, or not at all.

Acoustic variance: the speaker post is a far-field microphone

The guest sits in a car, often a meter or more from the post, with the engine running. That makes drive-thru audio a far-field, noisy channel. Phone-tuned ASR settings often assume the opposite.

Research shows how fast accuracy falls in that setting. The CHiME-4 benchmark records the same read speech in four real outdoor and public places. One 2021 paper's single-channel baseline scored 18.81% word error rate on a bus and 7.56% at a street junction. Same system, same task: a 2.5x spread from environment alone.

Conversational far-field speech is harder still. In the CHiME-6 challenge, dinner-party speech was captured by distant microphone arrays. The best official baseline scored 51.3% WER on the evaluation set, even with oracle segmentation.

Large modern models help but do not erase the effect. In OpenAI's Whisper paper, many models beat Whisper at 40 dB signal-to-noise ratio. Below 10 dB of added pub noise, all of them degraded quickly and fell behind it. The lesson for drive thru voice AI testing: rank your ASR choice at the SNR your worst posts produce, not at studio quality.

The audio path matters too. In an Intel case study, the drive-thru AI connected through the existing base station, speakers and crew headsets. Crew used headset commands to mark items unavailable. So the agent hears whatever that base station passes. Old posts, worn foam covers and gain settings vary by store. For the noise test design itself, see our guide to testing STT under background noise.

Dual lanes add a sequencing problem. Two posts feed one kitchen queue. If lane two finishes first, tickets can swap, and a guest pays for someone else's meal. Interleave test orders from both lanes and check which ticket lands where.

Human variance: accents, Spanish and the crew

ASR error is not spread evenly across speakers. A 2020 PNAS study tested five commercial ASR systems on matched audio. Average word error rate was 0.35 for Black speakers and 0.19 for white speakers. Your store map is also a speaker map. Our accent robustness guide covers how to build those caller profiles.

Spanish orders are a store-level variable, not a footnote. Per the Census Bureau, 78.3% of US residents age 5 and over spoke only English at home in 2018–2022. Spanish was 61.1% of the rest. That share varies a lot by county, so it belongs in your strata. Test pure Spanish orders and mid-order switches. Our multilingual handling guide has scripts for both.

Crew interjections are the drive-thru's own failure mode. A crew member on the headset says "we're out of shakes" while the guest is ordering. If that audio reaches the agent, it may treat it as the guest's speech. Test it on purpose. Inject crew lines into the order and assert the ticket ignores them, or applies them only when intended.

How a misheard word corrupts the POS ticket silently

The drive-thru call is the front door to an agentic workflow. Speech becomes an order object. The order object becomes a POS ticket. The ticket drives the kitchen display, the total and the payment. A readback confirms it, and the record is written back for reporting.

An error at the first step rarely raises an alarm at the later ones. "No pickles" heard as "pickles" makes a valid ticket. "Two" heard as "ten" makes a valid ticket with a large total. The guest says "yeah" to a fast readback over engine noise. Every system downstream records success.

That is why transcript-only scoring fails here. The transcript may be wrong in the same way the agent was wrong, so a judge reading it agrees with the error. Score from the audio and from the POS write, as our piece on transcript vs audio evaluation explains. Check the structured fields too: item, size, quantity, modifiers and price. Our guide to tool argument accuracy shows how to assert them. Modifier edge cases, like half-and-half items, are covered in why half-and-half orders break.

The store-stratified rollout plan

Store-stratified rollout plan: a release plan that groups stores by the factors that change agent behavior, tests each group to a fixed evidence bar, and widens the rollout one wave at a time behind pass/fail gates.

It has four parts: tiers, conditions, sample size and gate. Here is a worked version for a 500-store fleet. The thresholds are starting assumptions to tune, not industry standards.

Step one: define strata from store attributes

Pick five attributes you already hold in the store master. Lanes (1 or 2). Menu zone (say 4). Volume tier (high or low). Noise exposure (highway-adjacent or standard). Spanish share (above or below 15%). That gives up to 2 × 4 × 2 × 2 × 2 = 64 strata. Some will be empty, and that is fine.

Step two: set conditions per stratum

Each stratum gets a test matrix: three SNR noise levels, two speaker types, a dual-lane interleave if relevant, and crew interjections. Use noise recorded at your own posts.

Step three: run the waves behind gates

Pick wave-1 canaries by risk: a dual-lane highway store with high Spanish share, not the store near your office. Our shadow testing guide covers running the agent silently beside the crew first, which makes a good wave 0.5.

Worked sample-size math for test orders per stratum

Three formulas cover most sample-size questions in drive thru voice AI testing.

Zero-failure demonstration. To claim a failure rate at or below p with 95% confidence and zero failures, you need n = ln(0.05) / ln(1 − p) orders. That is 59 orders for 5%, 99 for 3%, 149 for 2% and 299 for 1%. One failure resets the claim, so plan for more.

Wilson lower bound. With failures allowed, gate on the lower edge of the Wilson interval, not the raw rate. 148 of 150 passing gives a lower bound of 95.3%. 141 of 150 gives 89.0%. The raw rates (98.7% and 94%) look closer than they are. With 150 orders, a lower bound of 94% needs at most 3 failures.

Canary detection. If a fraction q of stores carries a store-specific defect, k random stores find it with probability 1 − (1 − q)^k. At q = 5%, ten stores give 40%, and you need 59 for 95%.

The chart is the argument for stratification. Random canaries need dozens of stores to catch a rare store type. Stratified canaries need one store per type, because the type is the defect's address.

Budget for a 64-stratum fleet: 64 × 150 = 9,600 simulated orders, plus 500 × 20 = 10,000 per-store menu orders. That is impossible by hand and routine for an automated simulator.

How to test drive-thru voice AI before a 500-store rollout

1. Export the store master. Pull lanes, menu zone, volume, noise exposure and Spanish share for every store. Assign each store a stratum key.

2. Run the menu suite on every store. Point the simulator at each store's live menu sync. Assert readback total equals POS total and no invalid items reach the ticket.

3. Build per-stratum caller profiles. Combine accent, speech pace, language, noise level and crew interjections. Use recorded post noise at three SNR levels.

4. Run 150 orders per stratum. Score item, size, quantity, modifiers and total against the POS write. Gate on a Wilson lower bound of 94% or your own bar.

5. Shadow, then canary. Run silently beside the crew in the riskiest strata. Then let the agent take orders at 8 canary stores with crew takeover enabled.

6. Cover every stratum. Add one live store per occupied stratum. Check each store daily against its control limit.

7. Expand in waves. Move to 25% of the fleet, then all stores, only when the previous gate passes.

8. Arm per-store rollback. Wire triggers so any store can drop back to crew order-taking without a release.

9. Re-run the suite on every change. Menu updates, LTO launches, prompt edits and model swaps all go back through wave 0 first.

A simple Python sampler and per-store pass rate check

This simplified script picks one canary per stratum, sizes zero-failure tests, gates strata with a Wilson lower bound and flags stores with a p-chart. It uses only the standard library.

import math
import random
from collections import defaultdict
from dataclasses import dataclass


@dataclass
class Store:
    store_id: str
    lanes: int             # 1 or 2 speaker posts
    menu_zone: str         # pricing / LTO zone
    volume: str            # "high" or "low"
    noise: str             # "highway" or "standard"
    spanish_share: float   # estimated share of Spanish orders


def stratum(s: Store) -> tuple:
    return (s.lanes, s.menu_zone, s.volume, s.noise, s.spanish_share >= 0.15)


def pick_canaries(stores, per_stratum=1, seed=7):
    rng = random.Random(seed)
    groups = defaultdict(list)
    for s in stores:
        groups[stratum(s)].append(s)
    picks = []
    for key in sorted(groups):
        members = groups[key]
        picks += rng.sample(members, min(per_stratum, len(members)))
    return picks, groups


def zero_failure_n(max_fail_rate, confidence=0.95):
    """Orders needed with zero failures to claim fail rate <= max_fail_rate."""
    return math.ceil(math.log(1 - confidence) / math.log(1 - max_fail_rate))


def wilson_lower(passes, n, z=1.96):
    if n == 0:
        return 0.0
    p = passes / n
    denom = 1 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre - margin) / denom


def gate(results, min_lower=0.94, min_n=150):
    """results: {stratum: (passes, n)}. A stratum passes only on enough evidence."""
    report = []
    for key, (passes, n) in sorted(results.items()):
        lower = wilson_lower(passes, n)
        ok = n >= min_n and lower >= min_lower
        report.append((key, passes, n, round(lower, 3), "PASS" if ok else "HOLD"))
    return report


def below_control_limit(store_passes, store_n, fleet_rate, sigmas=3):
    """p-chart lower limit: flags a store whose daily pass rate is unusually low."""
    lcl = fleet_rate - sigmas * math.sqrt(fleet_rate * (1 - fleet_rate) / store_n)
    return store_passes / store_n < lcl, round(lcl, 3)


if __name__ == "__main__":
    rng = random.Random(1)
    zones = ["northeast", "south", "midwest", "west"]
    stores = [
        Store(f"S{i:03d}", rng.choice([1, 1, 2]), rng.choice(zones),
              rng.choice(["high", "low"]), rng.choice(["standard", "standard", "highway"]),
              rng.choice([0.02, 0.05, 0.2, 0.35]))
        for i in range(500)
    ]
    canaries, groups = pick_canaries(stores)
    print(f"{len(groups)} strata, {len(canaries)} canary stores")
    print("zero-failure orders for <=5% / <=2% fail:", zero_failure_n(0.05), zero_failure_n(0.02))
    sim = {("2-lane", "highway", "spanish"): (141, 150),
           ("1-lane", "standard", "english"): (148, 150)}
    for row in gate(sim):
        print(row)
    print("store S042 flagged:", below_control_limit(196, 240, fleet_rate=0.92))

Output on the synthetic 500-store list above:

64 strata, 64 canary stores
zero-failure orders for <=5% / <=2% fail: 59 149
(('1-lane', 'standard', 'english'), 148, 150, 0.953, 'PASS')
(('2-lane', 'highway', 'spanish'), 141, 150, 0.89, 'HOLD')
store S042 flagged: (True, 0.867)

The dual-lane, highway, Spanish stratum holds even at 94% raw accuracy. Its lower bound is 89%, below the 94% gate. That is the stratum a single pilot store would have hidden.

Rollback triggers and live monitoring after launch

Gates protect the rollout. Rollback triggers protect each store after it. Make them per store, automatic and reversible. Falling back to crew order-taking should be a config flag, not a deploy.

The p-chart in the code is a control chart for proportions. It sets each store's daily lower limit from the fleet rate and its own volume. With a fleet AI-completed rate of 92% and 240 orders a day, the limit is 86.7%. A store at 81.7% gets flagged.

Live monitoring needs a sample of real audio, not just counters. Our guide to monitoring voice agents in production covers tool-call failures and latency spikes. For prompt and model updates, follow the release discipline in shipping prompt changes safely. Every change goes back through wave 0.

Where Evalgent fits in a drive-thru rollout

Evalgent is a testing, evaluation and monitoring platform for voice agents. For drive thru voice AI testing, it generates test scenarios from your agent's prompt and flows, including branches and hard constraints like "never sell an expired LTO." Its caller profiles set accent, speech pace, interruption level, background noise and language, including Spanish. That maps directly to the per-stratum conditions above.

Evalgent judges from the audio, not only the transcript, and every verdict comes with evidence. Custom metrics, such as "readback total equals POS total," can run on every call. The simulator reaches agents over PSTN, SIP or WebRTC, so it fits a drive-thru agent that exposes one of those test endpoints.

After launch, teams can push drive-thru recordings in for monitoring. Calls are transcribed, mapped to scenarios and judged, with tool-call failures and latency spikes flagged. Evalgent works with your team to set up evals for the first agent.

Frequently asked questions

How do I test AI voice agents for drive-thru ordering before rolling out to hundreds of stores?

Group stores into strata by lanes, menu zone, volume, noise exposure and language mix. Simulate about 20 orders at every store against its live menu. Run 150 simulated orders per stratum and gate on a Wilson lower bound. Then launch to canaries, cover one store per stratum, and expand in waves with per-store rollback triggers.

How many test orders per store do I need?

The number of test orders per store depends on the claim. With zero failures, 59 orders support a 5% failure ceiling at 95% confidence, and 149 support 2%. For menu and price checks, about 20 scripted orders per store is enough. Acoustic and accent conditions are cheaper to test per stratum than per store.

Why does drive-thru voice AI fail at some stores but not others?

Drive-thru voice AI fails at some stores because stores differ in ways the pilot never sampled. Menus, LTOs, price zones, dayparts, speaker posts, highway noise, dual lanes, accents, Spanish share and crew habits all vary. Defects cluster by store type, so a stale price file or noisy post breaks whole groups of stores at once.

What is a canary store?

A canary store is a live location that gets a new drive-thru agent, or a change to it, before the rest of the fleet. Crew can take over at any time. Pick canaries from the riskiest strata, such as dual-lane highway stores, so problems show up early and on a small number of stores.

How accurate is drive-thru voice AI?

Intouch Insight's 2026 mystery shops measured 90.8% order accuracy at AI drive-thrus, up from 82.5% in 2025. Nearly three in ten guests repeated part of the order. Its 2025 report found 62% of wrong AI orders came from customization, and accuracy was 95% when crew stepped in.

How to test drive-thru voice AI with background noise?

Record real noise at your own speaker posts: engines, wind, rain and traffic. Mix it into test orders at three set signal-to-noise ratios, such as 20, 10 and 5 dB. Score item, size, quantity and modifiers against the POS write, not the transcript. Compare results per noise level, because published benchmarks show accuracy falling sharply in noisy settings.

When should you roll back drive-thru voice AI at a store?

Roll back a store when its readback total disagrees with the POS total. Also roll back when its AI-completed rate falls below its p-chart limit two days running, or remakes double against its pre-AI baseline. Rollback should be a per-store config flag that returns order-taking to crew without a new release.

How to monitor drive-thru voice AI after launch?

Track per-store AI-completed rate, remake and void rate, repeat-ask rate, latency at peak dayparts and language fallbacks. Set control limits from each store's own volume. Sample real audio daily and judge it against the POS ticket. Re-run the full pre-launch suite after any menu, LTO, prompt or model change.

The bottom line

One passing drive-thru store proves almost nothing about 500, because failures cluster in store types you did not sample. Stratify stores, test every store's menu data, gate each stratum on a lower confidence bound, and give every store its own rollback switch.