Benchmarking Voice Agents on Your Own Data

A voice agent that tops a public leaderboard can still fail your calls, and the reason is not mysterious. The leaderboard measured a different population — different accents, different vocabulary, different tasks — than the one that phones your business. Its winner is the best agent for that data, which is not necessarily the best agent for yours. The only benchmark that predicts your production is one built on your own calls. This guide explains why, and how to build it.

We will cover why generic benchmarks don't transfer, what "your own data" actually means, how to build a first-party benchmark, and the leaderboard trap that catches buyers. It extends the vendor benchmarking guide and the case that vendor metrics mislead.

Why generic benchmarks don't transfer

The technical name for the problem is dataset shift: a model measured on one distribution behaves differently on another. Your callers are a specific distribution — a particular mix of accents, a domain vocabulary of product names and account formats, a set of tasks, and the acoustic conditions of your channels. A public benchmark is a different distribution, usually a cleaner and more general one.

An agent tuned to score well on the general case can underperform on yours, and vice versa. This is a question of external validity: results generalize to your setting only when the test resembles your setting. A leaderboard built on read-aloud audiobook speech tells you almost nothing about a noisy collections call full of dollar amounts and dates. The mismatch is why the demo and the benchmark both look great while production disappoints.

What "your own data" actually means

"Your own data" is not just recordings — it is the distribution your callers create, captured well enough to test against. Four dimensions matter most.

Scenarios — the actual reasons people call you, in their real proportions, including the long tail of edge cases. Vocabulary and entities — your product names, account formats, place names, and the numbers and dates your calls hinge on, which a generic benchmark never contains. Accents and speech patterns — the real range of your callers, not a standardized sample. Acoustic conditions — the lines, devices, and background noise your calls actually come over.

A benchmark that captures these four is representative of your traffic. One that captures none of them is a generic test wearing your logo, and it will mislead you the same way a vendor's leaderboard does.

How to benchmark on your own data

The build is a one-time investment that pays off on every future comparison.

1. Sample representative real calls — Draw from actual traffic across time, so the sample) of scenarios matches production rather than a hand-picked few.

2. Cover your distribution — Include the common calls in their real proportions and the costly edge cases, so the benchmark isn't skewed toward the easy path.

3. Capture your vocabulary — Ensure your product names, account formats, and the critical entities your tasks depend on appear, since these are where generic accuracy collapses.

4. Represent your callers — Span the accents, pace, and acoustic conditions your callers actually have, using synthetic callers to reach the range at scale.

5. Handle the data responsibly — Redact personal information and keep the set access-controlled, aligning with a framework like the NIST AI Risk Management Framework so the benchmark itself passes a security review.

6. Score on your own outcomes — Define "resolved" for your tasks and score against it, not against a generic accuracy metric that ignores whether the call succeeded.

7. Refresh it as traffic changes — Update the benchmark as new products, scenarios, and caller patterns appear, so it keeps reflecting reality.

Measure what your calls hinge on

A generic accuracy number hides the errors that actually cost you. On your data, weight the metrics toward what your calls depend on. Transcription should be scored on critical entities — a name, an account number, a dollar amount — not just an aggregate word error rate, because one wrong digit can fail a task a good average would pass. Latency should be judged at the tail against a real threshold like the ITU-T G.114 150ms comfort limit. And task success should be defined by your outcomes, so the benchmark rewards agents that resolve your calls rather than agents that merely transcribe cleanly. The point of using your own data is lost if you then score it with someone else's definition of good.

The leaderboard trap

Public leaderboards create a second, subtler problem: once a benchmark becomes the number vendors compete on, they optimize for it. That is overfitting to the benchmark — improving the score without improving the outcome it was meant to represent. A vendor can climb a leaderboard by tuning to its quirks and still handle your calls no better than before.

So a leaderboard tells you two things poorly: how an agent does on data that isn't yours, and how well it was tuned to that specific test. Neither predicts your production. Use public results as background — a rough signal that a vendor is in the running — and then run your own benchmark to decide. Your data is the tiebreaker that a leaderboard can't be.

When a public benchmark is enough

Not every decision needs a full own-data benchmark. If you are early, screening a long list of vendors to find a shortlist, a public leaderboard is a reasonable first filter — it cheaply rules out the clearly uncompetitive. The moment the decision carries real weight, though — a shortlist to choose between, a contract to sign, a launch to a live queue — the leaderboard has done its job and your own data has to take over. The rule of thumb: use public benchmarks to widen the funnel, and your own data to close it. The higher the stakes, the more the ranking has to come from your calls.

Generic benchmark vs your own data

The contrast is the decision.

Common mistakes

The errors follow from skipping the point. Trusting a leaderboard ranking as if it were your ranking. Building a "custom" benchmark that still uses clean, general audio instead of your real calls. Leaving out your domain vocabulary, so the benchmark never tests the entities your tasks depend on. Sampling only the easy scenarios, so agents look equivalent until production. Scoring on generic accuracy rather than your outcomes. And building the benchmark once and letting it drift out of date as your products and callers change. Each one quietly turns an "own-data" benchmark back into a generic one.

Benchmarking on your own data with Evalgent

Evalgent is built to benchmark on your data, not a generic proxy. Scenarios are drawn from your real calls — your tasks, your edge cases, your vocabulary — as a fixed, versioned suite. Profiles reproduce the accents, pace, and acoustic conditions of your callers, so the benchmark spans your distribution rather than a clean sample. Metrics score transcription on your critical entities, latency at the tail, and task success against your own definition of resolved, so the numbers reflect your outcomes. Evaluations run the suite at your expected concurrency, and Reviews let your team replay any call behind a score. Because the benchmark is your traffic, its results predict your production — and because you own it, no vendor can tune to it.

The result is a comparison grounded in reality: agents ranked on the calls you actually get, scored by what success means to you. To benchmark the agents you are weighing on your own call data, book a demo. For turning the results into a decision, see the vendor scorecard.

The bottom line

Benchmarking voice agents on your own data means testing them on your real calls — your accents, vocabulary, scenarios, and noise — and scoring against your outcomes. Public leaderboards measure a different population and can be gamed, so their winner is not necessarily yours.

Build the benchmark from first-party calls, weight it toward what your calls depend on, and refresh it as your traffic changes. A leaderboard tells you which agent is best on someone else's data; your own benchmark tells you which agent is best on yours — the only ranking that predicts production.

Frequently asked questions

Why benchmark voice agents on your own data?

Because your callers are a specific distribution — accents, vocabulary, scenarios, and noise — that public benchmarks don't match. An agent that tops a general leaderboard can still fail your calls, since it was measured on different data. Only a benchmark built from your real traffic, scored against your outcomes, predicts how an agent will perform in your production, which is the ranking that matters.

Why don't public voice AI benchmarks predict production?

Because of distribution shift: results measured on one population don't transfer to another. A leaderboard built on clean, general speech tells you little about your noisy, domain-specific calls. Public benchmarks also get optimized once vendors compete on them, inflating scores without improving real performance. Both problems mean a leaderboard rank can't stand in for a benchmark on your own data.

What counts as "your own data" for a benchmark?

The distribution your callers create, captured well enough to test against: your real scenarios in their real proportions, your vocabulary and critical entities like account formats and product names, the accents and speech patterns of your callers, and the acoustic conditions of your channels. A benchmark that reflects those four dimensions is representative; one that reflects none is a generic test wearing your logo.

How do you build a benchmark from your own calls?

Sample representative real calls across time, cover your scenario distribution including edge cases, and make sure your domain vocabulary and critical entities appear. Represent your callers' accents and channel conditions, using synthetic callers to reach the range at scale. Redact personal data and keep the set controlled, then score against your own definition of a resolved call rather than a generic accuracy metric.

Can you use your production call data for benchmarking safely?

Yes, with care. Treat the benchmark like any sensitive asset: redact or mask personal information as you curate calls, keep the set access-controlled, and prefer synthetic callers that reproduce the pattern of a hard call without a real customer's private details. Aligning the handling with a recognized risk framework lets you use realistic data while still passing a security review.

How is this different from a generic accuracy benchmark?

A generic accuracy benchmark scores clean, general audio against a broad metric, so it measures a population that isn't yours and a definition of good that ignores whether calls succeed. Benchmarking on your own data tests your real calls and scores against your task outcomes. The first tells you how an agent does in general; the second tells you how it does for you.

Do public leaderboards have any value?

Some, as background. A leaderboard can hint that a vendor is broadly competitive and worth including in your evaluation. But it can't decide the purchase, because it measures data that isn't yours and can be gamed by vendors tuning to it. Use public results to build a shortlist, then run your own benchmark on your call data to make the actual decision.

How often should you refresh a first-party benchmark?

Refresh it whenever your traffic changes — new products, new scenarios, shifting caller patterns — and on a regular cadence regardless, because distributions drift over time. A benchmark that reflected last year's calls slowly stops predicting this year's. Feed new failure modes back in as production surfaces them, so the benchmark keeps matching the reality it is supposed to measure.