How to Build a Golden Dataset for Voice Eval

# How to build a golden dataset for voice agent evaluation

Quick answer

A golden dataset for voice agent evaluation is a curated set of real and synthetic calls paired with verified correct answers, or ground truth. You build it by sampling representative flows and edge cases. Then you label them against clear rubrics, check that labelers agree, size the set for coverage, and version it as the product changes.

Every repeatable benchmark rests on one thing you rarely see discussed: the answer key. You cannot score a voice agent without knowing what the right answer was. That answer key is the golden dataset, and its quality caps the quality of every number you report. A weak set produces confident, meaningless scores. A strong one turns evaluation into a decision you can defend. This guide covers how to build a strong one.

What a golden dataset actually is

A golden dataset is the fixed reference an evaluation scores against. Each item is a call, or a slice of a call, with the correct output attached.

> Golden dataset: a curated collection of real and synthetic calls, each paired with verified correct answers used as the benchmark's answer key. In machine learning terms it is a labeled test set, held fixed so results stay comparable over time.

The correct answers are the ground truth. Ground truth is what actually happened or what should have happened, established by a human rather than by the agent under test. For a transcript, ground truth is the exact words spoken. For a task, it is whether the caller's goal was met. Attaching ground truth to raw calls is data labeling, and it is most of the work.

This is why a golden set is not a pile of recordings. Recordings alone tell you nothing about whether the agent got them right. The labels are what make the set gradeable. Still deciding whether you need this at all? The difference between ad hoc checks and a real reference set is covered in our guide on testing versus evaluation for voice agents. A golden dataset is the artifact that makes true voice agent evaluation possible.

Why the golden set is the foundation of any benchmark

A benchmark is a comparison. It only means something if every agent faces the same questions and is graded by the same answer key. Change the questions between runs and the scores stop being comparable. Change the answer key and you are measuring your own inconsistency, not the agents.

The golden dataset is that stable answer key. Hold it fixed and you can rerun it against a new model, a prompt change, or a rival vendor and trust the delta. That is the whole point of building one benchmark and reusing it. It lets you compare voice agents on the same test cases instead of on marketing claims. It also lets you benchmark on your own data rather than a public leaderboard that measured a different population.

Without a golden set, teams fall back on vibes. Someone listens to a few calls, forms an impression, and calls it evaluation. That does not scale and does not survive a disagreement. A golden dataset replaces opinion with a repeatable procedure. The investment is front-loaded, but it pays back on every run after the first.

What belongs in a golden dataset

A good set covers your traffic in the proportions it actually occurs, plus the hard cases that break agents. Each component has a right way to build it and a common way it goes wrong.

Notice the balance between real and synthetic. Real calls anchor the set to your actual traffic and keep it honest. Synthetic calls fill the gaps real data leaves, especially rare emergencies and edge conditions you cannot wait months to observe. You need both. The mix is what makes the set representative without being at the mercy of what happened to phone in last month.

Coverage is the other theme. A set that only holds clean, cooperative calls flatters every agent. The value lives in the hard slices. Think of the caller with a strong accent on a bad line, the name that has to be spelled, the long account number, the interruption mid-sentence. Those are the calls that separate a good agent from a fragile one.

How to build your golden dataset

Follow these steps in order. Each one produces an artifact the next step depends on, so skipping ahead creates rework.

1. Define the flows and outcomes first. List the intents your agent handles and the outcome that counts as success for each. This is your rubric, and it must exist before any labeling starts.

2. Sample real calls representatively. Pull production calls across several weeks, times of day, and channels. Use stratified sampling so each flow appears in roughly the proportion it occurs live.

3. Map your coverage gaps. Tag the sampled calls by flow, accent, and condition, then find what is thin or missing. Rare and high-stakes flows are almost always underrepresented.

4. Fill gaps with synthetic calls. Generate targeted scenarios for the thin slices, including the edge cases real data barely contains. Aim for controlled variety, not clean scripts.

5. Establish ground truth. Have humans transcribe the audio verbatim and record the correct outcome for each call. Never treat the agent's own output as truth.

6. Label against the rubric with multiple raters. Have at least two people label a shared portion so you can measure agreement and catch ambiguous cases early.

7. Redact and secure sensitive data. Mask personal information as you curate, keep the set access-controlled, and prefer synthetic stand-ins where a real customer's details are not needed.

8. Version and freeze the set. Assign a version number, record how it was built, and lock it. From here every benchmark run uses this exact version until you deliberately update it.

The output is a labeled, versioned, representative set you can rerun forever. Building it once is the expensive part. Reusing it is nearly free, which is exactly why the discipline of steps five through eight matters more than it looks.

How big should the set be

Size follows coverage, not a round number. The question is not how many calls. It is whether every flow and every hard condition has enough examples to give a stable score. A rare flow with three examples produces a score that swings wildly if one call is unusual. The same flow with thirty examples gives a number you can trust.

Work bottom up. Decide the smallest slice you want to report on, such as a single accent within a single flow. Give that slice enough calls that adding a few more would not move its score much. Then sum across slices. For most production agents that lands in the hundreds to low thousands of calls, weighted toward the flows and conditions that carry your risk.

Resist the urge to pad the easy cases. A thousand clean, cooperative calls tell you little you did not already know. Fifty hard ones tell you where the agent breaks. Spend your labeling budget where the information is. That means the edges and the failures, not the happy path you are already confident about. Watch for sampling bias as you cut: a set that quietly overrepresents your best calls will overstate the agent.

Labeling with ground truth and clear rubrics

Labels are only as good as the rules behind them. Before anyone touches a call, write the rubric. It should define what a correct transcript looks like, what a resolved call looks like, and how to handle the gray cases. Ambiguity in the rubric shows up later as noise in your scores.

For transcription, decide your unit and metric up front. Scoring at the word level versus the character level changes what your numbers mean. We unpack that distinction in word error rate versus character error rate. Decide too whether you grade the audio directly or the transcript, since those measure different things. That choice is covered in transcript versus audio evaluation.

For task success, the rubric must be concrete. "Did the agent resolve the call" is not enough. Spell out what resolution requires for each flow, including the entities that must be correct. A single wrong digit in an account number can fail a task that a lenient rubric would pass. This kind of scenario-level grading is central to disciplined voice agent testing.

Making labels agree

Two careful people will still disagree on hard calls. If you do not measure that disagreement, you cannot trust the labels. So have multiple raters label a shared subset and compute inter-rater reliability. A common statistic is Cohen's kappa, which corrects for agreement that would happen by chance.

Low agreement is a signal, not a failure. It usually means the rubric is vague, not that the raters are careless. When kappa is low, go back and sharpen the rule, add examples of the gray cases, and relabel. High agreement is what lets you trust a score. It shows the number reflects the agent, not the mood of whoever graded that day. This is also why an independent evaluator is valuable. A neutral party has no incentive to bend an ambiguous label toward a flattering result.

Versioning as the product changes

Your product will not sit still, and your golden set cannot either. New flows launch. Callers start asking about things that did not exist last quarter. A set that reflected last year's traffic slowly stops predicting this year's. So the set is a living artifact under version control, not a one-time deliverable.

Treat updates deliberately. When you add calls, cut stale ones, or fix labels, cut a new version with a clear changelog. Keep old versions so past benchmark results stay reproducible. This matters more than it sounds. If you quietly edit the set, every historical score becomes uninterpretable. You no longer know what it was measured against.

A useful rhythm is to refresh on two triggers. Refresh when the product changes in a way that adds or alters flows. And refresh on a regular cadence regardless, because caller behavior drifts even when you ship nothing. Feed new failure modes back in as production surfaces them, so the set keeps matching the reality it is supposed to measure. A frozen set that never updates is as misleading as one that changes on every run.

Frequently asked questions

What is a golden dataset for voice agent evaluation?

A golden dataset for voice agent evaluation is a curated set of real and synthetic calls, each paired with verified correct answers called ground truth. It serves as the fixed answer key a benchmark scores against. Because it stays constant across runs, it lets you compare models, prompts, and vendors on identical questions rather than on shifting or subjective criteria.

How do you build a golden dataset for a voice agent?

Start by defining the flows and success outcomes, then sample real production calls across weeks and channels using stratified sampling. Map coverage gaps and fill them with synthetic calls for rare and edge cases. Establish ground truth with human transcription, label against a clear rubric using multiple raters, redact sensitive data, then version and freeze the set for reuse.

How large should a voice agent golden dataset be?

Size the set by coverage, not a target count. Every flow and hard condition needs enough examples that its score is stable and would not swing on one unusual call. For most production agents that means hundreds to low thousands of calls, weighted toward risky flows and edge cases rather than the easy happy path you are already confident about.

What is ground truth in voice agent evaluation?

Ground truth is the verified correct answer for each item in the dataset, set by a human rather than the agent under test. For transcription it is the exact words spoken. For a task it is whether the caller's goal was actually met. Ground truth is the reference the agent's output is compared against, so its accuracy caps every score.

Should a golden dataset use real calls or synthetic calls?

Use both. Real calls anchor the set to your actual traffic and keep scores honest, while synthetic callers fill gaps real data barely contains, such as rare emergencies and specific edge conditions. Relying only on real data leaves blind spots, and relying only on synthetic data risks a set too clean to reflect live conditions. The mix makes it representative.

How do you make sure labelers agree on the answers?

Have at least two raters label a shared subset, then measure inter-rater reliability with a statistic like Cohen's kappa, which corrects for chance agreement. Low agreement usually signals a vague rubric rather than careless raters. Sharpen the rule, add examples for gray cases, and relabel. High agreement is what lets a score reflect the agent instead of the grader.

How often should you update a golden dataset?

Update it on two triggers. Refresh when the product changes in a way that adds or alters flows, and refresh on a regular cadence regardless, because caller behavior drifts over time. Cut a new version with a changelog on each update and keep old versions so past results stay reproducible. Feed new production failure modes back in as they surface.

Can you use production call data safely in a golden dataset?

Yes, with care. Redact or mask personal information as you curate calls, keep the set access-controlled, and prefer synthetic stand-ins where a real customer's details are not needed. Treat the dataset like any sensitive asset. Handling it against a recognized privacy and security framework lets you use realistic data while still passing an internal or vendor security review.

The bottom line

A golden dataset is the labeled answer key that makes every voice agent benchmark repeatable and defensible. Build it from representative real and synthetic calls, label it against clear rubrics with rater agreement, size it for coverage, and version it as your product changes.

Want a golden dataset built and scored by a neutral third party? Book a demo to see how Evalgent, as an independent evaluator, builds a benchmark on your calls.