How to Test Knowledge Grounding in Voice Agents

# How to test knowledge grounding in voice agents

Quick answer

To test knowledge grounding in voice agents, prove every answer comes from your approved knowledge base, not the model's own memory. Check that the agent retrieves the right passage, states only what that passage supports, cites its source, and says "I don't know" or transfers when the knowledge base has no answer.

A voice agent can sound confident and still be wrong about your business. It can quote a return window you never published. It can invent a fee. It can answer a question your documents do not cover, in a fluent voice, on a live call. None of this is a speech problem. It is a grounding problem.

Grounding is whether the agent answers from your knowledge, or from its own priors. This guide shows how to test that a voice agent stays anchored to an approved knowledge base. It covers retrieval correctness, grounded versus ungrounded answers, missing and conflicting entries, correct refusal, and staying current when the source changes. It builds on our broader voice agent evaluation guide and narrows to one question: does the answer trace back to a document you approved?

What knowledge grounding means for a voice agent

Most production voice agents do not keep your policies inside the model. They keep them in a knowledge base and fetch the relevant text at answer time. This pattern is retrieval-augmented generation, or RAG, introduced in the 2020 Lewis et al. paper.

> Knowledge grounding: answering from retrieved, approved source text rather than the model's internal memory. A grounded answer can be traced back to a specific passage the agent actually retrieved.

The mechanism has three stages. First, the agent turns the caller's question into a search over your documents, usually with vector search over embeddings. Second, it retrieves the passages that best match. Third, it writes an answer conditioned on those passages. Grounding can break at any stage.

The idea comes from grounding in communication: two speakers building shared, verified understanding. For a voice agent, the shared reference is your knowledge base. If the agent answers without checking it, the reference is gone.

How grounding testing differs from hallucination testing

These two overlap, but they are not the same test. Hallucination testing asks a broad question. Did the agent make something up? Our guide to testing for hallucinations in voice agents covers that general case, including fabricated facts of any kind.

Grounding testing is narrower and more mechanical. It asks whether each answer is supported by a retrieved passage you approved. A hallucination) is any confident falsehood. An ungrounded answer is a specific failure: the agent had a knowledge base and did not use it correctly.

The distinction matters for scoring. A hallucination test flags the false claim. A grounding test also tells you why it happened. Did retrieval miss the passage? Did the agent ignore what it retrieved? Did the source itself have a gap? That diagnosis is what lets you fix the pipeline, not just the symptom.

The grounding failure modes you need to test for

Grounding does not fail in one way. It fails in six. Each has a distinct cause and a distinct test. Treat them as separate cases, because a fix for one rarely fixes another.

The table is the spine of a grounding test plan. Every question in your test set should map to one of these modes. That way a failure tells you exactly which stage of retrieval or generation to inspect.

How to test knowledge grounding in a voice agent

This is the core workflow. Run it as a repeatable suite, not a one-time spot check. The steps move from building a question set to scoring the two things that matter: retrieval and grounding.

1. Build a grounded question set from your knowledge base. Write real caller questions that your documents actually answer. Cover the common paths first, then the edges. Aim for coverage of every major topic in the base.

2. Record the expected source passage for each question. For every question, note which passage holds the correct answer. This is your ground truth. Without it, you cannot score retrieval.

3. Add out-of-scope and unanswerable questions. Include questions the base does not cover. These test refusal, not recall. A good set is roughly a third unanswerable.

4. Seed conflicts and stale entries on purpose. Plant a contradiction between two documents. Keep a copy of an old value. These probe the hardest failure modes.

5. Run the questions through the agent and capture retrieval logs. For each call, log the passages the agent retrieved and the final spoken answer. You need both to diagnose failures.

6. Score retrieval first, then grounding. Check whether the right passage was retrieved. Then check whether the spoken answer is supported by that passage and nothing else.

7. Score refusal on the unanswerable set. For every out-of-scope question, the pass condition is a clean "I don't know" or a transfer, never an invented answer.

8. Re-run the full suite after every knowledge base change. New content changes what the agent can and should retrieve. A grounding suite is only useful if it runs on each update.

Testing retrieval correctness: the right passage

Retrieval is the first place grounding breaks. If the agent fetches the wrong passage, even a perfect writer produces a wrong answer. So test retrieval on its own, before you judge the spoken reply.

The measures come straight from information retrieval. Two matter most here, both from precision and recall. Recall asks whether the correct passage was retrieved at all. Precision asks whether the retrieved set was mostly relevant, not padded with noise.

Score retrieval against the expected passage you recorded in step two. For each question, was the correct passage in the retrieved set? If not, the answer was doomed before generation. Fixing this often means better chunking, better embeddings, or a rewritten source document, not a prompt change.

Watch for near-miss retrieval. The agent pulls a passage about the right topic but the wrong specific case. It retrieves the refund rule for online orders when the caller asked about in-store. The topic matches; the answer does not. Only per-question ground truth catches this.

Testing grounded versus ungrounded answers

Retrieving the right passage is necessary, not sufficient. The agent can still ignore it. Grounded-answer testing checks the link between the retrieved text and the words the caller heard.

The test is a support check. Take the spoken answer, take the retrieved passage, and ask a simple question. Is every claim in the answer supported by that passage? An automated judge or a reviewer reads both and marks each claim as supported, unsupported, or contradicted.

Three outcomes fail. The answer adds a fact the passage does not contain. The answer contradicts the passage. The answer is right by luck but traces to no retrieved source. That last one is subtle and dangerous. A correct answer with no grounding will drift the moment your policy changes, because it was never reading your policy.

This is where independent, source-linked scoring earns its keep. Grounding two ways to the same evidence, spoken answer and cited passage, is the check most internal dashboards skip.

Testing the "I don't know" and transfer paths

The hardest grounding test is the empty case. The knowledge base has no answer. The correct behavior is to say so, or to hand off, and to never improvise.

This is the failure that hurts most in production. An agent that invents a policy on an unanswerable question exposes you to a promise you cannot keep. So the unanswerable third of your set is not filler. It is the most important part.

Score it strictly. For each out-of-scope question, a pass is a clear admission of uncertainty or a transfer to a human. Any confident answer is a hard failure, even if it sounds plausible. Test the boundary too: questions the base almost covers, where the tempting move is to stretch a nearby passage into an answer it does not support.

Watch the refusal wording. "I don't have that information, let me connect you" passes. A vague deflection that leaves the caller stuck does not. Grounding includes knowing, and saying, where your knowledge ends.

Testing citation and source attribution

Grounding is only auditable if you can trace an answer to its source. That is why citation belongs in the test, even on a voice channel where the agent rarely reads a URL aloud.

Attribution can live in two places. The agent can name the source in speech when it helps the caller. More often, it logs the retrieved passage and document ID behind the scenes. Either way, every factual claim should map to a specific source you can inspect after the call.

Test this by requiring a source for each grounded answer in the logs. If a claim has no linked passage, treat it as ungrounded, even if it happens to be true. Untraceable answers cannot be audited, and unauditable answers cannot be trusted at scale.

Keeping grounding tests current as the knowledge base changes

A knowledge base is not static. Prices change. Policies get rewritten. Documents get added and retired. Every one of these edits can change what the agent should retrieve, and can silently break a grounded answer that used to pass.

Version the knowledge base and tie each test run to a version. When content changes, three things can happen. A question that was unanswerable becomes answerable. A correct answer becomes stale. A new conflict appears between old and new documents. A grounding suite that only ran at launch catches none of these.

Make re-running cheap. Testing on your own base and your own updates is the point, and our guide to benchmarking voice agents on your own data goes deeper on building that set. The goal is a suite that runs on every content change, not a report that ages the day it ships.

Grounding on audio, not just transcripts

Most grounding checks read the transcript. That is fine for the words. But a voice agent can be grounded in text and still fail the caller on air. It can read the right passage in a rushed, robotic monotone. It can be interrupted with a correction and plow on with the old, now-wrong answer.

So confirm grounding on the audio for the calls that matter, especially barge-in cases where the caller corrects a fact mid-answer. Our comparison of transcript versus audio evaluation explains the trade-off. A hybrid works: transcript grounding at scale, audio review on the high-risk sample.

Why an independent evaluator strengthens grounding tests

Grounding is easy to grade generously when you built the agent. You know what it was supposed to say, so you read the intended answer into an ambiguous one. This is why grounding results from the vendor who ships the agent deserve scrutiny, a theme in our note on independent voice AI evaluation.

Evalgent is a third-party evaluator that scores grounding against your approved source, not the agent's intent. Every graded answer links to the passage it should have used. That separation, and the source-linked evidence behind each score, is what makes a grounding result something you can defend to a regulator or a customer. For the wider testing picture, see our overview of AI voice agent testing and the difference between testing and evaluation.

Frequently asked questions

How do you test knowledge grounding in a voice agent?

Build a question set from your knowledge base, and record the expected source passage for each question. Run the questions through the agent, log what it retrieved, and check the spoken answer against that passage. Add unanswerable questions to test refusal. Re-run the suite after every knowledge base change.

What is the difference between grounding and hallucination in voice agents?

Hallucination is any confident falsehood the agent produces. Ungrounded answering is a specific cause: the agent has an approved knowledge base and fails to answer from it. Grounding testing checks whether each answer traces to a retrieved passage. It diagnoses why a wrong answer happened, not just that it happened.

How do you test that a voice agent retrieves the right passage?

Record the correct source passage for each test question in advance. Run the question, then log which passages the agent actually retrieved. Compare the two using recall and precision. If the correct passage was never retrieved, the answer failed before generation, and the fix is usually in chunking, embeddings, or the source document.

How do you test a voice agent when the knowledge base has no answer?

Include out-of-scope questions the base does not cover, roughly a third of your set. The pass condition is a clear "I don't know" or a transfer to a human. Any confident answer is a hard failure, even a plausible one. Also test boundary questions the base almost covers, where the agent may overreach.

How do you test grounding when the knowledge base is updated?

Version the knowledge base and link each test run to a version. After any content change, re-run the full suite. Watch for three shifts: unanswerable questions that became answerable, correct answers that went stale, and new conflicts between old and new documents. A suite that only ran at launch will miss all three.

What is retrieval-augmented generation in a voice agent?

Retrieval-augmented generation, or RAG, means the agent fetches relevant text from your knowledge base at answer time and writes its reply from that text. It turns the question into a search, retrieves matching passages, then generates a grounded answer. This keeps policies out of the model and in documents you control and can update.

How do you test citation and source attribution in a voice agent?

Require every grounded answer to link to a specific source passage in the logs, even when the agent does not read a source aloud. During review, confirm each factual claim maps to a retrieved passage. If a claim has no linked source, treat it as ungrounded, even if it is correct. Untraceable answers cannot be audited.

How do you test grounding on audio calls, not just transcripts?

Transcript checks catch the words but miss delivery and turn-taking. Confirm grounding on audio for high-risk calls, especially barge-in cases where a caller corrects a fact mid-answer and the agent should update. A hybrid works: score grounding on transcripts at scale, then review audio on a targeted sample of the calls that matter.

The bottom line

Knowledge grounding testing proves a voice agent answers from your approved knowledge base, not its own memory. Test retrieval correctness, grounded answers, correct refusal, and staying current after every knowledge base change.

Ready to see how your agent grounds its answers against your own knowledge base? Book a demo and get a source-linked grounding report you can defend.