How to test for hallucinations in AI voice agents

A hallucination in a voice agent is worse than in a chatbot. There is no link to check and no text to reread — the caller just hears a confident, wrong answer and acts on it. Evalgent tests for exactly this. Here is how hallucinations happen and how to catch them.

Hallucination: a confident, fluent answer that is not grounded in fact — the agent states wrong information, or invents an answer, instead of admitting it does not know.

What a hallucination sounds like

These are the symptoms on a real call:

• The agent quotes a price, policy, or hours that are simply wrong.
• It invents a step or a product that does not exist.
• It answers an out-of-scope question instead of deflecting.
• It agrees with a false premise a caller states.
• It fills a gap with a plausible guess rather than saying it does not know.

The danger is confidence. The agent sounds exactly as sure when it is wrong as when it is right.

Why voice agents hallucinate

Hallucination is usually a grounding problem, not a model defect. The model fills gaps with plausible text because nothing forces it to check a source or refuse. The table maps each cause to what you hear and how to fix it.

How to test for hallucinations in AI voice agents

1. Build a fact set — List the true answers to your top questions from a trusted source of record.

2. Write adversarial scenarios — Ask out-of-scope, ambiguous, and false-premise questions designed to tempt a guess.

3. Check against source — Assert each answer matches the fact set, not just that it sounds reasonable.

4. Test the refusal path — Confirm the agent says it does not know, or escalates, when the answer is not available.

5. Judge carefully — Use an LLM judge for groundedness with caution, since it grades language, not truth.

6. Re-run on every change — Re-test after prompt, model, or knowledge-base updates, because any can reintroduce drift.

A worked example

A caller asked whether a plan covered a specific procedure. The knowledge base did not mention it, so the agent should have deflected. Instead it said, "Yes, that's covered" — a fluent, confident guess. The transcript looked like a normal, helpful answer. Only checking the response against the source of record, and flagging any claim with no grounding, exposed the hallucination before it reached real callers.

Testing for hallucinations with Evalgent

Evalgent tests groundedness on realistic calls, not just tidy transcripts. Scenarios ask the out-of-scope, ambiguous, and false-premise questions that tempt a guess. Profiles vary caller persona and phrasing so the agent faces leading questions the way real callers ask them. Metrics assert answers against a source of record and flag ungrounded claims, with thresholds you set. Evaluations run the adversarial suite as batches of synthetic callers before release. Reviews let you replay a bad call and hear the confident, wrong answer in context. On why automated judging alone is not enough, see the limits of LLM-as-judge. For the wider method, see the AI voice agent testing pillar.

The bottom line

A voice agent hallucinates when it answers confidently without grounding, and the caller cannot see the mistake — they only hear certainty. Test with adversarial and out-of-scope questions, check every answer against a source of record, and make the agent refuse when it does not know.

Frequently asked questions

What is a hallucination in a voice agent?

A hallucination is a confident, fluent answer that is not grounded in fact. The agent states wrong information — a made-up policy, price, or step — or invents an answer instead of admitting it does not know. In voice, it is especially risky because the caller has no link to verify and often acts on the answer immediately.

Why do voice agents hallucinate?

Usually because nothing forces the model to check a source or refuse. Without retrieval, a clear scope, or a fallback for unknowns, the model fills gaps with plausible text. Vague prompts, stale knowledge bases, and leading caller questions all make it worse by giving the model room to guess.

How do you test a voice agent for hallucinations?

Build a fact set of true answers from a trusted source, then ask out-of-scope, ambiguous, and false-premise questions designed to tempt a guess. Assert each answer against the fact set rather than judging whether it sounds reasonable, and confirm the agent refuses or escalates when the answer is not available.

How do you stop a voice agent from making things up?

Ground it. Add retrieval so it answers from a current source rather than memory, constrain its scope in the prompt, and give it an explicit refusal path for unknowns. Then test the refusal path directly, so the agent reliably says it does not know instead of guessing.

Can you use an LLM to detect hallucinations?

You can, but with caution. An LLM judge can flag answers that are unsupported by a source, which is useful at scale. But it grades language, not truth, and can miss confident errors or invent its own. Pair automated judging with checks against a source of record and human review of flagged calls.

What is the difference between a hallucination and a wrong answer?

A wrong answer can come from bad data or a misheard input. A hallucination is specifically an answer the model fabricated with no grounding — it sounds authoritative but traces to nothing. The fix differs too: wrong answers often need better inputs, while hallucinations need grounding, scope limits, and a refusal path.

Do grounded voice agents still hallucinate?

Less, but not never. Retrieval reduces hallucination sharply, yet an agent can still misread a source, answer beyond what the source says, or drift on an out-of-scope question. Grounding lowers the rate; testing keeps it low. Re-run adversarial scenarios after every prompt, model, or knowledge-base change.

How often should you test for hallucinations?

Test before every release, and re-run after any change to the prompt, model, or knowledge base — each can reintroduce drift. Treat groundedness as a release gate, not a one-time check. Production monitoring helps too, since new caller questions surface gaps your original fact set never covered.