How to test tool calling in AI voice agents

Tool calling is where a voice agent stops talking and starts doing. It is also where production incidents concentrate. Evalgent tests this layer directly. Here is what breaks, and how to catch it before a caller does.

Tool calling: the mechanism that lets a voice agent's LLM invoke an external function during a call. The model emits a function name plus named arguments, then uses the result to continue the conversation.

What tool-calling failures look like

These are the symptoms that show up on a real call:

• The agent says it acted — "You're all booked" — but no function ran.
• It calls the wrong tool, running a lookup when the caller asked to cancel.
• The arguments are wrong: a misheard date, a made-up account number, the wrong currency.
• It stalls or talks over the caller while a slow tool runs.
• A tool errors, and the agent invents a confident answer instead of recovering.

The dangerous part is that most of these are invisible in the transcript. The agent sounds fine. The action behind it failed.

Why tool calling fails

Under real audio, the biggest failure mode is arguments, not tool choice. Research on realtime voice models found argument-value errors account for 54–57% of tool-calling failures. The model gets the shape of the call right. It just mis-hears the exact value from speech. The table maps each common cause to what you would hear and how to fix it.

How to test tool calling in AI voice agents

1. Map the tools — List every function, its required arguments, and the caller intents that should trigger it.

2. Write a scenario per tool path — One realistic call that must fire each tool. Add edge cases: wrong intent, missing info, ambiguous request.

3. Assert the call, not the words — Check that the correct function fired with the correct arguments. Do not trust the transcript's claim of success.

4. Vary the caller — Run accents, noise, and fast speech, so misheard-argument failures actually surface.

5. Test the failure path — Force tool errors and timeouts. Assert the agent recovers or escalates instead of inventing an answer.

6. Gate every release — Re-run the suite on each prompt or model change. Block on any drop in tool-call accuracy.

A worked example

A caller says, "Move my Friday appointment to next Tuesday at 10." The agent should call `reschedule_appointment` with the booking ID and a normalized ISO datetime. In one test, a fast model chose the right tool. But it passed "next Tuesday" as a raw string. The downstream API rejected it quietly and kept the old slot. The agent still told the caller it was done. Nothing in the transcript looked wrong. Only an argument-level assertion, checking the exact payload, caught it.

Testing tool calling with Evalgent

Evalgent drives realistic conversations and checks the tool layer, not just the transcript. Scenarios push the agent down each tool path, including the wrong-intent and missing-argument edges. Profiles vary caller voice, accent, and noise, so misheard arguments show up the way they do in production. Metrics assert the right function fired with the right arguments, with pass/fail thresholds on tool-call accuracy. Evaluations run the whole suite as batches of synthetic callers before release. Reviews let you replay a failing call with the audio, transcript, and exact tool payload side by side. For the wider discipline, see the AI voice agent testing pillar. For model selection, see best LLM for voice agents.

The bottom line

Tool calling is where a voice agent stops talking and starts acting, so it is where testing matters most. Assert the exact function and arguments on every tool path — not the transcript — and gate releases on tool-call accuracy.

Frequently asked questions

What is tool calling in a voice agent?

Tool calling, also called function calling, is how a voice agent takes actions. The LLM emits a structured call — a function name plus named arguments — to book, look up, or transfer, then uses the result to continue. It is the same mechanism used in text agents, running live inside a spoken call.

What is the difference between tool calling and function calling?

There is no real difference. The two terms describe the same mechanism: an LLM invoking an external function with structured arguments. "Function calling" is the older API term; "tool calling" is the newer, broader one. In a voice agent, both mean the model taking an action mid-call rather than only speaking.

Why do voice agents call the wrong tool?

Usually because tool descriptions are vague or overlap, so the model cannot tell two functions apart. A weak prompt, missing negative examples, or an ambiguous request make it worse. Tightening each tool's description and testing every intent path is the most reliable fix.

How do you test that a voice agent calls tools correctly?

Write a scenario for each tool path, drive it with a realistic call, and assert that the correct function fired with the correct arguments — not that the agent claimed success. Vary accent and noise to surface misheard values, and re-run the suite on every prompt or model change.

Why do voice agents pass the wrong arguments to a tool?

Because the argument is spoken, and speech is noisy. Under real audio, argument-value errors are the most common tool-calling failure — the model picks the right function but mis-hears the date, number, or name. Confirming values back to the caller and testing across accents and noise are the main defenses.

How do you stop a voice agent from calling a tool at the wrong time?

Gate tool calls on end-of-turn detection, so the agent does not fire mid-utterance and cut the caller off. For slow tools, stream a short filler line while the call runs. Then test timing directly, asserting the tool fires at the right point in the conversation.

What happens when a tool call fails silently?

The tool errors or rejects the input, returns nothing useful, and the agent tells the caller the action succeeded anyway. These are the costliest failures because they are invisible in the transcript. Surfacing tool errors back to the model, and asserting outcomes in testing, are what expose them.

Should a voice agent confirm before calling a tool?

For high-stakes actions — payments, cancellations, transfers — yes. A quick confirmation of the key argument catches misheard values before they cause damage. For low-risk lookups, confirmation adds friction and is usually skipped. Test both paths so the agent confirms when it should and does not stall when it should not.