How to Test Function Schema Validation in Voice Agents

# How to test function schema validation in voice agents

A voice agent hears a caller, decides to act, and emits a structured tool call. That call has to match a declared contract before any system runs it. When the contract breaks, the call fails at the door or, worse, slips through half-formed. This post shows how to test function schema validation in voice agents so those breaks surface in a test suite, not a live call.

Quick answer

Function schema validation tests whether a voice agent's tool call conforms to its declared schema before execution: required fields present, correct types, valid enums, right formats, no extra or hallucinated parameters, and well-formed JSON. You test it by validating each generated call against the schema and asserting on the specific violation.

What function schema validation actually checks

Every tool a voice agent can call ships with a schema. The schema is a contract. It names the parameters, marks which are required, sets each type, lists allowed enum values, and often pins formats like dates or currency. This is a form of data validation applied to machine-generated input.

Schema validation asks one question. Does the call the agent produced match that contract? It does not ask whether the values are correct for the caller. It asks whether the payload is even shaped right.

The model builds the call from messy speech. A caller says a date, an amount, a name, a choice. The agent maps that into a rigid structure. The mapping can go wrong in ways that have nothing to do with hearing the caller correctly.

Most function schemas today are expressed in JSON Schema, the same standard that governs many API request bodies. That gives you a precise, machine-readable contract to validate against. If you know the schema, you can check any call automatically.

Why schema-invalid calls fail before they help

A schema-invalid call has two fates, and both are bad.

The first fate is a hard throw. The tool runtime rejects the payload, the call errors out, and the turn stalls. The caller hears silence or a fallback line. Nothing happened.

The second fate is quieter. A loose runtime coerces or drops fields and runs anyway. A missing amount becomes zero. An unknown enum falls to a default. The call succeeds on paper and does the wrong thing in the system of record.

Both fates start upstream of the values. The agent could have every value right and still ship a broken envelope. That is why schema conformance is its own test surface. It sits between the model and the runtime, and it catches faults that value checks never see.

This is the design by contract idea applied to voice. The tool publishes a contract. The agent must honor it. A test suite is how you prove the contract holds under real speech.

How schema validation differs from argument accuracy and silent failures

These three checks look alike and test different things. Keeping them separate keeps your suite honest.

Schema validation checks the shape. Is the payload well-formed and conformant? A call can be perfectly shaped and still carry the wrong booking date. That wrong date is a job for tool-argument accuracy, which checks the values extracted from speech.

Argument accuracy checks the values. Given a valid shape, are the numbers, names, and dates right? It assumes the envelope already passed schema checks.

Silent tool failure checks execution. The call left the agent well-formed and correct, the tool ran, and it failed anyway, yet the agent told the caller it worked. That is covered in detecting silent tool failures.

Schema validation is the earliest gate of the three. It runs before execution. Nothing downstream matters if the payload never conforms. For the wider framing of these gates, see the guide on tool calling in voice agents.

The schema violation types you must test

Schema faults cluster into a handful of categories. Each maps to a rule in the contract. Each needs its own assertion.

Missing required fields. The schema marks a parameter required and the agent omits it. This is the most common fault and the most dangerous, because loose runtimes fill the gap with a default.

Wrong types. A field typed as a number arrives as a string, or a boolean arrives as the word "yes." A strong type system rejects this; a permissive one coerces it silently.

Invalid enum values. The schema lists allowed choices and the agent invents one outside the set. Enums are an enumerated type, so any value off the list is invalid by definition.

Bad formats. The value is the right type but the wrong shape. A date is not ISO 8601. A currency amount carries a symbol the schema forbids. A phone number keeps spoken punctuation.

Hallucinated or extra parameters. The agent adds fields the schema never declared. If the schema sets `additionalProperties` to false, the call is invalid. If it does not, the extra field rides along and confuses the tool.

Malformed JSON. The payload is not well-formed at all. A truncated string, an unescaped quote, or a trailing comma breaks the parser before any field check runs.

Each of these fails independently of whether the caller was understood. A schema conformance suite exercises all six against real speech input.

Schema violation, example, and how to test it

The table maps each violation to a concrete example and the test that catches it. Use it to structure a schema conformance suite.

How to test function schema validation step by step

Build the suite once and run it on every candidate build. The steps below turn the contract into repeatable checks.

1. Collect the tool schemas. Export the exact schema for every tool the agent can call. Treat it as the source of truth. If the schema drifts from what the runtime enforces, your tests validate the wrong contract.

2. Assemble a speech corpus. Gather audio or transcripts that should trigger each tool. Include clean cases, edge cases, ambiguous phrasing, and inputs that stress formats like dates and amounts.

3. Capture the raw tool call. Run each input and record the exact payload the agent emits, before any runtime coercion. You need the call as generated, not as cleaned up.

4. Parse for well-formedness first. Before checking fields, confirm the payload parses. A malformed payload fails here and skips the field-level checks.

5. Validate against the schema. Run each parsed call through a schema validator in strict mode. Turn off silent coercion so a string in a number field fails instead of passing.

6. Assert on the specific violation. Do not record a generic pass or fail. Record which rule broke: missing field, wrong type, bad enum, bad format, extra field. The category tells you where to fix the prompt or the tool definition.

7. Check for hallucinated fields. Compare the call's keys against the schema. Flag any key the contract never declared, even when the runtime tolerates it.

8. Score and gate. Compute a conformance rate per tool and per violation type. Set a release gate on the rules that carry real risk, and block builds that regress.

Run this suite in continuous integration so every prompt change and model swap re-checks conformance automatically.

Turning speech into a conformant call

The hard part is that the input is spoken, not typed. A form field constrains a human. A microphone does not. The agent has to translate open speech into a closed contract on every turn.

Consider a date. A caller says "the fifteenth." The agent must resolve the month, the year, and the ISO format, all from two words and prior context. Any step can produce a value that is the right day but the wrong shape.

Consider an amount. A caller says "nineteen ninety-nine." Is that a price of 19.99 or a year? The schema wants a number with two decimal places. The agent has to strip the words, pick the reading, and format the result.

Consider a choice. A caller says "the premium one." The enum lists `basic`, `plus`, and `pro`. There is no `premium`. The agent has to map speech onto the allowed set or fail conformance.

These pressures are why schema faults spike on voice input in ways they never do on typed forms. Your test corpus has to reproduce that pressure, not sanitize it away.

Strict validation versus permissive runtimes

A quiet risk hides in your runtime. Many tool layers are permissive by default. They coerce a string to a number, drop an unknown enum to a default, or ignore extra fields. That permissiveness masks faults in production and, if you are not careful, in your tests too.

Test in strict mode even if production runs permissive. Strict validation shows you the true conformance rate of the raw call. Then you can decide, per field, whether permissive handling is acceptable or whether it is silently corrupting data.

The gap between strict and permissive results is itself a finding. A high strict-mode failure rate with a low production error rate means the runtime is papering over broken calls. That is a fragile setup. A schema tweak or a runtime change can turn a masked fault into an outage.

Scoring and gating on schema conformance

Report conformance as a rate, not a single number. Per tool, track the share of calls that pass strict validation. Per violation type, track how often each rule breaks. This tells you whether the problem is enums, formats, or missing fields.

Weight the gate by risk. A missing required field on a payment tool matters more than an empty optional note on a lookup tool. Set hard gates on the high-risk rules and softer thresholds on cosmetic ones.

Because you are the one declaring the contract, this is a place where independent evaluation pays off. A neutral suite validates the calls a vendor's own dashboard may never surface. Evalgent, as an independent evaluator, runs these conformance checks against your schemas and your speech, so the results reflect your contracts rather than a vendor's defaults.

Fold the schema suite into your broader test plan. It complements value and execution checks rather than replacing them. For the full picture of what belongs in a program, see voice agent evaluation and the primer on AI voice agent testing.

Common mistakes that hide schema faults

Teams miss schema faults for predictable reasons. Watch for these.

Testing only the happy path leaves the edge cases that break formats unexercised. Validating the coerced call instead of the raw call hides type faults the runtime silently fixed. Checking values before checking shape lets a malformed payload masquerade as a value error.

Another trap is testing against a stale schema. If your tool definition changed but your test schema did not, you are proving conformance to a contract that no longer exists. Pin the schema to the same source the runtime uses.

Finally, do not conflate a schema pass with a correct call. A conformant payload can still carry the wrong value. Run conformance, accuracy, and execution checks as three distinct gates, and read them together. The guide on testing versus evaluation explains why these layers stay separate.

Frequently asked questions

What is function schema validation for voice agents?

It is the test of whether a voice agent's tool call conforms to its declared schema before execution. The check confirms required fields are present, types match, enums are valid, formats are correct, no extra parameters appear, and the JSON is well-formed. It tests the shape of the call, not the accuracy of the values inside it.

How do you test that a tool call matches its schema?

Capture the raw call the agent emits, parse it for well-formedness, then validate it against the tool's schema in strict mode. Assert on the specific rule that breaks, such as a missing field or an invalid enum. Record a conformance rate per tool and per violation type across a speech corpus that includes edge cases.

What is the difference between schema validation and argument accuracy?

Schema validation checks the shape of the call: fields, types, enums, formats, and well-formedness. Argument accuracy checks the values inside a valid shape, such as whether the booking date matches what the caller said. A call can pass schema validation and still fail on accuracy. Run both as separate gates.

Why does a voice agent produce a schema-invalid tool call?

Because it translates open speech into a closed contract on every turn. Spoken dates, amounts, and choices do not arrive in the schema's format. The model must resolve and reshape them, and any step can drop a required field, pick the wrong type, or invent an enum value the contract never allowed.

How do you test enum values in a tool call?

Assert that each enum field's value is a member of the allowed set defined in the schema. Feed inputs where callers use synonyms or off-list phrasing, such as "premium" when the enum lists basic, plus, and pro. A robust agent maps speech onto the allowed set or fails conformance rather than inventing a value.

How do you catch hallucinated parameters in a tool call?

Compare the keys in the generated call against the schema's declared properties. Flag any key the contract never declared. When the schema sets additionalProperties to false, treat extra fields as hard failures. Do this even if your runtime tolerates extra fields, since a tolerant runtime can still pass corrupt data to the underlying tool.

How do you test date and currency formats in tool calls?

Validate the value against the declared format, not just the type. For dates, check the payload is a valid ISO 8601 string, not spoken text like "next Tuesday." For currency, confirm the number has the required decimal places and no forbidden symbols. Include spoken edge cases in the corpus so formatting faults surface.

Should you test in strict mode if production is permissive?

Yes. Strict validation reveals the true conformance rate of the raw call before any runtime coercion hides faults. Compare strict-mode results with production behavior. A large gap means the runtime is masking broken calls, which is fragile. Strict testing lets you decide, per field, whether permissive handling is safe or silently corrupting data.

The bottom line

Schema validation is the first gate a tool call must pass. Test conformance before values and execution, and broken calls surface in your suite instead of a live call.

Ready to see whether your agent's tool calls conform under real speech? Book a demo and Evalgent will run schema conformance checks against your own schemas and calls.