How to Evaluate a Realtime Voice API

# How to evaluate a realtime voice API

Quick answer

> Quick answer: To evaluate a realtime voice API, test it as a product, not just a model. Run your real call scenarios over a live stream and measure time-to-first-audio, end-to-end latency under load, barge-in handling, mid-stream tool calls, session state, reconnection, rate limits, and pricing shape.

A realtime voice API is the interface you build a voice agent on. It is not the model. The model decides what the agent says. The API decides how fast, how reliably, and how flexibly that response reaches your caller. You can pick a great model and still ship a bad agent if the transport layer fails under load.

This guide is about the interface. Our guide to evaluating a speech-to-speech model covers output quality: task success, naturalness, and prosody. This post covers the plumbing around it. Connection model, streaming stability, interruption handling, tool calls, sessions, reliability, and cost.

Evalgent is an independent evaluator. We stress-test a realtime voice API on your own call scenarios, without favoring any provider.

What is a realtime voice API?

> Realtime voice API: a streaming interface that accepts live audio and returns audio or events over a persistent connection, with latency low enough to hold a natural conversation.

A batch API takes a request and returns a full response. A realtime voice API is different. It keeps a connection open and exchanges audio in both directions as the call happens. Many modern realtime voice providers use a speech-to-speech design, so audio flows in and audio flows out with no visible text layer.

That streaming shape changes what you have to test. You are no longer scoring a single response. You are scoring a live, stateful, two-way session that has to survive network trouble, interruptions, and concurrency. The unit of evaluation is the whole call, not the reply.

Why the API matters as much as the model

Teams often evaluate the model and skip the interface. That is a mistake. Two providers can wrap similar models and still behave nothing alike in production.

One holds the connection open cleanly. The other drops sessions after a few minutes. One streams the first audio in 300 milliseconds. The other buffers a full sentence before it speaks. One lets you call a tool mid-turn. The other forces you to end the turn first. These are API decisions, and callers feel every one of them.

So the evaluation splits into two lenses. The model lens asks whether the agent said the right thing. The API lens asks whether the interface delivered it fast, intact, and under load. This post is the second lens. For the first, see our voice agent evaluation pillar.

The transport model: WebSocket or WebRTC

The connection model is the first thing to check. Most realtime voice APIs use one of two transports.

A WebSocket is a persistent, bidirectional channel over TCP. It is simple to integrate from a server and easy to log. The tradeoff is that TCP retransmits lost packets, which can add delay when the network degrades. The MDN WebSocket reference documents the client side well.

WebRTC is built for realtime media. It runs audio over UDP, tolerates packet loss, and adapts to changing conditions. It is more complex to set up but tends to hold up better on bad networks. The MDN WebRTC API docs cover the details.

There is no universal winner. WebSocket is fine for server-to-server calls on stable networks. WebRTC is stronger for last-mile audio to phones and browsers with jitter. Evaluate the transport your actual traffic will use, not the demo path.

Latency: measure it under load, not on a demo

Latency is where realtime voice APIs win or lose. A calm demo hides the truth. You have to measure the numbers that matter, and you have to measure them while the system is busy.

Two figures drive the caller experience. Time-to-first-audio is the gap from when the caller stops talking to when the agent starts. End-to-end latency is the full round trip through your stack. A streaming API should begin emitting audio before the full response is ready, so the caller hears speech sooner.

Never report a single average. Report percentiles. The median can look fine while the 95th percentile ruins one call in twenty. Measure p50, p95, and p99 at realistic concurrency, because latency climbs as load rises. Our guide to latency in voice agents and our post on reducing STT latency go deeper on the budget.

Interruption and barge-in over the stream

Real callers interrupt. A realtime voice API has to handle that gracefully, and how it does so is an API concern, not just a model one.

> Barge-in: the ability of a caller to interrupt the agent mid-speech and have the agent stop and listen.

When the caller talks over the agent, the API should stop the outbound audio quickly and route the new input for processing. Watch for two failures. The agent keeps talking over the caller. Or the agent stops but loses the thread of what it was doing. Both wreck the conversation.

Test barge-in on the live stream, repeatedly, across many turns. Measure how fast the outbound audio cuts off. Our barge-in vs turn-taking guide explains the distinction, and turn-taking is the underlying conversational mechanic you are testing against.

Tool and function calling mid-stream

Most useful voice agents look things up. They check an order, book a slot, or read a balance. A realtime voice API has to support tool calls without breaking the conversation.

The key question is whether the API can emit a tool call in the middle of a turn and then resume speaking with the result. A weak interface forces the turn to end first, which adds a dead pause. A strong one lets the agent say "let me check that" and keep the line warm while the tool runs.

Test this with real tools and real latency. Add a slow tool on purpose. See whether the agent fills the silence or goes quiet. Our tool calling guide covers the patterns and pitfalls.

Session, state, and context handling

A call is stateful. The API holds context across many turns, and how it manages that state matters.

Check how long a session can stay open. Check whether context persists correctly across a long call. Check what happens when you update instructions or inject data mid-session. Some APIs let you change the system prompt or push new context live. Others freeze it at connection time. If your use case needs mid-call updates, test them explicitly.

Also check the failure mode. When a session hits its limit or times out, does the API warn you, or does the call just die? A clean, observable session boundary is far better than a silent drop.

Reliability, reconnection, and observability

Networks fail. The realtime voice API you pick has to fail well, because production traffic is not a clean lab.

Test reconnection directly. Drop the connection mid-call and see what happens. Does the session resume, or start over? Does audio in flight get lost? A strong API supports resumption or at least a fast, clean reconnect. A weak one loses the caller.

Observability decides whether you can debug later. After a failed call, check what you can reconstruct. Good realtime voice APIs expose events, timestamps, and reason codes for disconnects. If a dropped call leaves you nothing to inspect, you cannot fix the pattern. This is a core reason to benchmark providers on your own data, not their marketing.

Rate limits, concurrency, and pricing shape

Two operational questions decide whether an API survives your traffic and your budget.

First, concurrency. Find the documented limits on simultaneous sessions and requests. Then test near them. Many realtime voice APIs enforce rate limiting, and behavior at the ceiling differs. Some queue. Some reject with a clear error. Some degrade quietly, which is the worst case, because latency creeps up with no signal.

Second, pricing shape. Realtime pricing is rarely a flat per-minute number. It may bill input audio, output audio, and tool usage separately. It may charge for idle connection time. Model your real traffic mix, then compute cost per call, not cost per token. Our orchestration comparison covers how these costs stack across a full pipeline.

Realtime voice API evaluation criteria

Use this table as a scorecard. For each criterion, note what a strong API looks like and the red flag that should stop a purchase.

How to run a realtime voice API evaluation

Do not judge a realtime voice API from a vendor demo. Build a small harness and put the API through your own calls. Here is the sequence.

1. Collect real scenarios. Pull 20 to 50 representative calls from your traffic. Include hard names, digits, interruptions, and edge cases. These become your fixed test set.

2. Build a test harness. Write a client that opens the connection, streams your scenario audio in, and captures everything out: audio, events, and timestamps. Reuse it across every provider.

3. Replay each scenario. Play your recorded calls through the API as if they were live. Trigger barge-in at set points. Fire tool calls with realistic delays. Keep the inputs identical across providers.

4. Measure the timing. Record time-to-first-audio and end-to-end latency for every turn. Compute p50, p95, and p99. Do this at single-call load first, then at your target concurrency.

5. Stress the connection. Drop and resume sessions mid-call. Run long calls to hit session limits. Push past the rate limit on purpose and watch how the API responds.

6. Score reliability and behavior. Log dropped sessions, failed reconnects, and lost audio. Grade barge-in speed and tool-call recovery by listening to the captured audio.

7. Model the cost. Apply real pricing to your traffic mix. Compute cost per call, including idle time and tool usage, not just a headline rate.

8. Rerun on every provider and every change. Run the same scenarios through each candidate, and rerun after any provider update. Only identical inputs give a fair comparison.

An independent evaluator runs this harness for you and reports the numbers without a stake in the result. That is the role Evalgent plays.

Frequently asked questions

What is a realtime voice API?

A realtime voice API is a streaming interface that accepts live audio and returns audio or events over a persistent connection, fast enough for natural conversation. Unlike a batch API, it keeps the connection open and exchanges data both ways during the call. Many realtime providers use a speech-to-speech design, so audio flows in and audio flows out.

How do you evaluate a realtime voice API before building on it?

Test it as a product on your own call scenarios, not on a vendor demo. Build a harness that replays real calls, then measure time-to-first-audio and latency percentiles under load. Check barge-in, mid-stream tool calls, session handling, reconnection, rate-limit behavior, and pricing shape. Rerun the same scenarios across every provider you compare.

Is WebSocket or WebRTC better for a realtime voice API?

Neither wins universally. WebSocket is a persistent TCP channel that is simple to integrate and log, but retransmissions can add delay on poor networks. WebRTC runs audio over UDP, tolerates packet loss, and adapts to bad conditions, at the cost of setup complexity. Choose based on your real traffic path: server-to-server or last-mile to phones and browsers.

How do you measure time-to-first-audio for a realtime voice API?

Time the gap from when the caller stops speaking to when the agent's first audio plays. Capture it per turn across many calls, then report p50, p95, and p99, never a single average. Measure at realistic concurrency, because latency climbs under load. A streaming API should begin emitting audio before the full response is ready.

Does a realtime voice API support barge-in and interruptions?

Support varies, so test it directly. Talk over the agent mid-speech and check that the outbound audio stops quickly and the new input is processed. Watch for two failures: the agent keeps talking over the caller, or it stops but loses its place. Measure cutoff speed on the live stream across many turns, not once.

Can you call tools or functions mid-stream with a realtime voice API?

Some can, some cannot, and it matters a lot. A strong API emits a tool call in the middle of a turn and resumes speaking with the result, so the agent can say "let me check" without a dead pause. A weak one forces the turn to end first. Test with real tools and deliberately slow responses.

How do you test reliability and reconnection for a realtime voice API?

Drop the connection mid-call and observe the result. A strong API resumes the session or reconnects fast and cleanly. A weak one restarts the call or loses in-flight audio. Run long calls to hit session limits and confirm you get a clear warning, not a silent drop. Check what events and reason codes you can inspect afterward.

How is a realtime voice API priced?

Realtime pricing is rarely a flat per-minute rate. Providers often bill input audio, output audio, and tool usage separately, and some charge for idle connection time. Model your real traffic mix and compute cost per call rather than cost per token. Compare that figure across providers, since headline rates hide the components that dominate a real bill.

The bottom line

A realtime voice API must be evaluated as a product, not a model. Run your own calls through a harness and measure latency under load, barge-in, tool calls, reliability, and cost.

Ready to see how a realtime voice API holds up on your calls? Book a demo and get an independent evaluation on your own data.