Voice agent observability tool: a 14-point buyer's and builder's checklist

Most observability tools were designed for text. A request goes in, a completion comes out, and the trace is a tidy tree of LLM calls. A phone call isn't shaped like that. It's two audio streams running at the same time over three or four network legs. Your caller can talk over the agent. Your STT can invent words during silence. And the latency your dashboard shows isn't the latency your caller hears.

This is a checklist for the engineer who owns a LiveKit or Pipecat voice agent. Use it to grade a tool in a demo, or as a spec for your own build on OpenTelemetry, Prometheus and Grafana. Each item has a priority, a pass condition and a quick test. It's vendor-neutral on purpose.

If you're still sorting out where observability ends and testing begins, read voice agent testing vs monitoring vs observability first. For the exact fields to log, see what to log on every voice agent call.

Why voice agent observability is not LLM observability

LLM observability treats the model call as the unit of work. Voice observability has to treat the turn as the unit of work, and a turn exists in time and in audio, not just in text. There are four structural differences.

1. Audio is the ground truth, and the transcript is a lossy copy of it. When a call goes wrong, the transcript is the first thing people read. But the transcript comes out of your STT, which can fail. In Careless Whisper (Koenecke et al., FAccT 2024), the authors found that roughly 1% of Whisper transcriptions contained entire hallucinated phrases or sentences that were not in the audio. Hallucinations happened more often for speakers with longer stretches of non-vocal audio. Phone calls are full of silence: hold music, a caller looking for a card, a dog barking. A tool that only shows the transcript can't tell you whether the caller said something or the STT invented it. You need the audio, played back at the right moment in the trace.

2. Timing is the product. In a chat app, a 1.5-second response is fine. On a call it feels like the line dropped. The order of events matters too. Did the agent start speaking before the caller finished? Did the caller barge in, and did the agent stop? A list of LLM calls can't answer those questions. You need a timeline with both speakers on it.

3. A call crosses several network legs, and each one has its own failure modes. A typical inbound call goes PSTN carrier, then the SIP provider (Twilio, Telnyx), then a media bridge (SIP to WebRTC, or a Media Streams WebSocket), then your agent. Packet loss on the carrier leg never shows up in your agent's WebRTC stats. LiveKit's transport stats only cover the leg between the SFU and your agent. Twilio's Voice Insights Call Summary reports `carrier_edge` and `sip_edge` metrics separately. A Media Streams call only gets a `carrier_edge`, so the WebSocket leg to your bot isn't measured at all. Our phone audio quality guide walks through every leg.

4. Things overlap. STT keeps streaming while TTS plays. A tool call can run while the agent says "let me check that." The caller can interrupt halfway through a sentence. Your traces need to show work that runs in parallel and work that gets cancelled. A cancelled TTS stream is a normal part of a call, not an error.

The voice agent observability checklist

Here is the scoring model. Each capability gets a priority weight: must = 3, should = 2, nice = 1. Score each one 0 (absent), 1 (partial or manual) or 2 (works on a real call in the demo). Multiply weight by score and add them up. The maximum is 72.

Two rules keep the score honest. First, any must scored 0 disqualifies the tool. A high total can't make up for missing audio playback. Second, only score what you saw working on a call you placed yourself. Item 9 counts with a should weight in the total, but it becomes a disqualifying must if your lines are regulated. Slides and roadmaps score 0. As a starting point, we suggest 55 or above (about 76%) with no zeroed musts as the bar for a tool you'd rely on in production. That threshold is our recommendation, not an industry standard. Tune it to your risk.

The rest of this section explains why each item matters and what usually goes wrong.

1. Per-turn trace with timings

The trace hierarchy that works for cascaded agents is call, then turn, then stt, llm, tool and tts spans. Pipecat's OpenTelemetry tracing uses that exact shape: a conversation span, turn spans with `turn.was_interrupted`, and service spans carrying `metrics.ttfb`. One gotcha: `enable_turn_tracking` defaults to off, so you get no turn spans unless you set it. LiveKit Agents also exports OpenTelemetry traces and exposes per-turn metrics such as `end_of_turn_delay`, `llm_node_ttft`, `tts_node_ttfb` and `e2e_latency` through its data hooks.

What to check in a demo: whether the tool understands turns or just lists LLM calls. If you can't answer "which turn was slow, and which hop made it slow?" in one click, it fails.

2. Stereo audio aligned to the trace

Mono recordings hide overlap. If the caller and agent are mixed onto one channel, you can't tell who spoke first or measure the gap between them. You need two channels and a shared clock offset, so that a span at `t = 12.480 s` lines up with second 12.48 of the audio. Where you record matters too. An agent-side recording doesn't hear what the carrier leg did to the audio. A carrier-side dual-channel recording does. Call recording and metrics for LiveKit and Pipecat covers the tap points and how to align them.

3. Audio signal health

Low input level, clipping and packet loss all make STT worse, and none of them show up in an LLM trace. The minimum set is input level in dBFS per turn, the share of clipped samples, the share of silence, and packet loss and jitter for each leg you can see. Remember that RTCP loss counts only packets that never arrived. Packets that arrive too late and get dropped by the jitter buffer count as `packetsDiscarded` in WebRTC stats. Those packets are just as lost to your STT. Computing level and clipping yourself is cheap:

# Simplified: per-turn audio health from 16-bit PCM (one channel).
import numpy as np

def audio_health(pcm16: bytes, frame_ms: int = 20, sample_rate: int = 16000) -> dict:
    x = np.frombuffer(pcm16, dtype=np.int16).astype(np.float32) / 32768.0
    if x.size == 0:
        return {"rms_dbfs": None, "clip_ratio": None, "silence_ratio": None}
    n = int(sample_rate * frame_ms / 1000)
    frames = x[: len(x) // n * n].reshape(-1, n)
    frame_rms = np.sqrt((frames ** 2).mean(axis=1) + 1e-12)
    frame_dbfs = 20 * np.log10(frame_rms)
    return {
        "rms_dbfs": float(20 * np.log10(np.sqrt((x ** 2).mean()) + 1e-12)),
        "clip_ratio": float((np.abs(x) >= 0.999).mean()),     # samples at full scale
        "silence_ratio": float((frame_dbfs < -50).mean()),     # frames below -50 dBFS
    }

The -50 dBFS silence floor and the full-scale clip test are starting points. Calibrate them on your own calls. Then segment STT accuracy by these values. That's how you find out a carrier or a device type is the problem, not your model.

4. Caller-heard vs agent-side latency

This is the item most tools get wrong, and it's the one your callers notice first.

Agent-side latency starts the clock when the agent decides the caller stopped talking, and stops it when the agent pushes its first audio frame. The caller's experience starts earlier and ends later. Their audio has to cross the carrier, the SIP provider and a jitter buffer before your VAD hears it. Your VAD then needs some silence before it marks end of speech. On the way back, the agent's audio crosses the same legs again before the caller hears it.

Here is a worked example with assumed numbers, not measurements:

In this example the dashboard says 700 ms and the caller waits over a second. That 360 ms gap isn't noise. It's built into how the clock is measured. The ITU-T G.114 planning guide treats 150 ms of one-way delay as the comfortable limit for human conversation. Your network legs use up part of that before your agent does anything.

To measure caller-heard latency, use the carrier-side stereo recording. For each turn, find where the caller channel goes silent and where the agent channel starts speaking, and take the difference. That's the number to alert on. Use the agent-side breakdown to explain it. For hop-by-hop debugging, see debugging LiveKit agent latency and measuring Pipecat latency.

5. Interruption and overlap events

Turn-taking failures are the most common thing callers complain about. There are three to log as span events with timestamps: a real barge-in (the caller interrupts and the agent stops), a false interruption (a cough or "uh-huh" stops the agent), and agent-over-caller overlap (the agent starts talking before the caller finishes). LiveKit emits an `agent_false_interruption` event and can resume speech after one. Pipecat marks interrupted turns with `turn.was_interrupted`. A tool should show interruption rate by version, because an endpointing change can double it overnight. LiveKit false interruptions explains the settings involved.

6. Tool call inputs, outputs and status

Tool calls are where a voice agent actually does things, like booking, refunding or transferring. The OpenTelemetry GenAI conventions define `execute_tool` spans with `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result`. Capture all four, plus latency and a status that separates `empty` from `ok`. An API that returns `200` with an empty list is the classic silent failure. The agent then says "I don't see any appointments" with total confidence. See detecting silent tool failures.

7. Version stamps on every call

Without versions, you can't answer "did Tuesday's change cause this?" Put the agent build, prompt hash, LLM model, STT model and settings, TTS model and voice, and framework version on the root span. OpenTelemetry has `gen_ai.agent.version` and `gen_ai.prompt.name`. Use your own attribute for the prompt hash. Pin the framework version for a specific reason: LiveKit Agents 1.7.0 renamed 12 span attributes to an `lk.pii.` prefix, and queries on the old names quietly return nothing. Shipping voice agent prompt changes safely shows how versions feed release gates.

8. PII redaction and access control

Voice spreads PII further than chat does. A spoken card number ends up in interim STT results, the LLM context, span attributes, logs, recordings and tool arguments. LiveKit's `set_tracer_provider(..., allow_pii=False)` strips `lk.pii.*` and GenAI content attributes from exported spans, but it only covers spans. Its observability docs also note that the built-in Agent Insights, including its PII redaction, only work for LiveKit Cloud projects. They don't cover fully self-hosted deployments. If you self-host, redaction is your job. The six places PII leaks, and how to test each one, are in voice agent PII leakage detection.

9. Audit trail and retention

For regulated work, it isn't enough to store calls. You need a record of who looked at them. The HIPAA Security Rule's audit controls standard, 45 CFR 164.312(b), requires mechanisms that "record and examine activity" in systems that contain electronic PHI. PCI DSS v4.0 requirement 10.5.1 asks for at least 12 months of audit log history, with the most recent three months immediately available. For a contact center, "audit trail" should mean an append-only log of every view, playback, export and deletion of a recording or transcript. Each entry should name the user and the call ID. Retention should be set per artifact type, because audio, transcripts and traces usually need different lifetimes.

10. Evaluation and scoring hooks

Observability tells you what happened. Evaluation tells you whether it was good. The two need to share an ID. OpenTelemetry GenAI now defines evaluation attributes (`gen_ai.evaluation.name`, `gen_ai.evaluation.score.value`, `gen_ai.evaluation.score.label`, `gen_ai.evaluation.explanation`), so a score from any scorer can travel with the trace. In practice you need an API that writes a score against a `call_id` or turn ID, and filters and alerts that can use it. That's what lets you ask "show me every call where the booking-correctness score dropped below 0.8 after version 42." This is also where an independent evaluator plugs in. A tool that only lets its own built-in judge write scores is locking you in.

11. Alerting on voice-specific signals

Uptime alerts miss most voice failures. Alert on what callers feel: p95 caller-heard latency, tool error and empty-result rate, false interruption rate, dead-air seconds per call, transfer and hang-up rate in the first 10 seconds, and STT confidence. Slice each one by agent version. Every alert should link to example traces.

If you build alerts in Prometheus, keep `call_id` out of metric labels. Prometheus warns that every unique label combination creates a new time series, and that labels shouldn't hold high-cardinality values such as user IDs (naming and label guidance). Put the trace ID in an exemplar instead. Grafana can then jump from a latency spike straight to a call.

12. Cost per call

Roll up STT minutes, LLM input, output and cached tokens, TTS characters and telephony minutes for each call, and tag them by version. A prompt change that adds 2,000 tokens of context to every turn shows up here before it shows up on your invoice. Voice agent cost by use case has the per-minute arithmetic.

13. Export and no lock-in

Ask for three things: OTLP ingest (so you instrument once with open SDKs), raw audio download, and bulk export of traces, scores and metadata in an open format. If you can't leave with your data, you'll end up re-instrumenting later.

14. Live session view

Watching a call live looks great in a demo, but you'll rarely use it day to day. Don't let it sway your decision.

Reference architecture on open building blocks

You can meet every must on the checklist with open components. Here's the shape that works for a LiveKit or Pipecat agent on Twilio or Telnyx SIP.

1. Agent instrumentation. LiveKit Agents or Pipecat emit OpenTelemetry spans. You add a root `voice.call` span with version stamps, audio health per turn, and interruption events.

2. OpenTelemetry Collector. Receives OTLP. It runs an attributes or transform processor to drop or hash content attributes before anything is stored, and tail sampling to keep every errored or slow call plus a sample of normal ones.

3. Trace store. Any OTLP-compatible backend, such as Grafana Tempo or Jaeger.

4. Metrics. Prometheus histograms for latency and counters for interruptions and tool status, with trace exemplars. Grafana for dashboards and alerts.

5. Audio. Carrier-side dual-channel recordings and agent-side stereo recordings go to object storage under a `call_id` key, with retention set per bucket.

6. Telephony insights. A post-call job pulls the Twilio Voice Insights summary. A partial summary is ready about 10 minutes after hangup, and the complete one can take about 30 minutes. That makes it a nightly or per-call enrichment, not a real-time alert.

7. Evaluation. A scorer reads audio and traces and writes scores back as evaluation attributes or rows keyed by `call_id`.

8. Audit log. The UI or API you use to play recordings writes an append-only access log.

The one rule that holds all of this together is one `call_id` everywhere. Set it as `gen_ai.conversation.id` on spans. Pass it to Pipecat as `conversation_id`, into LiveKit tracer metadata, into the Twilio `` custom parameters, and into your object storage keys.

An OpenTelemetry span for one voice turn

Below is a runnable-shaped example of the spans for one turn. It uses the standard OpenTelemetry Python SDK and only GenAI attribute names that appear in the current GenAI attribute registry. Anything prefixed `voice.` is our own namespace, because OpenTelemetry has no voice conventions yet. Note that the GenAI conventions now live in their own semantic-conventions-genai repository and are still marked as in development, so pin a version. The code is simplified, and in a real agent the framework creates most of these spans for you.

# Illustrative: spans for one voice turn. pip install opentelemetry-sdk opentelemetry-exporter-otlp
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.trace import Status, StatusCode

provider = TracerProvider(resource=Resource.create({"service.name": "support-voice-agent"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("voice-agent")

CALL_ID = "call_8f2c"

def run_turn(turn_index: int, health: dict) -> None:
    with tracer.start_as_current_span("voice.turn") as turn:
        turn.set_attribute("gen_ai.conversation.id", CALL_ID)
        turn.set_attribute("gen_ai.agent.name", "support-agent")
        turn.set_attribute("gen_ai.agent.version", "2026.10.03-b42")
        turn.set_attribute("voice.turn.index", turn_index)
        turn.set_attribute("voice.prompt.sha256", "9c1e...")          # your prompt hash
        turn.set_attribute("voice.audio.rms_dbfs", health["rms_dbfs"])
        turn.set_attribute("voice.audio.clip_ratio", health["clip_ratio"])
        turn.set_attribute("voice.endpointing_delay_ms", 150)

        with tracer.start_as_current_span("voice.stt") as stt:
            stt.set_attribute("gen_ai.provider.name", "deepgram")      # free-form outside the known list
            stt.set_attribute("gen_ai.request.model", "nova-3")
            stt.set_attribute("voice.stt.confidence", 0.91)

        with tracer.start_as_current_span("chat gpt-4.1-mini") as llm:
            llm.set_attribute("gen_ai.operation.name", "chat")
            llm.set_attribute("gen_ai.provider.name", "openai")
            llm.set_attribute("gen_ai.request.model", "gpt-4.1-mini")
            llm.set_attribute("gen_ai.prompt.name", "support-main")
            llm.set_attribute("gen_ai.response.time_to_first_chunk", 0.38)  # seconds
            llm.set_attribute("gen_ai.usage.input_tokens", 1840)
            llm.set_attribute("gen_ai.usage.output_tokens", 42)

            with tracer.start_as_current_span("execute_tool lookup_order") as tool:
                tool.set_attribute("gen_ai.operation.name", "execute_tool")
                tool.set_attribute("gen_ai.tool.name", "lookup_order")
                tool.set_attribute("gen_ai.tool.call.id", "call_VSPy")
                tool.set_attribute("voice.tool.result_status", "empty")    # ok | empty | error | timeout
                tool.set_status(Status(StatusCode.ERROR, "empty_result"))

        with tracer.start_as_current_span("voice.tts") as tts:
            tts.set_attribute("gen_ai.provider.name", "cartesia")
            tts.set_attribute("voice.tts.ttfb_ms", 150)
            # The caller barged in 1.2 s into playback: an event, not an error.
            tts.add_event("voice.interruption", {"voice.interruption.kind": "barge_in",
                                                 "voice.playback_offset_ms": 1200})
        turn.set_attribute("voice.turn.interrupted", True)

Three choices here are deliberate. The empty tool result sets an error status, so the "errored traces" filter finds it. The interruption is a span event, not an error, because barge-in is normal behavior. And the tool span sits under the LLM span that asked for it, so you can see which model output led to which action. Leave argument and result content (`gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`) to the collector's redaction step, or don't record them on regulated lines. The OpenTelemetry guide for voice agents covers collector config and the LiveKit and Pipecat wiring.

How much telemetry this produces

Here's an illustrative sizing, not a benchmark. Assume 2,000 calls a day, 12 turns per call, and 5 spans per turn (turn, STT, LLM, one tool on average, TTS). That's 60 spans per call and 120,000 spans per day. Add audio health samples every 5 seconds on a 4-minute call, which is 48 points per call, or 96,000 a day if you store them as span events. Any trace backend handles that volume easily. Audio storage is what grows: two channels per call, kept for months. Tail sampling in the collector can cut trace volume a lot. Don't sample audio for calls you might have to audit, though.

Build vs buy for an in-house voice team

If your team of 50 to 500 people runs its own LiveKit or Pipecat agent, the question isn't whether you can build this. It's whether you should build the whole thing. The estimates below are our planning assumptions for one engineer who already knows OpenTelemetry. They aren't measured data.

Our general guidance: build the telemetry plumbing, because it's cheap and it keeps your data portable. Buy, or reuse your framework vendor's cloud features, where the hard part is UI (audio-trace playback) or compliance (audit and attested redaction). Treat evaluation as its own decision. Building scorers is easy. Keeping them calibrated, independent of the team that wrote the prompt, and comparable across versions is the part that takes ongoing work.

How to test a voice agent observability tool in one afternoon

Use this protocol in a vendor demo or against your own build. Run it on a staging number that's connected the same way as production.

1. Script 10 calls. Include a normal booking, a tool that returns empty, a tool that times out, two deliberate barge-ins, one cough mid-sentence, 8 seconds of silence, a whispered turn, a shouted turn, and a read-aloud test card number (use a published test PAN such as 4111 1111 1111 1111).

2. Record carrier-side stereo for all 10 calls through your telephony provider, separately from the tool.

3. Measure 20 turns by hand. In the carrier recording, mark caller speech end and agent speech start, and take the difference. Compare with the tool's latency for the same turns. A tool that reports agent-side latency only will read consistently low.

4. Count events. The tool should show the 2 barge-ins, 1 false interruption, 1 empty tool result, 1 timeout and the long silence. Score item 5 at 0 for any miss.

5. Search for the card number across transcripts, traces, logs and tool arguments. Any plain-text hit fails item 8.

6. Change the prompt and place one more call. Filter by version. Alerts and dashboards should split cleanly.

7. Export everything and open it without the vendor's SDK.

8. Score the 14 items with the weights above. Any must at 0 is a no.

Once the tool can see your calls, the next question is whether your agent is good at them. That's a separate job: scenario-based testing before release and independent scoring of production calls. Monitoring AI voice agents in production covers what to watch day to day, and continuous voice agent auditing covers scoring calls on a schedule. Evalgent works at that layer. It runs independent evaluations against your scenarios and thresholds, and it complements whatever observability stack you pick rather than replacing it.

Frequently asked questions

What is a voice agent observability tool?

A voice agent observability tool collects and connects everything that happens on a call: per-turn traces across STT, LLM, tools and TTS, stereo audio aligned to those traces, audio signal health, latency as the caller hears it, interruptions and versions. It lets you explain why a specific call failed, not just see that error rates went up.

How is voice AI observability different from LLM observability?

LLM observability treats a model call as the unit of work. Voice AI observability treats the turn as the unit of work, and a turn happens in audio and time. It has to capture both audio channels, overlap and interruptions, several network legs, and the caller's latency. A transcript-only trace can't tell an STT hallucination from what the caller actually said.

What should an observability layer capture about audio signal health?

At minimum: input level in dBFS for each turn, share of clipped samples, share of silent frames, and packet loss and jitter for each network leg you can see. Count late packets the jitter buffer dropped as loss too. For PSTN calls, pull carrier-leg metrics from your telephony provider, because agent-side stats only cover the last leg.

Why does my dashboard latency look lower than what callers experience?

Most agent-side metrics start timing when the agent's own end-of-speech detector fires and stop when the first audio frame is sent. That misses inbound network and jitter buffering, the VAD's silence window and outbound playout. Measure caller-heard latency from carrier-side stereo recordings and use the agent breakdown to explain it.

Can I build voice agent observability with OpenTelemetry, Prometheus and Grafana?

Yes. LiveKit Agents and Pipecat both export OpenTelemetry traces. Add a collector for redaction and tail sampling, a trace store, Prometheus histograms with trace exemplars, Grafana alerts and object storage for stereo audio. The hard parts are aligning audio with traces, caller-heard latency, and audit and access controls.

What does a contact center need for AI observability audit trails?

An append-only log of every view, playback, export and deletion of a recording or transcript, with the user, time and call ID. Set retention per artifact type. HIPAA's audit controls standard requires recording and examining activity in systems holding electronic PHI. PCI DSS v4.0 asks for 12 months of audit log history.

Should observability tools score call quality too?

They should accept scores, not necessarily produce them. Look for an API that attaches scores to a call or turn ID, plus filters and alerts on those scores. Keeping scoring separate from the vendor's built-in judge lets you use independent evaluators and compare versions on the same criteria.

What is the most common gap in voice agent observability tools?

Treating the transcript as the truth and agent-side latency as the caller's latency. Both make calls look better than they were. Test any tool by placing scripted calls, measuring silence gaps in a carrier-side recording by hand, and checking that interruptions, empty tool results and spoken PII all show up correctly.

The bottom line

A voice agent observability tool earns its place only if it shows you the call the way the caller lived it: two audio channels, real latency, interruptions, tool results and versions, all on one timeline with PII handled. Grade every tool, including your own build, against these 14 items using calls you placed yourself, and never accept a missing must because the total score looks high.