Evalgent
Back to Blog
Voice AI Testing

STT Observability: Partial Transcripts, Confidence and Endpointing Events

Deepesh Jayal
14 min read
STT Observability: Partial Transcripts, Confidence and Endpointing Events
On this page

Here is an illustrative case. A caller says "book me for March thirteenth." The recognizer's first guess is "March thirty." The agent starts its model call on that guess. Moments later the final transcript says "thirteen," but the tool call already carries day 30. Every span is green, and the booking is wrong. That is the gap STT observability voice agent teams close: the final text is rarely what the agent acted on.

This post is the observability view of the speech-to-text layer. It covers which events a streaming recognizer emits and which to keep on a span. It then reads a trace for three failures: acting on a partial, a late transcript revision, and a weak word behind a bad tool argument. It ends with what no trace can show without the audio.

The neighbours cover other jobs. ASR drift monitoring tracks accuracy over weeks. STT evaluation tests offline, and the word error rate guide defines the metric. The streaming vs batch post explains why partials exist. Here the question is narrower: what should one turn's trace say about speech-to-text?

Provider facts are as of November 21, 2026, and every field name below is provider-specific. All worked examples are illustrative.

What a streaming STT provider emits

Streaming recognizers send many messages per utterance, not one transcript. Partial transcripts arrive first, then settled text. Four kinds matter for tracing: tentative text, settled text, an end-of-speech signal, and per-word detail.

SignalDeepgram liveAssemblyAI streamingGoogle STT v2OpenAI Realtime transcription
Text still changingis_final: falseTurn, end_of_turn: falseisFinal false, with stabilitydelta events
Text settledis_final: trueend_of_turn: trueisFinal truecompleted event
Pause or turn endspeech_final: truesilence checks, end_of_turnnot verifiedclient commit, input_audio_buffer.commit
Gap backstopUtteranceEndmax_turn_silencenot verifiednot verified
Speech onsetSpeechStartedSpeechStartednot verifiednot verified
Word score and timeswords[] with confidence, start, endwords[] with confidence, start, endresultEndOffsetnot verified

"Not verified" means I did not confirm it in the docs I opened. It does not mean the provider lacks it. Sources: Deepgram's interim results, endpointing, utterance end and speech started pages; AssemblyAI's message sequence and turn detection pages; Google's StreamingRecognitionResult reference; and OpenAI's realtime transcription guide.

A few details change how you trace.

  • Interim text is a guess. Deepgram says interim results are corrected as more audio arrives. It also warns they can be weaker near the end of the stream. Interim results are off by default and need interim_results=true.
  • Google marks stability only on interim results. Its docs call stability an estimate of how likely the recognizer is to keep its guess. A value of 0.0 means not set, not unstable.
  • OpenAI transcription is keyed by item. Its guide says later deltas can correct earlier text. It also says ordering across different speech turns is not guaranteed, so reconcile by item_id.
  • Word times can clip. In Deepgram's examples, a word whose end equals the running duration may be cut off by the stream boundary. A later interim result can revise it.

A final is not always the end of the turn

The most common tracing mistake is treating one flag as "the caller is done." The flags answer different questions.

Deepgram separates is_final from speech_final. The first means the text for a segment is settled. The second means endpointing heard a pause. Its docs warn against using speech_final alone, because a long utterance can produce several is_final: true messages first. Concatenate the finals until speech_final arrives. The Deepgram latency post walks through that sequence with a worked message table.

AssemblyAI frames it per turn. Partial Turn messages carry end_of_turn: false, and each message replaces the previous text for that turn_order. Its docs say to treat a turn as complete only where both end_of_turn and turn_is_formatted are true. So an end_of_turn: true message alone is not always the last word.

OpenAI's guide describes a completed event as the final transcript for the committed item, with deltas before it.

That is the speech_final vs is_final distinction in practice. The tracing lesson is simple. Your span must record which event your agent treated as "ready," not just what the provider sent. That field is what separates a safe turn from a guess.

Illustrative timeline of one caller turn showing two partial transcripts, a model call started on a partial, agent audio starting, and the final transcript arriving 1,200 milliseconds later.

What to put on the span, and what to keep as events

OpenTelemetry's traces concepts page defines a span event as a structured log message on a span. It is typically used for a meaningful, singular point in time. Attributes are key-value metadata that annotate the span. Use span events for moments and attributes for summaries.

Do not record every partial as an event. A long utterance can emit dozens, and the volume buys little. Record the first partial, the last partial before the final, the final, and the endpoint signal. Keep a count of the rest.

A workable STT observability voice agent design keeps few events and many summaries. The names below are suggestions, not OpenTelemetry standard names. The span attributes post lists the wider field set, and the span model post shows where the STT span sits.

FieldKindWhat it answers
voice.stt.first_partial_msattributeHow long until any text appeared?
voice.stt.partial_countattributeHow unstable was this turn?
voice.stt.final_msattributeWhen did settled text arrive?
voice.stt.endpoint_signalattributeWhich signal ended the turn: pause, gap, force, or client?
voice.stt.min_word_confidenceattributeWas any word weak?
voice.stt.text_hash_finalattributeWhich text is the settled one?
voice.llm.input_stt_stateattribute on the model spanDid the model read a partial or a final?
voice.llm.input_text_hashattribute on the model spanExactly which text did it read?
stt.partial_first, stt.final, stt.endpointeventsWhen did each moment happen?

Two rules keep this useful. First, store offsets from one call clock, and record when your process received each message. Provider timestamps describe positions in the audio stream. They do not say when your agent could act. The timestamp alignment post explains why those clocks disagree.

Second, store hashes and lengths, not text, where transcripts hold personal data. A hash still tells you whether two texts matched.

Also record the configuration in force. Endpointing milliseconds, utterance-end gap, model name and language belong on the call or STT span. When a setting changes, the trace should show it.

Seeing that the agent acted on a partial

A partial-driven action leaves three marks in a trace. The model span started before the final event. Its input state says partial. Its text hash differs from the final hash.

Teams do this on purpose when they use early generation to save time. The end-of-turn post covers that trade. The trouble is doing it by accident, or doing it on a turn that carries an entity. Either way, the trace should say which one happened.

This simplified function turns a turn summary into three flags. It assumes all offsets share one clock.

def stt_turn_flags(turn):
    used = turn["llm_input"]
    final = turn["stt_final"]
    acted_on_partial = used["stt_state"] == "partial"
    final_after_llm_start = final["offset_ms"] > used["llm_start_offset_ms"]
    final_after_audio = final["offset_ms"] > turn["agent_audio_start_ms"]
    text_changed = used["text_hash"] != final["text_hash"]
    return {
        "acted_on_partial": acted_on_partial,
        "final_after_llm_start": final_after_llm_start,
        "changed_after_response": final_after_audio and text_changed,
    }

Count the flags per slot, not per call. A partial that only added a comma is harmless. A partial that held a digit string is not. The tool argument accuracy post explains why those are the turns to protect.

A transcript that changed after the agent responded

This is the quiet one. The model answered from text A. The final arrived as text B. The caller heard a reply to something they never said.

In the illustrative turn above, the model call starts at 2,400 ms on the second partial. Agent audio begins at 3,100 ms. The final arrives at 4,300 ms. That is 1,200 ms after the agent started speaking, and the text differs. The trace flags changed_after_response.

What the flag cannot tell you is whether the reply was wrong. A revision from "march thirty" to "march thirteen" matters. A revision from "ok" to "okay" does not. Do not alert on every transcript revision. Alert on revisions where the changed words sit in a slot your tool call reads.

Check the interaction with barge-in as well. If the caller spoke over the reply, the late final may be the caller correcting the agent. The turn-taking evaluation post covers that. A trace can show the ordering. It cannot show who was talking.

Low-confidence turns that preceded a wrong tool argument

Confidence is the field that links speech to action. Deepgram describes a per-word value from 0 to 1, an estimate of the probability the word is right. AssemblyAI's words[] objects carry a confidence too. Join the weakest word in a slot to the tool span that consumed it.

Illustrative bar chart of five word confidences for a transcript, with four words near 0.98 and the word thirty at 0.48, beside a tool argument day equals 30.

Use the word-level score, not the transcript score. Deepgram's confidence page says that for Nova-3 streaming, the transcript-level value is the median of the word scores. One weak entity can leave that median untouched. In the illustration above, the median is 0.98 while the minimum is 0.48.

Keep the caveats in view. Deepgram says scores from different providers are not directly comparable, and a high score does not guarantee a low error rate. It also says heavy accents or a wrong language model can produce confident wrong words. So low confidence is a useful flag, and high confidence is not proof.

The practical query is a join. For every tool call that failed validation or was corrected by the caller, pull the minimum confidence of the words in its slot. If failures cluster under low scores, you have a lead. If they cluster under high scores, the problem is elsewhere, perhaps in the audio, which is the next section.

The entity accuracy post explains why names and numbers dominate these errors. The silent tool failure post covers what happens after a bad argument is accepted.

Endpointing events: what to record

Endpointing decides when a turn ends. Deepgram's endpointing value is a number of milliseconds of silence, and its default is 10 ms. Its docs suggest 300 to 500 ms where callers pause mid-thought. Setting it to false disables pause detection.

For tracing, three things matter.

  • The configured value. Put it on the span. A cut-off complaint with no setting on record is a guess.
  • The signal that fired. Record whether the turn ended on a pause flag, a gap event, a forced end or a client commit.
  • The gap. Compute the time between the last word's end and the endpoint message. It is your best read of how long the agent waited.

Deepgram's UtteranceEnd is a separate message. It needs interim_results=true and utterance_end_ms from 1,000 to 5,000 ms on hosted endpoints. A last_word_end of -1 means the result was already finalized, and Deepgram says to ignore it. Logging that message as a second turn end would double-count the turn.

AssemblyAI exposes min_turn_silence and max_turn_silence, with defaults that depend on its mode. Its docs also describe a ForceEndpoint message. The endpointing choice guide compares the methods, and the STT latency post covers the latency side. The Flux and Nova-3 comparison shows how one vendor changed the clock.

What stays invisible without the audio

Any STT observability voice agent setup has this blind spot. A trace of STT events records what the recognizer reported. It does not record what the caller said. That gap is real, and naming it keeps your conclusions honest.

Two-column diagram of what STT trace events can show, such as event order and weak words, and what needs the recording, such as the caller's actual words.

Without the recording, you cannot tell these apart from events alone:

  • No transcript because no speech, or no hearing. An empty turn looks the same either way.
  • A confident wrong word. The score is high and the text is wrong.
  • Clipped or muffled input. Events show the effect, not the channel.
  • Background speech. Another voice can become part of the transcript.
  • Word timings that are the recognizer's view. They are not measured against the recording.

Deepgram's own note on SpeechStarted shows the limit. Its timestamp may not match the first word's start, because detection and word timing run as separate systems. Two timestamps from one provider can disagree.

The honest verdict for those cases is "undetermined." A trace can say the transcript was empty. It cannot say why. To settle it, link the recording to the trace, as the recording-to-trace post shows, and listen at the offset. The logs accuracy post covers checking an agent's own account against independent records.

How to add STT observability to a voice agent in seven steps

1. List the events your provider emits - read the streaming docs for interim, final, endpoint, speech-start and word fields, and note which are on by default.

2. Turn on the signals you need - partials, speech-start events and utterance-end messages are often opt-in, so confirm each one reaches your code.

3. Stamp every message twice - keep the provider's audio offset and your own receive time, both relative to one call clock.

4. Write the span - record first partial, last partial, final, endpoint signal, partial count and minimum word confidence for each turn.

5. Tag what the model read - on the model span, record whether its input was a partial or a final, plus the text hash.

6. Record the configuration - add endpointing and gap settings, model name and language, so changes show up in the trace.

7. Seed a test call - say a known digit string with a pause, then check that the trace shows the partial, the final, the flags and the weak word.

Step 7 matters most. A test with a known script tells you whether the instrumentation is right before you trust it on live calls.

Where Evalgent fits

Evalgent Observe, which is launching, ingests OpenTelemetry-format data from any stack. It works from the traces and call data you send, traces each failure to the step or handoff that caused it, and groups failures into root causes. Where your data is blind, it says "undetermined" and shows the coverage gap. Evalgent never sits in the media path. Evalgent Monitor, which is live, flags problems such as tool-call failures and latency spikes in production calls, with fix recommendations.

For testing, Evalgent Reviews replay test calls with audio and transcript together and show the exact tool payloads. Because you script the caller, you know what was said, which closes the gap described above. Call timestamp measurement, such as endpointing and first audio, is part of that. To walk through your own turn traces, you can book a demo.

For the wider picture, see what is voice agent observability, root cause analysis for voice agents and testing vs monitoring vs observability.

Frequently asked questions

What is STT observability for a voice agent?

STT observability is the practice of recording what the speech recognizer emitted on each turn and when. That means partial and final results, endpoint signals, word confidence and timings, plus the text the agent used. It lets you explain a bad turn from its trace instead of guessing from the final transcript.

What is the difference between a partial and a final transcript?

A partial transcript is a tentative guess that the recognizer may revise as more audio arrives. A final transcript is settled for that segment. Deepgram marks them with is_final, AssemblyAI with end_of_turn, and Google with isFinal. Acting on a partial is faster, but the text can still change.

What does speech_final mean compared with is_final?

In Deepgram, is_final means the text for a segment is settled. speech_final means endpointing detected a pause. Deepgram warns not to use speech_final alone, because long utterances can send several is_final messages first. Join the finals until speech_final arrives, then treat the turn as complete.

How do I tell that my agent acted on a partial transcript?

Compare timestamps and text. If the model call started before the final event, and the model span's input text hash differs from the final hash, the agent acted on a partial. Recording an input state of partial or final on the model span makes the check a simple filter.

Can I trust an STT confidence score?

Treat it as a flag, not proof. Deepgram describes word confidence as an estimate of correctness and says scores are not comparable across providers. It also says confident wrong words can occur. Use word-level scores for names and numbers, because a transcript-level median can hide one weak entity.

Which endpointing events should I log?

Log the configured endpointing or silence settings, which signal ended the turn, and the gap between the last word's end and that signal. For Deepgram, also log UtteranceEnd separately and ignore a last_word_end of -1. For AssemblyAI, log the end-of-turn message and any forced end.

Should I store every partial transcript as a span event?

Usually not. A long utterance can emit many partials, and most add little. Record the first partial, the last partial before the final, the final and the endpoint signal, and keep a count of the rest. Store hashes and lengths instead of text where transcripts hold personal data.

What can a trace not tell me about speech-to-text?

A trace cannot tell you what the caller actually said. An empty transcript could mean silence or missed speech. A high-confidence word can still be wrong. Clipped audio, background voices and recognizer word times need the recording to check. Where only events exist, the honest verdict is undetermined.

The bottom line

STT observability voice agent teams can trust starts with recording, on one span per turn, what the recognizer emitted, when your agent acted, and which text the model read. A trace then shows partial-driven actions, late revisions and weak words behind bad arguments, while the caller's real words stay undetermined without the recording.

Related Articles