Evalgent
Back to Blog
Voice AI Testing

RAG Observability for Voice Agents: What Was Retrieved vs What Was Said

Deepesh Jayal
14 min read
RAG Observability for Voice Agents: What Was Retrieved vs What Was Said
On this page

A caller asks about a return window. The agent answers "fourteen days" in a calm voice. Was that from your policy? From a stale chunk? From the model's memory? Or did retrieval never run?

Without a retrieval record, you cannot tell. The transcript holds the answer. The knowledge base holds the policy. Nothing joins them.

This post is the observability view of retrieval. It covers what to capture on a retrieval span, how a trace compares retrieved text with spoken text, why timing matters, and what stays undetermined. Pre-launch testing is in how to test knowledge grounding in voice agents. Standards and provider APIs are as of November 21, 2026. Every example is illustrative.

Why retrieval needs its own record

Lewis et al. describe retrieval-augmented generation as a pre-trained seq2seq model, the parametric memory. It is paired with a dense Wikipedia index, the non-parametric memory, reached by a neural retriever. The abstract lists providing provenance and updating knowledge as open problems for models that keep knowledge in their weights.

That split gives a voice agent a debugging handle. An answer comes from the passages retrieved for this turn, or from the model itself. A trace that stores the passages shows which one the agent leaned on. A transcript alone cannot.

Hallucination rate asks how often answers are false. Grounding tests check answers on scripted calls. This post covers live calls, where nobody scripted the question and the trace is the record.

What to capture on a retrieval span

OpenTelemetry defines a retrieval span in its GenAI conventions. On the retrieval page, gen_ai.operation.name should be retrieval. gen_ai.retrieval.top_k is recommended. gen_ai.retrieval.documents is opt-in, and each document should carry an id and a score when available. gen_ai.retrieval.query.text is opt-in, with a warning that it may contain sensitive information.

A gen_ai.retrieval.duration metric runs from the moment the retriever receives the query until it returns documents or fails. The page is marked Development as of November 21, 2026, so names can change. Our GenAI conventions post tracks the wider status.

The conventions cover part of what a RAG observability voice agent trace needs. This table marks which fields are standard and which are suggestions.

FieldWhy it mattersStandard?
Query as sentThe text retrieval saw, which may differ from the caller's wordsYes, opt-in
Query rewrite, before and afterA rewrite can drop or add a constraintNo
Knowledge base versionTies an answer to the content that was liveNo
Top-k and count returnedMakes empty results visibleTop-k only
Chunk IDs with scoresJoins a claim to its sourceYes, opt-in
Retrieval durationThe caller waits while it runsYes
Empty or low-score flagMarks weak or missing supportNo
Chunks placed in the promptRetrieved is not the same as shownNo
Speculative flag, used or discardedExplains lookups started earlyNo
Error typeSeparates a failed lookup from an empty oneYes
Five records for one turn: caller words, query as sent, retrieved chunks, prompt contents and spoken answer, with four gaps where the chain can drift between them.

Two fields deserve extra thought. A chunk ID without a knowledge base version means little once content is edited. And retrieval may return five chunks while the prompt budget keeps three. Log only the retrieval result and you credit the model with evidence it never saw.

Query and document text can hold personal data. Store IDs and scores always, and text only under your masking rules, as in PII redaction for logs and traces. For neighbouring fields, see voice agent span attributes and what to log on every call.

Hosted retrieval tools return evidence only if you ask. OpenAI's file search guide says search results are null by default and come back when you pass include: ["file_search_call.results"]. The file_search_call item lists the queries used, and answer annotations of type file_citation carry file_id and filename. Anthropic's citations documentation describes cited_text with a document_index and a location. At defaults you may hold the answer but not the evidence.

The query as sent is not the question as asked

A typed search box passes the user's text straight in. A voice agent passes recognizer output, often reworded before search. Errors compound at each step, so a RAG observability voice agent trace has to show each step.

Deepgram's interim results guide is a useful model. Interim results are preliminary transcripts marked is_final: false, and they may change. A finalized segment carries is_final: true. The page says to append each final to a buffer, and not to rely on speech_final alone, because long utterances can produce several finals first.

A query built from one fragment, or an interim, can be a partial question. Record which transcript segments fed the query, and whether they were interim or final. Our STT observability post covers those events.

Here is an illustrative case. The caller says "Do you ship to Alaska?" The recognizer writes "do you ship to Alabama". The query as sent is "shipping to Alabama". Retrieval returns a shipping chunk, and the agent answers with confidence. The trace shows the wrong place name. It cannot show what the caller said, so that claim is undetermined unless someone plays the recording. Voice agent logs versus what happened makes the same point about transcripts.

Spoken queries differ from typed ones in three ways:

  • They are short. "What about opened ones?" needs earlier turns, so a query rewrite does real work.
  • Recognition errors land inside the query. Names and numbers suffer most.
  • The caller cannot see the results, so they cannot spot the error and retype.

Log the query rewrite as before and after. You can then see an added word the caller never said, or a dropped constraint such as "opened".

Comparing what was retrieved with what was said

With a retrieval record and the answer text, you can compare claim by claim. Split the answer into factual claims. For each, ask what the prompt contained. Six outcomes cover most turns.

OutcomeTrace evidenceReading
SupportedA chunk in the prompt states the claimGrounded in that chunk
ContradictedA chunk in the prompt conflicts with itChunk ignored or misread
UnsupportedNo chunk in the prompt backs itMemory, inference or earlier turn; undetermined which
Retrieved, not in promptSupport sits in a chunk the model never sawRight by luck or memory
Retrieval never ranA factual claim, no retrieval spanRouting or tool-choice question
UndeterminedChunk text or claim missing from the recordSay so, show the gap

Published metrics supply vocabulary. The RAGAS faithfulness definition is claims supported by the retrieved context divided by total claims, from 0 to 1. Context precision evaluates how well the retriever ranks relevant chunks above irrelevant ones, as the mean of precision@k. Context recall counts how many relevant documents were retrieved, so it needs a reference. The RAGAs paper frames these as reference-free checks.

Live calls rarely have ground truth. A claim-level support check needs only the answer and the context, so it fits production traces better than recall.

A worked turn

This turn is illustrative. The shop, chunks and scores are invented. The caller asked about returning an opened item. The query as sent, after rewrite, was "return window for opened items". Retrieval returned three chunks, and only the top two fit the prompt budget.

ChunkScoreTextIn prompt?
kb-1140.82Unopened items: return within 30 daysYes
kb-1180.74Opened items: return within 14 days, 10% restocking feeYes
kb-1310.58Refunds go to the original payment methodNo

The agent said three things: opened items can be returned within 14 days, there is no restocking fee, and refunds go to the original payment method.

ClaimEvidenceOutcome
14-day windowkb-118, in promptSupported
No restocking feekb-118 says 10%, in promptContradicted
Refund to original methodkb-131, retrieved, not in promptRetrieved, not in prompt

The score depends on the denominator. Against what the model saw, one claim of three is supported: 1 / 3 = 0.33. Against everything retrieved, two of three: 2 / 3 = 0.67. The second hides that the refund claim was never grounded in the generation. Log both and the gap becomes a finding. A tight token budget is one possible cause, which this record cannot confirm.

Illustrative turn: three spoken claims matched to chunks kb-118 and kb-131, with outcomes supported, contradicted and retrieved but not in prompt, scoring 0.33 against the prompt and 0.67 against retrieval.

Splitting an answer into claims is a judgement. A person or an LLM judge makes it, and either can err. Sample verdicts for human review, as in measuring hallucination rate.

When retrieval never ran

Some wrong answers have no retrieval span. The agent answered from memory, or a routing step skipped the tool. Flag turns with a factual claim and no span, then ask why. A greeting rightly needs none.

A related failure is a lookup that errored while the agent kept talking. A clean span proves little, as detecting silent tool failures argues. Check the error type and the returned count. Zero documents plus a confident answer is a finding.

Why timing matters on a phone call

The caller waits in silence while retrieval runs. In a sequential design the retrieval span sits between the final transcript and the first model token. Its duration lands in the gap the caller hears.

Take an illustrative turn, measured from the end of caller speech to first audio. Query rewrite takes 180 ms and retrieval 340 ms. Model first token takes 420 ms and speech first audio 310 ms. The total is 1,250 ms. Rewrite plus retrieval is 520 ms, or 41.6 percent of the wait, for a step the caller never hears. Track retrieval latency at the tail, not only the mean. Our posts on time to first audio and what to measure for latency cover the full chain.

Speculative retrieval and what it does to the trace

Speculative retrieval starts the search before the turn is confirmed. It might run on a final transcript segment while the caller is still talking. If the caller adds words, the query is stale and the result is discarded.

LiveKit's turn-taking tuning page documents the same idea for generation. Preemptive generation starts LLM generation when a final transcript arrives, before the turn is confirmed. If the final transcript changes, the preemptive response should be cancelled. The page says it does not always reduce latency and advises checking observability metrics. It covers generation, not retrieval, but the trade-off carries over: less waiting on a hit, wasted work on a miss.

Illustrative timeline of one turn: sequential retrieval gives 1,250 ms to first audio, a speculative hit gives 730 ms, and a speculative miss returns to 1,250 ms plus one wasted lookup.

If the speculative result is used, rewrite and retrieval leave the path after the endpoint. That is 1,250 minus 520, or 730 ms. If the transcript changes, the path returns to 1,250 ms and you paid for a wasted lookup. These figures assume the speculative call finishes in time.

For the trace, add three fields: the speculative flag, the query used for that call, and used or discarded. Without them, a chunk in the trace may come from a question the caller never finished asking.

What stays undetermined

A RAG observability voice agent record narrows a wrong answer. It does not close every question. Say "undetermined" when the data is blind.

QuestionWhat the trace showsIf evidence is missing
Is the knowledge base correct?The chunk and version usedUndetermined; a supported answer can still be wrong
Was it reachable?An error type or empty result, if instrumentedUndetermined if no span exists
Why did the model ignore a chunk?That it was in the prompt and contradictedCause undetermined: wording, order or truncation
What did the caller say?The recognizer's textUndetermined without the recording
Where did an unsupported claim come from?That no chunk backs itUndetermined

The first row matters most. Faithfulness measures consistency with the retrieved context, not truth. If the knowledge base holds an old price, a faithful answer repeats it. Only the policy owner or the system of record can say whether that chunk was right.

The second row is a coverage question. A missing span could mean retrieval never ran, or ran uninstrumented. Count factual-claim turns with no span and report that number beside your findings. Sampling can create holes too. This comparison does not analyse call audio. It works from text the agent produced and records the system kept.

How to add RAG observability to a voice agent

Use this order for RAG observability voice agent work. Each step adds one field and one answerable question.

1. Wrap retrieval in its own span - Record the operation name retrieval, start, end and error type. This shows whether retrieval ran and how long it took.

2. Record the query before and after rewrite - Store both strings and the transcript segment IDs, interim or final.

3. Store chunk IDs and scores - Keep them every turn, with text only under your masking rules. This joins a claim to its source.

4. Version the knowledge base - Stamp each span with the content version, so an answer ties to what was live.

5. Log what reached the prompt - Record chunk IDs placed in the prompt and the token count. This separates retrieved from shown.

6. Mark speculative calls - Add a flag, the query used, and used or discarded.

7. Flag empty and low-score results - Pick your own cut-off from your data and write it down.

8. Label claims and report coverage - Sample turns, assign an outcome from the table, and count claim turns with no span.

If time is short, do steps 1 to 3 first. For where these spans sit in a call, see the OpenTelemetry trace and span model for voice agents and what is voice agent observability.

Where Evalgent fits

Evalgent Observe, which is launching, ingests OpenTelemetry-format data from any stack. It works from the traces and call data you send, traces each failure to the step or handoff that caused it, and groups failures into root causes. A retrieval span gives it a step to point at. Where your data is blind, it says "undetermined" and shows the coverage gap. It never sits in the media path. Evalgent Monitor, which is live, flags problems such as tool-call failures and latency spikes in production calls, with fix recommendations.

For testing before launch, Evalgent Reviews replay test calls with audio and transcript together and show the exact tool payloads. Because you script the caller, you know what was asked, which closes the speech-to-text gap above. To walk through your own retrieval traces, you can book a demo. For the wider picture, see testing vs monitoring vs observability and root cause analysis for voice agents.

Frequently asked questions

What is RAG observability for a voice agent?

It is recording what retrieval returned on each turn and comparing it with what the agent said. The trace holds the query, chunk IDs, scores, timing and prompt contents. You can then tell whether an answer was supported, contradicted, unsupported or given without retrieval.

What should a retrieval span record?

Record the query as sent, documents requested and returned, chunk IDs with scores, duration and error type. Add the query before rewrite, the knowledge base version, chunks placed in the prompt, and a speculative flag. OpenTelemetry marks query text and documents as opt-in because they may be sensitive.

How do you tell if an answer came from a retrieved chunk?

Split the answer into claims and match each to a chunk that was in the prompt. A claim the chunk states is supported. A claim it conflicts with is contradicted. A claim with no chunk behind it is unsupported. A chunk that was retrieved but cut proves nothing.

How do speech recognition errors affect retrieval?

The query is built from recognizer text, so a misheard word becomes a wrong search. Interim transcripts can also change, so a query from a fragment may be partial. Log the transcript segments behind each query. The trace cannot confirm what the caller said without the recording.

Does retrieval add latency to a voice call?

Yes. In a sequential design the caller waits through the retrieval span, which sits between the final transcript and the first model token. In the illustrative budget above, rewrite and retrieval took 520 ms of 1,250 ms. Measure the tail on real calls, since your numbers will differ.

What is speculative retrieval in a voice agent?

It starts the search before the turn is confirmed, so the result is ready sooner. If the caller keeps talking and the transcript changes, the result is stale and discarded. LiveKit documents the same trade-off for preemptive generation. Record a speculative flag and whether the result was used.

What can a trace not tell you about a wrong answer?

It cannot tell you whether the knowledge base was correct, what the caller actually said, or why the model ignored a chunk. It cannot explain an unsupported claim either. Mark those undetermined, and review the recording or source document where the call matters.

Should you store retrieved text and queries in traces?

Store chunk IDs, scores and the knowledge base version by default. Store query and chunk text only after masking personal data, because both can contain it. With IDs and a version, you can fetch the source text when a review needs it.

The bottom line

RAG observability for a voice agent comes down to one comparison on every turn: what retrieval returned, what reached the prompt, and what the agent said. Capture those, show the gaps as undetermined, and treat the knowledge base itself as a source you still have to verify.

Related Articles