Evalgent
Back to Blog
Voice AI Testing

How to Cluster Voice Agent Failures Into Root Causes

Deepesh Jayal
14 min read
How to Cluster Voice Agent Failures Into Root Causes
On this page

A week of failed calls lands in a spreadsheet. There are 400 rows. Each has a judge label, a free-text reason and a call ID. Reading them one by one is slow, and it hides the pattern. Ten rows say "wrong date." Eight say "asked twice." Nobody can tell which group to fix first.

This post is about how to cluster voice agent failures into groups that each have one cause and one fix. It is a method post. It covers what to cluster on, rules versus text clustering, how to rank clusters, how to check a cluster has one cause, and what to do with the long tail. It ends with a six-step routine for one week of failed calls.

It does not repeat the neighbours. Voice agent root cause analysis covers the three evidence layers and one worked case. The failure layer map gives a nine-layer taxonomy. This post goes deeper on the grouping step they both rely on. It makes no benchmark claims, and its worked example is illustrative.

What a failure cluster is, and what it is not

Failure cluster (voice agent): a set of failed calls that share one cause, so one change would fix all of them.

That last clause is the test. A cluster is not a topic and not a label. "Booking problems" is a topic. It may hide three causes. A cluster is only useful when fixing it moves the count.

Three terms get mixed up, so keep them apart.

TermWhat it isExampleGood for
Failure labelA verdict on one call, from a judge or a rulewrong_date_bookedFinding failed calls
Voice agent failure taxonomyA fixed, shared list of labels and layersTelephony, speech recognition, toolsCommon language, reporting
Failure clusterA group of calls with one causeRelative dates after the v31 promptDeciding what to fix

A taxonomy is a vocabulary. A cluster is a diagnosis. Teams that stop at the taxonomy get a bar chart of labels and no idea what to change. Voice agent failure clustering gives a ranked list of fixes. You need both: the taxonomy names the symptom, and clustering finds the cause.

The failure patterns voice agent teams see are rarely one-to-one with labels. One label can come from several causes. One cause can show up under several labels. Clustering is how you untangle that.

The features to cluster on

You cluster on what you stored. If a field was not captured on the call, it cannot be a feature. Our post on what to log on every voice agent call lists the fields. This section says which ones do the grouping work.

Diagram of a two-pass method to cluster voice agent failures: 420 failed calls are grouped by structured keys first, then free-text reasons are clustered inside each group, then each cluster is split-tested and ranked
FeatureWhere it comes fromUse it asWatch out for
Failure labelJudge or rulesGroup keyLabels from different judges are not comparable
First failing span or stepTraceGroup keyCluster on where it started, not where the call ended
Release, prompt, model, tool versionVersion stamps on the callSplit dimensionMissing stamps hide a cause, so count them
Tool or providerSpan attributesSplit dimensionProvider and tool can both vary; split one at a time
IntentCall details or classifierSplit dimensionUse rate within intent, not raw counts
Cohort: carrier, language, channel, regionCall detailsSplit dimensionSmall cohorts give noisy lifts
Time windowCall timestampSplit dimensionA change point is a strong clue
Free-text failure reasonJudge outputText clustering inputWording varies; see the next section

Two choices matter most.

Use the first failing step, not the last. A caller hangs up at the end of almost every bad call. "Hangup" is a bad key. Trace back to the first step that went wrong, such as a slot that was filled with the wrong value or a tool span that returned nothing. The trace and span model for voice agents shows where that step lives.

Keep keys and splits separate. Keys define the cluster. Splits test it. If you put "release" in the key, you can never ask whether release explains the cluster. Version stamps belong in the split. OpenTelemetry has a service.version attribute for the service's version string, and its format is left to you. Prompt and model versions need attributes of your own. The GenAI conventions post covers what exists.

Rules first, then text: how to group voice agent failures

Do the cheap, exact grouping first. Then use text clustering only for what is left.

Pass one: rules on structured fields

A pivot on failure label and first failing step needs no model. It is exact, repeatable and easy to explain. Run it first.

Rules fail in one place. The judge's free-text reason is not structured. "Booked Thursday when the caller said Tuesday," "wrong day on confirmation" and "date was off by two days" are one problem in three phrasings. A rule that matches exact strings will count them as three.

Pass two: group the free-text reasons

This is where embedding or LLM grouping earns its place. Documented approaches share a shape: shorten each item, vectorize it, group the vectors, then name the groups.

  • Open coding first. Langfuse's error analysis guide tells you to write a free-text note on the first thing that went wrong in each trace, with "no predefined categories yet." Categories then come from the notes. It says an LLM can draft a taxonomy from your notes, and that you should split anything that conflates two root causes. It also says the first taxonomy is never the final one.
  • Embed and group. Anthropic's Clio paper summarizes each conversation, embeds the summaries, groups them with k-means, and has a model write a title and description for each group. The authors checked the result by hand. In their review, roughly 3% of conversations in a typical cluster did not clearly belong there. Their listed limits include weaker clustering for outlier topics. That study covers chat conversations, not phone calls, but the method transfers.
  • Density methods and noise. BERTopic embeds documents with pre-trained language models, clusters the embeddings, and describes each topic with a class-based TF-IDF procedure. HDBSCAN does not need you to set a cluster count. It labels points outside any cluster as noise, with the label -1. That "noise" bucket is useful for failures. It is your first view of the long tail.
  • Know the caveat. Arize's cookbook on clustering trace patterns summarizes each trace in under 25 words, vectorizes the summaries and clusters them. It warns that TF-IDF matches shared words, not meaning, so "asking for a refund" and "want my money back" may not cluster together. It also warns that a low unclustered count does not prove good clustering.

The table puts the options side by side.

MethodStrengthWeaknessUse it when
Rules on label and stepExact, auditable, no modelBlind to wording varietyAlways, as pass one
Embeddings plus k-means or HDBSCANGroups by meaningCluster count or size needs tuning; noiseHundreds of free-text reasons
LLM drafts and names groupsFast, readable namesCan merge or invent; needs reviewNaming clusters and drafting a taxonomy
Human review of samplesCatches wrong mergesSlowEvery cluster you plan to act on

Use the model to propose and a person to approve. A clustered week is a draft, not a verdict.

Size clusters and pick what to fix first

Count alone is a poor ranking. A cluster of 120 mild annoyances can matter less than 30 calls that wrote a wrong record. Rank by impact.

Impact = calls affected × severity.

Define severity on a short scale before you look at the data. Use three levels at first. The scale below is an example. Set your own, and write down what each level means.

SeverityMeaning in this example
1Annoying, but the caller got the outcome
2Extra effort: the caller had to repeat or wait, and the outcome was right
3Wrong or missing outcome in a record, or a required handoff was missed

This is not new. Failure mode and effects analysis, a long-standing quality method, prioritizes failures by how serious the consequences are, how often they occur and how easily they can be detected. ASQ's description rates severity on a scale from one to ten. The impact score above keeps the first two factors and drops the third to stay simple. Add detectability later if you have blind spots.

A worked example

This example is illustrative. It is a composite of an imagined clinic scheduling agent, and every number is an assumption. Say the team scored 6,000 calls in a week and flagged 420 as failed. A pass-one rule on label and first failing step gave this.

Cluster (label, first failing step)CallsSeverityImpact
Wrong date booked, date slot1403420
Repeated question, slot collection1202240
Talked over caller, turn handoff75175
Human requested, not transferred, intent routing30390
Small groups (14 of them)35mixednot scored
Undetermined, no trace linked20n/anot scored

The top cluster is "wrong date booked." It looks like the clear first fix. The next section shows why that reading is wrong.

Horizontal bar chart of illustrative impact scores: before the split one wrong-date cluster scores 420; after the split relative dates on prompt v31 score 270, repeated questions 240, day-first dates 150, handoff misses 90 and talk-over 75

Break ties with fix cost and trend

Impact is the first sort. Three tie-breakers come after it.

  • Cost to fix. A one-line prompt change beats a tool rewrite at equal impact.
  • Trend. A cluster that doubled since last week outranks a flat one.
  • Link to a change. A cluster that began at a release is the cheapest to confirm and to roll back.

Use rates, not raw counts, when you compare across intents. Forty failures in an intent with 200 calls is a bigger problem than 40 in an intent with 4,000. Keep the denominator in the table.

Validate that a cluster is one cause, not several

A cluster can look tidy and still hide two fixes. Merged clusters are the most common error. The Langfuse guide says the same thing in its own terms: split anything that conflates two root causes. Run three tests before you commit a fix.

1. The concentration test. Take the cluster and count its calls by every split dimension. If it is one cause, some value should hold almost all of the calls and be far more common in the cluster than in all calls. Compute lift:

lift = share of the cluster with this value ÷ share of all calls with this value

A value with a high share and a high lift is a suspect. A cluster where no single value dominates is a warning.

2. The block test. Cross two dimensions in the cluster. If the calls fall into one block, you have one cause. If they fall into two separate blocks, you have two.

3. The same-fix test. Read five calls from each block. Ask one question: would the same change clear all of them? If not, split.

Here is the check on the illustrative "wrong date" cluster. The 6,000-call base rates are assumptions: 30% of all calls ran prompt version v31, and 10% of callers spoke dates day-first.

Split test for the 140-call wrong-date cluster: 90 calls are relative dates on prompt v31 and 50 calls are day-first dates across versions, shown as two separate blocks rather than one
BlockCallsPrompt versionHow the caller said the dateHypothesis
1a90All on v31Relative ("next Friday")The v31 prompt no longer supplies today's date, so relative dates resolve wrongly
1b50Older versionsDay-first ("4 March")The date tool reads some numeric dates as month-first

No single value explains the parent. Prompt version covers 64% of the calls, and date style covers 64% as relative and 36% as day-first. The crosstab shows two blocks. The fixes differ: a prompt edit for 1a, a parser change for 1b. Merged, the cluster would send the team to one fix and leave half the calls broken.

The split also changes the ranking. Block 1a scores 270 and block 1b scores 150. "Repeated question" at 240 now outranks 1b. The prompt fix is still first, but the one-cause reading is gone. The repeated-question fix now comes second, and the date-parser fix comes third.

Treat both causes as hypotheses until a fix moves the count. The test for that is simple. Ship the change, then watch the cluster shrink on the next week's data.

The check in code

This is simplified pandas. It assumes one row per failed call with the columns shown, plus a table of all calls for base rates.

import pandas as pd

def rank_clusters(failed: pd.DataFrame) -> pd.DataFrame:
    failed = failed.assign(
        cluster=failed["failure_label"] + " | " + failed["first_failing_step"]
    )
    out = failed.groupby("cluster").agg(
        calls=("call_id", "nunique"),
        severity=("severity", "max"),
    )
    out["impact"] = out["calls"] * out["severity"]
    return out.sort_values("impact", ascending=False)

def split_test(failed, all_calls, cluster, dims):
    failed = failed.assign(
        cluster=failed["failure_label"] + " | " + failed["first_failing_step"]
    )
    sub = failed[failed["cluster"] == cluster]
    rows = []
    for d in dims:                      # e.g. ["prompt_version", "date_style"]
        in_cluster = sub[d].value_counts(normalize=True)
        base = all_calls[d].value_counts(normalize=True)
        for value, share in in_cluster.head(3).items():
            rows.append({"dimension": d, "value": value,
                         "share_of_cluster": round(share, 2),
                         "share_of_all_calls": round(base.get(value, 0), 2),
                         "lift": round(share / base.get(value, 1e-9), 1)})
    return pd.DataFrame(rows).sort_values("lift", ascending=False)

# Block test: look for two separate blocks, not one.
# pd.crosstab(sub["prompt_version"], sub["date_style"])

Run it on a sample first. The code only ranks and measures. A person still has to read the calls.

What to do with the long tail

Every week leaves calls that fit no cluster. Do not drop them and do not force them in. Handle them in three buckets.

Small clusters. Groups too small to rank. Keep them named and counted. A small group that grows week over week gets promoted.

Singletons that matter. One call with severity 3 can be worth a look even if it is alone. The Clio authors list weaker clustering of outlier topics as a limit. Route high-severity singletons to a person regardless of cluster size. Sampling live calls covers why you cannot sample these away.

Undetermined. Calls with no trace link, no version stamp or too little data. Label them "undetermined" and show the count. It is a coverage gap, not a failure to hide. If undetermined is 5% of failures, you can act on the rest. If it is half, fix the logging before you trust any cluster. The production feedback loop post covers how fixed clusters become regression tests.

Re-cluster every week. Langfuse's guide lists a prompt rewrite, a model swap and a new feature as reasons to run a new round. Keep cluster names stable, or saved filters and trend lines break.

How to cluster one week of failed calls in six steps

This routine turns a spreadsheet of failed calls into a ranked fix list. It assumes you have a judge label, a first failing step and version stamps. If you do not, step one tells you.

1. Pull and audit the week — Export every failed call with its label, first failing step, free-text reason, severity and call details. Count how many lack a trace link or version stamp. Those go to "undetermined" now, so they cannot distort the clusters.

2. Group by rules — Pivot on failure label and first failing step. Write the counts. Do this before any model, because it is exact and fast.

3. Cluster the free-text reasons inside each group — Within each large group, embed or summarize the reasons and cluster them. Let a model draft a name for each cluster. Keep the noise bucket separate.

4. Size and rank — Score severity with your written scale. Compute impact as calls times severity. Use rates within intent when you compare across intents. Sort, then apply the tie-breakers: cost to fix, trend, link to a change.

5. Split-test the top clusters — For each cluster you plan to fix, run the concentration, block and same-fix tests across version, tool, intent, cohort and time. Split any cluster that shows two blocks. Re-rank.

6. Handle the tail, then fix and recheck — Name the small clusters, send high-severity singletons to a person, and publish the undetermined count. Fix the top cause, write a test for it, and compare the cluster size next week.

Common mistakes when you group failures

  • Clustering on the last step. Calls end in hangups. Cluster on where the failure started.
  • Ranking by count. A frequent mild issue can outrank a rare harmful one. Use severity.
  • Letting the model name clusters unchecked. Names can merge two causes. Read samples.
  • Putting version in the key. Then you cannot test whether version explains the cluster.
  • Fixing without a test. The cluster returns. Turn each fixed cluster into a scenario.

A clustered list also feeds alerting. A cluster that grows fast is a candidate for a page or ticket. See voice agent alerting for the tiers.

Where Evalgent fits

Evalgent is a testing, evaluation and monitoring platform for voice agents. Its production call monitoring flags problems such as tool-call failures and latency spikes, with fix recommendations, which gives you the failed-call list this routine starts from.

Evalgent Observe, which is launching, does the grouping step described here. It traces each failure to the step or handoff that caused it and clusters failures into root causes, so a team fixes causes instead of single calls. It shows "undetermined" where the data supplied cannot support a verdict, and it shows coverage gaps instead of hiding them. It works with any stack through OpenTelemetry and an upload path for call details. Native connectors for ElevenLabs, Vapi and Retell are planned. Evalgent does not sit in the media path, so nothing touches live audio or slows a call.

You can run the routine without any tool. A spreadsheet and the code above are enough for a first pass. For the wider picture, see what voice agent observability is, and for the buyer checklist see voice agent observability.

Frequently asked questions

How do you cluster voice agent failures?

Group failed calls by failure label and the first step that went wrong. Cluster the judge's free-text reasons inside each group. Then split each cluster by version, tool, intent, cohort and time to confirm it has one cause. Rank the confirmed clusters by calls affected times severity.

What is a voice agent failure taxonomy?

A voice agent failure taxonomy is a shared list of failure labels and layers, such as speech recognition, turn-taking and tools. It gives a team common language. It is not a diagnosis. A taxonomy names the symptom, and clustering finds which cause behind it you should fix first.

Should I use embeddings or rules to group failures?

Use rules first for structured fields like label and failing step. They are exact and auditable. Use embeddings or an LLM only for free-text reasons, where wording varies. Keep a person in the loop to read samples and approve names before you act on a cluster.

How do I know a cluster is one root cause?

Count the cluster's calls by version, tool, intent and cohort. One cause shows a single value with high share and high lift. Cross two dimensions: one block means one cause, two blocks mean two. Then check that one change would clear a sample of calls from each block.

How do I decide which failure cluster to fix first?

Score impact as calls affected times severity, using a severity scale you define before looking at the data. Sort by impact. Break ties by cost to fix, trend since last week and whether the cluster began at a release. Compare intents by rate, not raw count.

How many failed calls do I need to cluster?

There is no fixed number. A few dozen failed calls can show one clear cause. A few hundred let text clustering work better. More calls help you split clusters by cohort without noisy results. Start with one week and read samples of each cluster before you trust it.

What should I do with failures that fit no cluster?

Do not force them in. Keep small groups named and counted, and promote any that grow. Send high-severity singletons to a person. Label calls with missing traces or version stamps as undetermined and report the count. A large undetermined share means fixing logging first.

How often should I re-cluster voice agent failures?

Re-cluster weekly, and again after any prompt, model or tool change. Keep cluster names stable so trends and saved filters keep working. Langfuse's guide also notes that the first taxonomy is never the final one, so expect names and splits to change as the agent changes.

The bottom line

To cluster voice agent failures well, group by where the failure started, split each group by what changed, and rank by calls affected times severity. A cluster is only worth fixing once the split test shows it has one cause.

Related Articles