# Evalgent — Full Reference

Evalgent is an AI voice agent evaluation platform that helps voice AI teams test, iterate, and monitor production-ready agents with confidence. It provides a structured pre-deployment evaluation layer covering functional testing, behavioral testing with realistic human interaction profiles, and limit testing to define failure boundaries. Every test run produces pass/fail verdicts, turn-level evidence from transcripts, statistical reliability scores (SSR), and human-reviewable results. Evaluation is infrastructure — not optional — for teams shipping reliable voice agents.

Website: https://www.evalgent.com

---

## Product Overview

### The Problem

Voice AI teams face four core problems without an evaluation layer:

1. **Agents work in demos but break with real users** — Demos don't reflect real behavior. Without structured evaluation across scenarios and human interaction patterns, teams can't validate agents before deployment.
2. **Teams can't tell if failures are edge cases or systemic** — Without repeatable evaluation, teams can't distinguish one-off failures from underlying reliability issues.
3. **No visibility into behavioral tolerance** — Without limit testing, deployment decisions rely on intuition rather than defined failure boundaries.
4. **Fixing one issue breaks something else** — Without a consistent re-evaluation framework, regressions go undetected across agent iterations.

### What Evalgent Is

- A pre-deployment testing layer that surfaces failures before users do
- A controlled execution framework with defined scenarios and behaviors
- Foundational infrastructure for shipping reliable voice agents
- A decision layer that determines production readiness

### What Evalgent Is Not

- Post-hoc analysis on production transcripts
- LLM-as-judge scoring alone
- A reporting or monitoring tool (it is a decision tool)

### How It Works

1. **Define** — Lock real scenarios and success criteria
2. **Run** — Execute them under realistic human behavior profiles
3. **Measure** — See what works, what fails, and where limits lie
4. **Act** — Get clear, actionable insights on what to fix, tune, or deploy

---

## Platform Features

### Scenarios — https://www.evalgent.com/platform/scenarios

Test scenarios are structured test cases that define what the voice agent must accomplish. Each scenario targets a single objective, isolating what the agent can and cannot do.

**Fields per scenario:**
- Name and description
- Objective — what the agent must accomplish end-to-end
- Success conditions — what passing looks like (e.g., "Refund reason collected", "Appointment confirmed")
- Criticality — how important this scenario is to overall readiness
- Max turns — upper bound on conversation length

**How scenarios are created:**
- Manual authoring: define each field directly
- AI-assisted generation: provide agent instructions and Evalgent generates a full scenario library with gap detection to identify missing edge cases

**Key capabilities:**
- One scenario = one objective (precise debugging, no guesswork)
- Intelligent gap detection identifies overlooked edge cases and industry-specific gaps
- Adaptive execution variability across runs

---

### Caller Profiles — https://www.evalgent.com/platform/profiles

Caller profiles define the simulated human caller used during evaluation. Each profile configures a tester agent to replicate a real segment of your user base, stress-testing the voice agent under realistic behavioral conditions.

**8 configurable parameters:**
1. **Latency** — response delay (immediate to slow)
2. **Noise** — background audio level (quiet to busy)
3. **Behaviour** — cooperative, impatient, confused, adversarial
4. **Speech pace** — slow to fast
5. **Interruption** — frequency of mid-utterance interruptions
6. **Accent** — regional and non-native speaker variation
7. **Language** — primary language of the caller
8. **Voice** — voice characteristics and register

Each profile represents a realistic caller segment. Combining multiple profiles in an evaluation tests the agent against the diversity of your actual user base.

---

### Metrics — https://www.evalgent.com/platform/metrics

Metrics define the evaluation criteria applied to each conversation. Evalgent supports two classes of metrics:

**Telemetry-based metrics** (measured from call data):
- Response latency
- Call duration
- Transfer rate
- Silence / hold time

**LLM-based metrics** (scored by AI from transcripts):
- Tone consistency
- Knowledge accuracy
- Empathy and politeness
- Task completion quality
- Compliance adherence

Teams define custom metrics relevant to their use case. Metrics are applied across every run in an evaluation campaign, producing per-metric scores and aggregate pass/fail determinations.

---

### Evaluations — https://www.evalgent.com/platform/evaluations

An evaluation is a structured test campaign. It combines scenarios, profiles, and metrics into a single run and produces comprehensive results.

**Campaign setup:**
1. **Define test matrix** — select which scenarios, profiles, and metrics to include (each combination becomes one test)
2. **Set success criteria** — configure runs per test and the SSR (Scenario Success Rate) pass threshold
3. **Review and launch** — confirm configuration; Evalgent runs every combination in parallel

**What you get back:**
- **Results matrix** — pass/fail rates across every scenario × profile combination
- **Evidences** — turn-level proof for every success condition from transcripts, recordings, and scored outcomes
- **Recommendations** (beta) — targeted suggestions to improve agent performance based on evaluation results
- **Overall SSR** — statistical reliability score across all runs

**Example:** 12 scenarios × 5 profiles × 3 runs = 180 total test conversations, producing a statistically meaningful reliability measurement.

---

### Reviews — https://www.evalgent.com/platform/reviews

Reviews are the human-in-the-loop correction layer. When an LLM scores a condition incorrectly, a team member can challenge the verdict through a structured appeal process.

**Review process:**
1. **Flag a judgement** — select a condition result, view the evidence and transcript, and flag the LLM verdict as incorrect
2. **Submit an appeal** — provide a comment explaining why the judgement is wrong, with references to specific turns or evidence the LLM missed
3. **Get a decision** — a reviewer examines the appeal, the original evidence, and the transcript; they approve (correcting the outcome) or reject (with notes)

**What you get back:**
- **Corrected outcomes** — approved appeals replace the original LLM judgement
- **Recalculated metrics** — SSR scores and pass/fail verdicts update automatically after corrections
- **Audit trail** — every appeal, decision, and reviewer note is preserved for traceability

Reviews prevent evaluation scores from drifting from reality over time as LLM scoring inevitably makes mistakes.

---

## Use Cases

### Test — https://www.evalgent.com/test-ai-voice-agent

Run scenario-based evaluations to validate AI voice agent reliability, detect failures, and ensure production readiness before launch. Use Evalgent to answer: "Is this agent ready to handle real users?"

---

### Iterate — https://www.evalgent.com/iterate-ai-voice-agent

Safely validate prompt changes, model updates, and conversation improvements before releasing to production. Re-run the same evaluation matrix after each change to catch regressions before they reach users. Use Evalgent to answer: "Did this change make things better or worse — and where?"

---

### Monitor — https://www.evalgent.com/monitor-ai-voice-agent

Continuously track AI voice agent performance in production, detect behavioral drift, and surface emerging failures before they impact real users. Use Evalgent to answer: "Is the agent still performing as expected after deployment?"

---

## Blog

Published at https://www.evalgent.com/blog.

### Published Posts

**"Why AI voice agents fail in production (and how to prevent it)"**
URL: https://www.evalgent.com/blog/why-voice-agents-fail-in-production
Date: 2026-01-15 | Category: Voice AI Evaluation | Read time: 8 min
AI voice agents that ace demos still break in production. This post covers the 5 root causes — acoustic diversity, accent variation, conversational unpredictability, edge cases at scale, and component coupling failures — how to test for each, and what production readiness actually means.

**"LLM as judge for voice agents: the hidden limits of transcript evaluation"**
URL: https://www.evalgent.com/blog/llm-as-judge-voice-ai-evaluation-limits
Date: 2026-04-23 | Category: Evaluation Methods | Read time: 14 min
LLM as judge gives voice agents high scores while real failures go undetected. This post exposes the 5 structural blind spots — ASR errors, latency, task completion, multimodal failures, and abandoned calls — and explains what outcome-based evaluation looks like instead.

**"Conversational AI testing: the complete voice agent stress testing guide"**
URL: https://www.evalgent.com/blog/stress-testing-voice-ai
Date: 2026-04-23 | Category: Testing Strategies | Read time: 13 min
Systematic conversational AI testing for voice agents across 5 dimensions: acoustic, speech pattern, conversational, edge cases, and system stress. Find breaking points across noise, accents, interruptions, and latency before real users do.

**"Voice agent regression testing: why LLM updates break production"**
URL: https://www.evalgent.com/blog/llm-update-regression-voice-agent
Date: 2026-02-10 | Category: Voice AI Evaluation | Read time: 9 min
Updating your LLM improves benchmarks but breaks production voice agents in 5 predictable ways. This post explains why model updates cause regressions and how to build a structured re-evaluation framework to catch them before they reach users.

**"How to automate voice agent testing: synthetic callers vs manual QA"**
URL: https://www.evalgent.com/blog/synthetic-callers-for-voice-agent-testing
Date: 2026-04-07 | Category: Voice AI Testing | Read time: 13 min
How AI test automation replaces manual QA for voice agents. Compares synthetic callers vs human testers across coverage, cost, and speed, with a 5-step framework to scale testing without hiring more QA staff.

**"ElevenLabs voice agent testing guide: what to check before going live"**
URL: https://www.evalgent.com/blog/elevenlabs-voice-agent-testing-guide
Date: 2026-04-24 | Category: Testing Strategies | Read time: 14 min
Test your ElevenLabs voice agent before going live. Covers the 5 pre-launch failure modes specific to ElevenLabs Conversational AI — scenario gaps, user behaviour, tool calls and handovers, concurrent load limits, and voice quality regression — with an 8-item pre-launch checklist.

---

## Technical Notes

- Blog content is stored in `src/data/blogPosts.ts` and statically rendered at build time
- All pages are pre-rendered at build time (Next.js App Router, SSG)
- Sitemap: https://www.evalgent.com/sitemap.xml
- Robots: https://www.evalgent.com/robots.txt