Voice Agent Metrics You Can Score with Jev (Score / Noul / Choice)

# Voice agent metrics you can score with Jev (Score / Noul / Choice)

Quick answer

Jev voice agent metrics are your quality rubric written as typed questions. Each call becomes a state. Each metric becomes a Score, Noul, or Choice question. Jev returns calibrated decisions in 70–500ms, scoring your whole suite in one parallel call. It is a scorer you must still validate, not independent evidence.

Most teams keep their voice agent metrics in a spreadsheet. The scores get filled in by an LLM judge on a thin sample of calls. That is slow, costly, and noisy. Jev changes how you compute the whole rubric.

Jev launched on September 15, 2026, in early access. TypeSafe AI calls it the first "System One Model." It is not an LLM and does not write text. You send it a state plus questions, and it returns typed answers. This post is a catalog. It maps each common voice agent metric to a Jev question type, so you can score every call instead of a sample.

How Jev question types map to voice agent metrics

Jev answers three question types. Every voice agent metric fits one of them. Pick the type by the shape of the answer you need.

> Jev: a System One model that maps a state plus questions to typed probabilistic decisions with calibrated confidence. It classifies and scores; it does not generate language.

• Score returns an ordered rating, such as a 2–10 scale. You get a continuous score, a distribution over levels, and a confidence. Use it for graded quality like tone or effort.
• Noul returns a yes-or-no probability plus confidence. Use it for behaviors and events, such as "did the agent escalate?"
• Choice picks one of up to 255 options. It returns a probability per option plus confidence. Use it for labels like intent or failure mode.

The state is one call. That means the transcript, plus any structured data you attach. You can add a knowledge-base snippet, the tools called, or a CRM outcome. Then you ask questions about that state.

Here is the key move for a metric suite. You can ask many questions about one state in a single parallel call. Grading a call on twelve metrics costs barely more latency than grading it on one. TypeSafe reports 0% type errors, so a Noul is always a valid probability and a Choice always lands inside your option set.

Score metrics: graded, ordinal quality

Score fits any metric with degrees. These are the ratings a human would give on a scale, not a clean pass or fail. Feed the transcript state and ask for an ordered level.

Caller sentiment and sentiment delta. Do not settle for one sentiment number. Ask Jev to score sentiment at the start and at the end. The delta tells you whether the call improved or soured the caller's mood. That trend is more useful than a single reading. See sentiment analysis for the general idea, and our sentiment score for voice agents guide for the metric detail.

Agent empathy and tone appropriateness. Score how warm and fitting the agent's language was. This reads the words, not the voice. It catches a curt or tone-deaf reply in the transcript.

Response clarity and intelligibility. Score how clear and easy to follow each answer was. A high score means the caller could act on what the agent said. A low score flags jargon or muddled phrasing.

Caller effort. Score how hard the caller had to work to get help. This is a customer effort score applied to a transcript. High effort predicts churn and repeat contacts.

On-script and instruction-following adherence. Score how closely the agent followed your playbook. This is your policy rubric as a graded dimension. It shows drift before it becomes a compliance problem.

Verbosity. Score the agent on a terse-to-rambling scale. Too terse frustrates callers. Too long wastes handle time. The middle of the scale is usually the target.

Resolution completeness. Score how fully the agent finished the caller's goal. This is a degree, not a binary. A partial fix and a full fix are different, and a Score captures the gap.

Noul metrics: behaviors and events

Noul fits any metric that is a yes or no. Did a thing happen? Ask one Noul per behavior. Each returns a probability and a confidence.

Grounding check. This is the standout. Put a knowledge-base snippet into the state alongside the transcript. Then ask a Noul: "Did the agent state anything not supported by the provided knowledge?" A "yes" flags an unsupported claim. This is a hallucination detector for the agent under test. It measures the hallucination rate of the tested voice agent.

Read that last point carefully. This grounding check is distinct from Jev's own no-hallucination property. Jev itself cannot hallucinate, because its output is schema-constrained. That is a fact about the scorer. The grounding check, by contrast, is a metric about the agent you are testing. Do not conflate the two. One describes the tool; the other describes your agent.

Repetition and looping. Ask whether the agent repeated itself or got stuck in a loop. A "yes" points to a broken dialog state.

Caller repetition. Ask whether the caller had to repeat themselves. This is a friction signal. It often means the agent missed or dropped something.

Misunderstanding and recovery. Ask two linked Nouls. Was there a misunderstanding? And did the agent recover from it? Recovery matters as much as the error.

Correct escalation. Ask whether the agent escalated or transferred when it should have. This tracks handoff quality. Our guide on escalation in voice agents covers the trigger design.

Confirmation before critical actions. Ask whether the agent confirmed before an irreversible step. Think refunds, cancellations, or account changes. A missing confirmation is a real risk.

Sensitive-info leaks and scope. Ask whether the agent leaked sensitive data or went off-scope. Pair this with your PII-handling rules for voice agents. A "yes" is a fast alert.

Personalization. Ask whether the agent used the caller's name or context. Personalization lifts satisfaction. A Noul makes it measurable.

Choice metrics: one label per call

Choice fits any metric that assigns a single category. It picks one option from your fixed set, up to 255 of them. Each option gets a probability, plus an overall confidence.

Primary intent or topic. Label what the caller actually called about. This is the backbone of your reporting. It lets you slice every other metric by reason for contact.

Dominant failure mode. When a call fails, label the single biggest cause. Options might include wrong info, bad handoff, or dropped context. This turns vague failures into a countable distribution.

Furthest conversation phase reached. Label how far the call got. Options run in order, such as greeting, discovery, resolution, or wrap-up. This shows where callers drop off.

Choice is where you build the segments that make the rest of your dashboard readable. It answers "what kind of call was this?" in one field.

The jev voice agent metrics catalog

This table maps standout metrics to a question type. It shows a representative sample from each group. Use it as a starting point, then extend it with the metrics above.

You do not have to stop at these rows. Verbosity, clarity, and personalization all fit the same pattern. Add each one as another question in the same call.

How to define your voice agent metrics as Jev questions

Follow this order to turn a rubric into a working Jev metric suite. Start offline, prove calibration, then gate humans on low confidence.

1. Pick the metric. Take one line from your scorecard. Name it in plain words. "Did the agent confirm before the refund?" beats "compliance."

2. Choose Score, Noul, or Choice. Match the type to the answer shape. A degree is a Score. A yes or no is a Noul. A single label is a Choice.

3. Write the question against the state. Name what goes into the state. For most metrics that is the transcript. For a grounding check, add the knowledge-base snippet too. For intent, you might add the CRM record.

4. Set a confidence threshold. Decide how sure Jev must be to auto-accept a judgment. Tune the threshold to the human capacity you have.

5. Route low-confidence judgments to humans. Below the threshold, send the call to a reviewer. Above it, accept the score. This is confidence-gated human-in-the-loop review.

6. Batch many questions in one call. Attach every metric question to the state and send them together. One request returns the full scorecard in well under a second.

7. Validate against labeled calls. Hold out a human-reviewed sample. Confirm a reported 0.9 confidence maps to roughly 90% accuracy on your data. Check precision and recall per metric before you trust it.

The TypeSafe docs cover the request format and schemas. Build the suite offline first, where a wrong score costs nothing live.

Why one parallel call changes voice agent evaluation metrics

Three properties make this catalog practical at scale. Each one removes a reason teams sample instead of measuring everything.

First, Jev scores the whole metric suite in one parallel call. You send the state once. You ask every question at once. The full scorecard comes back together. Adding a metric adds almost no latency.

Second, output is free. TypeSafe prices input at $0.042 per million tokens, with output at no cost. TypeSafe reports that Jev runs up to 400x cheaper than a comparable LLM on classification tasks, a figure echoed in LangChain's write-up on building a harness with Jev. So you score every call, not a 2% sample.

Third, calibrated confidence tells you which judgments to trust. When Jev reports 0.9, it is right about 90% of the time. See calibration in statistics) for the formal idea. You accept high-confidence scores and route the rest to people. That makes full-coverage scoring affordable and safe at once.

Together these turn a metric library from a sampling exercise into a full census of your traffic. For a ready-made rubric to port over, start with our voice agent metrics scorecard.

The honest limit: a Jev score is not independent evidence

Jev is a strong scorer. It is not a complete evaluation program. Keep four limits in view.

Jev classifies and scores only. It gives you no free-text explanation of why a call went wrong. For a narrative, you still need an LLM or a human.

Jev works on the transcript and structured state, not raw audio. Tone, pace, and talk-over live in the waveform. Pair Jev with audio checks to catch them.

Calibrated does not mean guaranteed on any single call. Treat confidence as a routing signal across thousands of calls, not a per-call certificate.

Most important, a Jev score is not independent evidence on its own. Running Jev in-house is still you grading your own agent. You do not grade your own homework with a scorer you never validated. That is the gap Evalgent fills.

Evalgent is the independent third party. We use calibrated scorers like Jev where they help, then we validate the metric library against real outcomes on your calls. We run independent voice AI evaluation on your live traffic and your own test cases. The model background lives in our Jev for voice agents pillar, and the scoring workflow in Jev for voice agent evaluation. Jev is in early access, so treat every published figure as a starting point to verify.

Frequently asked questions

What are Jev voice agent metrics?

They are your quality rubric expressed as Jev questions. Each metric becomes a Score, Noul, or Choice question about one call's state. Jev returns typed decisions with calibrated confidence in 70–500ms. You can score a whole suite in one parallel call, then route low-confidence judgments to human reviewers before trusting the numbers.

Which Jev question type fits each metric?

Match the type to the answer shape. Use Score for graded, ordinal quality like tone, effort, and clarity. Use Noul for yes-or-no behaviors like escalation, confirmation, or a grounding check. Use Choice for a single label like intent, failure mode, or the furthest phase reached. Each returns calibrated confidence.

How do I score voice agents with Jev in one call?

Turn the call into a state, usually the transcript plus structured fields. Attach every metric as a question to that state. Send them together in one parallel request. Jev returns the full scorecard in well under a second. Store the confidence with each score so you can route uncertain judgments to people.

Can Jev detect hallucinations in the tested agent?

Yes, with a grounding check. Add a knowledge-base snippet to the state, then ask a Noul: did the agent state anything the provided knowledge does not support? A "yes" flags an unsupported claim. This is a detector for the agent under test. It differs from Jev's own inability to hallucinate.

What is a sentiment delta and why use it?

Sentiment delta is the change in caller mood from the call's start to its end. Ask Jev to score sentiment at both points, then compare. The trend shows whether the call helped or hurt the caller. That is more useful than a single sentiment reading, which hides how the interaction moved.

How many metrics can Jev score at once?

You attach many questions to one state and send them together. Adding metrics costs almost no extra latency, since it runs in parallel. Because output is free and priced per input token, scoring twelve metrics is nearly as cheap as one. This is why a full metric suite per call is practical, not just a sample.

Is a Jev score independent evaluation evidence?

No. Running Jev in-house is still you grading your own agent with an automated scorer. It is not independent third-party evidence. Jev can make your scoring faster, cheaper, and more consistent. An outside audit that validates the metrics against real outcomes on your calls is what holds up under procurement or a customer's questions.

Does Jev evaluate audio quality?

No. Jev reads the transcript and structured state, not the raw waveform. Acoustic issues like pace, clipping, and overlapping speech do not appear in text. You must pair Jev with audio-based checks to catch them. Treat transcript scoring and acoustic analysis as two separate layers of your voice agent evaluation stack.

The bottom line

Every voice agent metric maps cleanly to a Jev Score, Noul, or Choice, so you can score your whole rubric on every call in one fast, calibrated request. But a scorer you run yourself is not independent evidence, which is why Evalgent validates the metric library against real outcomes on your calls.

Want to see the full metric catalog scored and independently validated on your real traffic? Book a demo and we will run it on a sample of your actual calls.