Jev for Voice Agent Evaluation: Calibrated Scoring at Scale

Most teams grade voice agents with an LLM-as-judge on a thin sample of calls. That is slow, expensive, and noisy. Jev launched on September 15, 2026, in early access. TypeSafe AI calls it the first "System One Model."

This post covers evaluation methodology: what the public evaluator study measured, a full rubric written as Jev questions, and the meta-metrics that validate the scorer. For metric definitions, see our Jev voice agent metrics catalog.

What Jev actually does in an evaluation

Jev is not an LLM. It does not generate explanations or freeform critique. It takes a state and returns typed, probabilistic answers with calibrated confidence. TypeSafe summarizes it as "unstructured state in, typed probabilistic decisions out."

Jev: a System One model that maps a state plus questions to typed probabilistic decisions with calibrated confidence. It classifies and scores. It does not generate language.

Three design choices matter for evaluation. Jev is trained with RLCD, short for Reinforcement Learning for Calibrated Decisions. LLMs are trained with RLHF or RLVR instead. Jev also uses a new architecture and a parallel sampler. The sampler returns all outputs in one query, not token by token.

For evaluation, the state is one call. That means the transcript plus any structured data you attach. Useful additions include the detected intent, the tools called, the CRM disposition, and a knowledge base snippet.

Jev answers three question types:

• Choice picks one of up to 255 options. It returns a probability per option plus a confidence. Above 255 options, TypeSafe uses a two-stage score-then-choose approach. Use Choice for labels like intent or failure mode.
• Score rates against ordered levels you define. It returns a continuous score, a distribution, a legend, and a confidence. Use it for empathy, clarity, or caller effort.
• Noul returns a yes-or-no probability. Use it for pass/fail checks like a required disclosure.

Jev has 0% type errors by construction, because answers are matched to your schema. A Noul is always a valid probability. A Choice always lands inside your option set. You never parse a paragraph to find the verdict.

The LangChain study: 500 decisions, one agent

LangChain published a study titled "Can Jev Be a Better Agent Evaluator?" It is the most useful public evidence on Jev as a judge. It is also narrow. Read the design before you read the numbers.

How the study was designed

LangChain captured five runs of a weather agent. They then replayed each captured run 100 times. That produced 500 evaluation decisions in total.

The setup was fixed on purpose. Replaying captured runs removes variability from the agent and from live web results. Any difference between repeated scores therefore comes from the judge alone. That isolates judge consistency cleanly.

Each judge made two kinds of judgment. One was a binary `does_pass` decision, compared against human labels. The other was a quality score, used to measure variance across repeated runs of the same case.

What the study found

Here are the binary agreement results against human labels:

Jev's mean per-case quality-score variance was 0.0000149. That is close to zero. Scoring the same case again produced nearly the same number.

Cost and speed followed the same pattern. Jev averaged about 0.44 seconds and $0.00035 per call. The full run cost $0.34 with Jev versus $28.17 with Claude. These are LangChain's figures, not ours.

Why variance matters as much as agreement

Look closely at GPT-5.6 Terra. It agreed with humans 99.8% of the time. Yet its score variance was 913x higher than Jev's. So Terra usually reached the right verdict but gave wobbly scores on the same case.

That wobble has a real cost. Imagine a regression test between two agent versions. If the judge's score shifts on every rerun, you cannot tell a real regression from judge noise. Low variance is what makes before-and-after comparisons meaningful.

This connects to inter-rater reliability. A scorer that disagrees with itself cannot reliably agree with anyone else. We prefer the phrase "highly consistent" over "deterministic." Jev's variance was tiny, not zero.

The honest limits of one study

This study is strong evidence of one thing. Jev was a consistent, accurate judge of one weather agent's captured runs. It is not universal proof.

• One agent type. A weather agent is not a voice agent handling billing disputes. Your domain will differ.
• Five base cases. 500 decisions came from five runs replayed 100 times. That measures consistency well. It samples diversity of calls thinly.
• Text, not audio. The study graded agent traces. Voice calls add transcription errors and acoustic issues the study did not touch.
• Easy binary task. Three of four judges scored above 96% on `does_pass`. Harder, fuzzier rubric items may separate judges differently.
• Early access. Jev versions change. The docs example already reports `jev-1.13.0`. Recheck results when the model updates.

So treat the study as a reason to test Jev, not a reason to skip testing. The rest of this post shows how to run that test on your own calls.

A full voice agent evaluation rubric as Jev questions

A useful rubric covers more than "did the call succeed." It needs outcome, compliance, grounding, conversation quality, caller experience, and diagnostic labels. Each item becomes one Jev question. The full catalog with definitions lives in our voice agent metrics guide. Here we focus on how to phrase and type each one.

Outcome metrics

Outcome metrics answer the business question: did the call do its job?

• Task success (Noul). "Did the agent complete the caller's primary stated goal?" A yes/no probability is the right shape.
• Resolution completeness (Score, 4 levels). Levels: not addressed, partially addressed, addressed with gaps, fully resolved. This catches calls that "succeed" but leave loose ends.
• Containment vs correct escalation (Choice). Options: contained correctly, escalated correctly, escalated unnecessarily, should have escalated but did not. Plain containment rate rewards trapping callers. This four-way split separates good containment from bad. Our escalation guide covers the escalation rules in depth.

Compliance metrics

Compliance items are usually strict rules. Write one Noul per rule, never one vague "was it compliant" question.

• Policy adherence (Noul). "Did the agent stay within the refund policy in the state?" Put the policy text in the state so Jev can compare.
• Required disclosures (Noul, one per disclosure). "Did the agent state that the call is recorded?" Split each disclosure into its own question.
• PII exposure (Noul). "Did the agent read back a full card number, SSN, or other sensitive identifier?" See our PII handling guide for what counts.
• Confirm before irreversible action (Noul). "Before cancelling, charging, or deleting, did the agent get explicit caller confirmation?" Pair this with the tool call log in the state. Our tool-calling guide explains how to attach tool traces.

For disclosures with exact required wording, a deterministic regex is often the better ground truth. Use Jev for paraphrased disclosures that a regex misses.

Grounding metrics

Grounding asks whether the agent said things it had no basis for.

• Unsupported claim (Noul). "Did the agent state any fact not supported by the knowledge base snippet in the state?" This only works if the relevant snippet is in the state. Jev cannot check facts it cannot see.
• Grounding quality (Score, 3 levels). Levels: contradicts source, partially supported, fully supported. Use this when a binary flag is too blunt.

Conversation quality metrics

These capture how the call felt, turn by turn.

• Empathy and tone (Score, 4 levels). Cold or dismissive, neutral, warm, warm and adaptive to caller emotion.
• Clarity (Score, 3 levels). Confusing, mostly clear, clear and concise.
• Verbosity (Score, 3 levels). Too terse, appropriate, too long for voice. Long monologues hurt more on a phone than on screen.
• Repetition or loop (Noul). "Did the agent repeat the same question or response three or more times?"
• Caller had to repeat (Noul). "Did the caller have to restate information they already gave?"
• Misunderstanding and recovery (Choice). Options: no misunderstanding, misunderstood and recovered, misunderstood and never recovered.
• Personalization (Score, 3 levels). Generic, used caller context once, used caller context throughout.

Caller experience metrics

These estimate what the caller would say if you asked them. Treat them as inferred signals, not survey results.

• Caller effort (Score, 5 levels, CES-style). Very low effort through very high effort. Loosely modeled on the Customer Effort Score.
• Sentiment at start and end (two Scores, 5 levels each). Ask twice: once about the opening turns, once about the closing turns. The delta, end minus start, is your sentiment shift.
• Inferred CSAT (Score, 5 levels). Very dissatisfied through very satisfied. Validate this against real post-call surveys where you have them.

Diagnostic labels

Labels do not grade the call. They explain it, so you can slice every other metric.

• Intent or topic (Choice). Your intent taxonomy, up to 255 options.
• Dominant failure mode (Choice). Options such as none, misrecognition, wrong tool, policy breach, hallucination, loop, premature end, latency complaint.
• Furthest phase reached (Choice). Options such as greeting, identification, verification, resolution, wrap-up. This shows where calls stall.

The metric-to-question map

Some things stay off this table. Latency, talk-over, and robotic cadence live in audio and timing data, not text. Measure those with timing logs and audio checks. Our latency guide and our post on transcript versus audio evaluation cover that layer.

Build the rubric as questions in code

The whole rubric fits in one request. Jev's parallel sampler answers every question in a single query. Grading twenty dimensions costs barely more latency than grading one.

The raw API request

Here is a trimmed request to the real endpoint. It shows one question of each type.

{
  "model": "jev-latest",
  "state": "KB: Refunds allowed within 30 days with receipt.\nTOOLS: lookup_order(ok), issue_refund(ok)\nTRANSCRIPT:\nAgent: This call is recorded. How can I help?\nCaller: I want a refund on order 4417.\nAgent: I see it was bought 12 days ago. I can refund it now. Shall I go ahead?\nCaller: Yes please.\nAgent: Done. You will see it in 5 business days.",
  "questions": {
    "task_success": {
      "type": "noul",
      "instructions": "Did the agent complete the caller's primary stated goal?"
    },
    "escalation": {
      "type": "choice",
      "instructions": "Classify how the agent handled containment and escalation.",
      "criteria": {
        "contained_correctly": "Resolved without a human, and that was appropriate",
        "escalated_correctly": "Handed to a human when policy or complexity required it",
        "escalated_unnecessarily": "Handed off a call it could have resolved",
        "missed_escalation": "Should have escalated but did not"
      }
    },
    "caller_effort": {
      "type": "score",
      "instructions": "How much effort did the caller spend to get their outcome?",
      "criteria": ["Very low", "Low", "Moderate", "High", "Very high"]
    }
  }
}

The response shape

The response follows the documented shape. The values below are illustrative, not from a real run.

{
  "model": "jev-1.13.0",
  "answers": {
    "task_success": {"type": "noul", "noul": 0.97},
    "escalation": {
      "type": "choice",
      "choice": "contained_correctly",
      "confidence": 0.91,
      "probabilities": {
        "contained_correctly": 0.94,
        "escalated_correctly": 0.0,
        "escalated_unnecessarily": 0.01,
        "missed_escalation": 0.05
      }
    },
    "caller_effort": {
      "type": "score",
      "score": 0.3,
      "confidence": 0.88,
      "legend": {"0": "Very low", "1": "Low", "2": "Moderate", "3": "High", "4": "Very high"},
      "probabilities": {"0": 0.7, "1": 0.3, "2": 0.0, "3": 0.0, "4": 0.0}
    }
  },
  "usage": {"input_tokens": 412, "output_tokens": 30}
}

Notice what you get for free. Each Choice and Score carries a confidence and a full distribution. The Score legend echoes your levels, so downstream code never guesses what "3" means.

The full rubric in Python

This uses the `typesafe-sdk` package. It bundles most of the rubric above into one `system_one` call per transcript.

# pip install typesafe-sdk
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()  # reads TYPESAFE_API_KEY, defaults to jev-latest

RUBRIC = {
    # Outcome
    "task_success": Noul(instructions="Did the agent complete the caller's primary stated goal?"),
    "resolution": Score(
        instructions="How completely was the caller's issue resolved?",
        criteria=["Not addressed", "Partially addressed", "Addressed with gaps", "Fully resolved"],
    ),
    "escalation": Choice(
        instructions="Classify containment and escalation handling.",
        criteria={
            "contained_correctly": "Resolved without a human, appropriately",
            "escalated_correctly": "Handed off when policy or complexity required it",
            "escalated_unnecessarily": "Handed off a call it could have resolved",
            "missed_escalation": "Should have escalated but did not",
        },
    ),
    # Compliance
    "recording_disclosure": Noul(instructions="Did the agent state that the call is recorded?"),
    "policy_breach": Noul(instructions="Did the agent offer anything outside the policy in the state?"),
    "pii_exposed": Noul(instructions="Did the agent read back a full card number or government ID?"),
    "confirmed_irreversible": Noul(
        instructions="Before any refund, charge, or cancellation, did the agent get explicit confirmation?"
    ),
    # Grounding
    "unsupported_claim": Noul(instructions="Did the agent state a fact not supported by the KB in the state?"),
    # Conversation quality
    "empathy": Score(
        instructions="Rate the agent's empathy and tone.",
        criteria=["Cold or dismissive", "Neutral", "Warm", "Warm and adaptive"],
    ),
    "verbosity": Score(
        instructions="Rate the length of agent turns for a phone call.",
        criteria=["Too terse", "Appropriate", "Too long for voice"],
    ),
    "loop": Noul(instructions="Did the agent repeat the same question three or more times?"),
    "caller_repeated": Noul(instructions="Did the caller restate information they already gave?"),
    "recovery": Choice(
        instructions="Did a misunderstanding occur, and was it recovered?",
        criteria={
            "none": "No misunderstanding",
            "recovered": "Misunderstood, then recovered",
            "unrecovered": "Misunderstood and never recovered",
        },
    ),
    # Caller experience
    "effort": Score(
        instructions="How much effort did the caller spend?",
        criteria=["Very low", "Low", "Moderate", "High", "Very high"],
    ),
    "sentiment_start": Score(
        instructions="Caller sentiment in the first three caller turns.",
        criteria=["Very negative", "Negative", "Neutral", "Positive", "Very positive"],
    ),
    "sentiment_end": Score(
        instructions="Caller sentiment in the last three caller turns.",
        criteria=["Very negative", "Negative", "Neutral", "Positive", "Very positive"],
    ),
    # Labels
    "failure_mode": Choice(
        instructions="What was the dominant failure mode, if any?",
        criteria={
            "none": "No material failure",
            "misrecognition": "Agent misheard or misparsed the caller",
            "wrong_tool": "Called the wrong tool or wrong arguments",
            "policy_breach": "Broke a stated policy",
            "hallucination": "Stated unsupported facts",
            "loop": "Got stuck repeating itself",
            "premature_end": "Ended or dropped the call too early",
        },
    ),
    "phase_reached": Choice(
        instructions="What was the furthest phase the call reached?",
        criteria={
            "greeting": "Greeting only",
            "identification": "Caller identified",
            "verification": "Identity verified",
            "resolution": "Resolution attempted",
            "wrap_up": "Call wrapped up",
        },
    ),
}


def build_state(call: dict) -> str:
    return (
        f"KB: {call['kb_snippet']}\n"
        f"POLICY: {call['policy']}\n"
        f"TOOLS: {call['tool_log']}\n"
        f"TRANSCRIPT:\n{call['transcript']}"
    )


def score_call(call: dict, threshold: float = 0.85) -> dict:
    response = client.system_one(state=build_state(call), questions=RUBRIC)
    row, review = {"call_id": call["id"]}, []
    for name, ans in response.answers.items():
        if ans.type == "noul":
            row[name] = ans.noul
            # Noul has no separate confidence; treat mid-range probabilities as uncertain
            if 0.2 < ans.noul < 0.8:
                review.append(name)
        else:
            row[name] = ans.choice if ans.type == "choice" else ans.score
            row[f"{name}_conf"] = ans.confidence
            if ans.confidence < threshold:
                review.append(name)
    row["sentiment_delta"] = row["sentiment_end"] - row["sentiment_start"]
    row["needs_review"] = review
    return row

Three details matter. The KB, policy, and tool log live in the state, because grounding and compliance need them. Every confidence is logged. Uncertain answers are flagged per metric, so a reviewer checks only the fuzzy item.

The 0.85 threshold and the 0.2 to 0.8 Noul band are placeholders. You will tune both from your own validation data, described next.

If you already use LangChain, the `langchain-typesafe` package exposes the same idea. `TypeSafeClassifier` accepts a state and questions such as `Noul(instructions=...)`. You read results from fields like `response.nouls["urgent"].noul`.

Meta-metrics: validate the scorer itself

A rubric tells you what to measure. Meta-metrics tell you whether your scorer measures it correctly. Skip this step and every dashboard number is an unverified claim.

Run these six checks on a held-out set of human-labeled calls. A few hundred calls per metric is a sensible starting point.

1. Agreement with human labels (Cohen's kappa)

Raw percent agreement flatters easy tasks. If 90% of calls pass, a scorer that always says "pass" gets 90% agreement. Cohen's kappa corrects for chance agreement.

Cohen's kappa: (observed agreement minus chance agreement) divided by (1 minus chance agreement). A kappa of 0 means chance-level. A kappa of 1 means perfect agreement.

Illustrative example: Jev and humans agree on 92% of calls, and chance agreement is 70%. Kappa is (0.92 - 0.70) / (1 - 0.70), or about 0.73. That is solid but not perfect.

Also compute precision and recall for rare, high-stakes flags. For PII exposure, recall matters most. A missed leak costs more than a false alarm. Measure human-to-human kappa too. If two humans only reach 0.6 on empathy, do not expect a scorer to beat that ceiling.

2. Calibration (reliability and ECE)

Calibration asks whether confidence means what it says. When Jev reports 0.9 confidence, it should be right about 90% of the time on your data. TypeSafe reports that higher confidence maps to higher accuracy. You still need to confirm it on your calls. See calibration in statistics) for the formal idea.

Build a reliability table. Bucket answers by confidence, such as 0.5–0.6, 0.6–0.7, and so on. In each bucket, compare mean confidence with actual accuracy against human labels.

Expected calibration error (ECE): the weighted average gap between confidence and accuracy across buckets. Each bucket is weighted by its share of answers. Lower is better.

An illustrative case: the 0.9 bucket holds 40% of answers and is 86% accurate. Its gap is 0.04, contributing 0.016 to ECE. Sum that across all buckets. A good calibration curve is what lets you pick a review threshold with confidence.

3. Consistency across reruns

Score the same call several times and measure the variance. This is what the LangChain study measured with its 100 replays. Jev's mean per-case variance there was 0.0000149.

Rerun a sample of your own calls five to ten times. Report mean per-case variance per Score metric. Also report label flip rate for Choice and Noul metrics. A high flip rate means your regression tests will produce false alarms.

4. Coverage

Coverage is the share of calls actually scored. A scorer that skips long calls quietly biases results, since long calls are often the troubled ones. Track percent scored and percent sent to review. Full coverage is the point of cheap scoring, as covered in score every call, not a sample.

5. Cost per judgment

Jev input costs $0.042 per million tokens, or $42 per billion. Output is free. Frontier LLM input runs about $0.20 to $10 per million tokens, with output around 5x input.

Illustrative arithmetic: a 3,000-token call state costs about $0.000126 to score at Jev's input price. Scoring 100,000 calls a month would cost about $12.60 in input tokens. Asking twenty questions in one call does not multiply that by twenty, because the state is sent once. Measure your real token counts from the `usage` field.

6. Latency

Jev returns answers in 70–500ms. Frontier LLMs take about 3 to 329 seconds on comparable tasks, per TypeSafe. That 40–200x gap matters for near-real-time monitoring and fast CI runs. TypeSafe's own workflow evals report 193.6x faster and 444.6x cheaper results. TypeSafe notes these are likely the high end of real-world gains.

Track p50 and p95 latency per call in your pipeline. The tail matters more than the median for live dashboards.

A meta-metric scorecard

How to build a Jev-based scoring pipeline

Follow this order to stand up jev for voice agent evaluation without over-trusting it. Start offline, prove calibration on your data, then gate human review on confidence.

1. Define your rubric as typed questions. Write each metric as a Choice, Score, or Noul. Keep wording specific. "Did the agent state the call is recorded?" beats "Was the call compliant?"

2. Convert each call into a state. Assemble the transcript, KB snippet, policy text, tool log, and CRM disposition. Missing context produces confident but uninformed answers.

3. Label a validation set. Have two humans label a few hundred calls per metric. Compute human-to-human kappa to set your ceiling.

4. Score the validation set with Jev. Send all questions in one parallel call per transcript. Log every probability and confidence.

5. Run the six meta-metrics. Kappa, precision and recall, ECE, rerun variance, coverage, cost, and latency. Drop or rewrite any question that fails.

6. Set per-metric review thresholds. Use the reliability curve to pick a confidence cutoff for each metric. Match the review volume to your human capacity.

7. Add deterministic and audio checks. Use regex for exact-wording disclosures. Use timing and audio analysis for latency and talk-over. Jev is one layer, not the whole stack.

8. Monitor drift and recalibrate. Recheck agreement and calibration when your agent, prompts, callers, or the Jev version changes. Our post on voice agent metric drift covers what to watch.

The TypeSafe docs and quickstart cover the request format and schemas in detail.

Why calibrated confidence changes human review

Calibrated confidence: when the model reports 0.9 confidence, it is right about 90% of the time. Higher confidence means higher accuracy.

TypeSafe's core argument: a model right 95% of the time that cannot say which 5% is wrong cannot automate a task. A calibrated model points at the 5%.

That enables confidence-gated review, the heart of human-in-the-loop voice agent evaluation. Confident scores are accepted. Uncertain ones go to a person. A 0.95 answer can still be wrong, but across thousands of calls you can estimate how many errors slip through.

The honest limits you must design around

Jev is a strong scorer, not a complete evaluation program. Five limits shape its use.

• No rationale. Jev decides and scores only. Narrative explanations still need an LLM or a human. See our post on LLM-as-judge limits.
• Text, not audio. Jev reads transcript state. Transcription errors flow straight into it.
• Calibrated is not correct on your data. Confidence earns trust only after validation.
• One study is not universal proof. The LangChain result is strong but narrow.
• Self-grading is not independent evidence. A vendor grading its own agent with any scorer is not an outside audit.

Where Evalgent fits

A scorer is not an evaluation program. Jev can be your scoring engine. Evalgent is your independent evaluator. Those are different jobs.

Run Jev in-house as a fast, calibrated judge. But the LangChain result covers one agent, not your billing disputes. Grading your own agent with your own Jev calls is still marking your own homework.

Evalgent runs independent voice AI evaluation on your real calls. We build the rubric with you, label a validation set, and measure kappa, calibration, and rerun consistency on your traffic. We combine calibrated scorers like Jev with deterministic verifiable rewards and human review. We benchmark on your own data, not a generic set. For background on the model, start with our Jev for voice agents pillar. For the judge comparison, see Jev as a judge for voice agents.

Jev can't abstain: always include an escape option

Jev always returns one of the options you define. It has no built-in "I don't know." When a call transcript holds no evidence for any option, such as a two-word turn, a dropped call or a caller talking about something else, Jev still picks the least-wrong option, and that label looks just like a confident answer. TypeSafe's own Choice docs recommend adding an "other" or "none of the above" option when the list may not cover every input.

In evaluation the cost is a silent error in your metrics. Calls where a criterion doesn't apply should land in `not_applicable` and be excluded from the pass rate, not be counted as a pass or a fail.

Four rules keep this from hurting you on live calls:

• Add an explicit escape option such as `not_enough_information`, `not_applicable` or `none_of_the_above` to every Choice question where evidence can be missing. Describe it as concretely as the real options, because Jev reads only the descriptions, not your option keys.
• Ask whether the evidence exists first. A Noul like "does the caller state a reason for calling?" before the Choice question separates "no evidence" from "evidence for option B".
• Check the distribution, not just the label. In one independent edge-case test, a forced choice with no valid option still returned a pick at 0.52 probability but only 0.04 confidence (jev-exploration). The numbers flagged the problem; the label did not. Set your human-review cutoff from your own labeled calls, because an independent study on data unlike Jev's training found its confidence can run high (jev-ood-calibration).
• Track the escape-option rate. If it climbs, your transcripts, STT or option set have drifted away from what callers actually say.

Frequently asked questions

How do you use Jev for voice agent evaluation?

Turn each call into a state with the transcript, KB snippet, policy, and tool log. Write every rubric metric as a Choice, Score, or Noul question. Send them all in one call. Jev returns typed answers with calibrated confidence in 70–500ms. Log every confidence, auto-accept confident answers, and route uncertain ones to human review.

Can Jev replace an LLM judge for voice agents?

For anything that fits a Choice, Score, or Noul, often yes. In LangChain's 500-decision study, Jev matched human labels on all 500 decisions. LLM judges scored 80.0% to 99.8% with 92–913x higher score variance. Jev gives no rationale, so keep an LLM or human for narrative feedback and root-cause analysis.

Why is calibrated confidence useful for scoring voice agents?

Calibrated confidence means a reported 0.9 maps to roughly 90% accuracy. That gives you a built-in triage signal. You accept high-confidence scores automatically and route low-confidence ones to a person. Human attention goes only where the scorer is unsure. Confirm calibration on your own labeled calls before relying on it.

Is Jev-based scoring independent evaluation evidence?

No. Running Jev on your own agent is still an automated judge operated by you. It is not independent third-party evidence. Jev can make in-house scoring faster and more consistent. An outside audit on your real calls, with validated agreement and calibration, is what holds up under procurement review or customer questions.

What Jev question type should I use for each dimension?

Match the type to the answer shape. Use Noul for yes/no rules like task success, disclosures, PII exposure, and unsupported claims. Use Score for ordered ratings like empathy, clarity, and caller effort. Use Choice for fixed label sets like intent, failure mode, escalation handling, and furthest phase reached.

Does Jev evaluate audio quality in voice agents?

No. Jev reads the transcript and structured state, not the raw waveform. Tone of voice, pace, clipping, and talk-over do not appear in text. Transcription errors also carry into the state. Pair Jev with timing logs and audio-based checks, and treat transcript scoring and acoustic analysis as separate layers.

How do I validate Jev scores against ground truth?

Label a few hundred calls per metric with two humans. Score them with Jev. Compute Cohen's kappa against human labels and compare it to human-to-human kappa. Build a reliability curve and expected calibration error. Rerun calls to measure variance. Recheck whenever your agent, prompts, callers, or Jev version changes.

How much does it cost to score every call with Jev?

Jev input costs $0.042 per million tokens, and output is free. LangChain reported about $0.00035 per call and $0.34 for its run versus $28.17 with Claude. As illustrative math, a 3,000-token call costs about $0.000126 to score. At that price, full coverage replaces sampling.

The bottom line

Jev makes it practical to score every voice agent call on a full rubric with calibrated, highly consistent decisions. Its numbers become trustworthy only after you validate agreement, calibration, and consistency on your own labeled calls.

Want independent validation of your scorer on your real traffic? Book a demo and we will measure agreement and calibration on a sample of your calls.