Verifiable Rewards vs LLM Judges vs Jev: 3 Ways to Score Voice Calls

Most teams score voice agents in one of two ways. They argue about which way is better. One camp writes code to check facts. Did the booking post? Was the refund amount right? The other camp asks a language model to grade the fuzzy stuff. Was the agent warm? Did the answer help?

Both camps are half right. And in September 2026 a third option arrived. TypeSafe AI shipped Jev, its first System One model. Jev returns typed, calibrated decisions. It does not generate text. It sits between rigid rules and a free-form judge.

This post goes past the definitions. We take one real-shaped voice call and score it three ways, with working code for each. Then we put hard numbers on agreement, variance, cost, and latency. Finally, we show how to stack all three so each does the job it is best at. If you want the broader Jev vs LLM question, covering replies, routing, and reasoning rather than scoring, read Jev vs LLM instead.

It builds on the two-way split in verifiable rewards vs LLM judges. That post predates Jev. This one adds the third paradigm and the evidence behind it.

The three ways to score a voice agent

Every scoring method answers one question. How do you turn a call into a number you trust? The three paradigms answer it differently.

Verifiable reward: a deterministic, programmatic check with a right answer known in advance. Code confirms an objective fact about the call. The check returns the same result every run.

LLM-as-judge: a language model prompted to grade a transcript against a rubric. It returns a score and usually a rationale. The score is an estimate, and it can change between runs.

System One model (Jev): a model that takes state plus typed questions and returns probabilistic decisions with calibrated confidence. It classifies and scores. It does not write text or explain itself.

These are not rivals for one job. They cover different parts of the same call. This is the scoring layer under any serious voice agent evaluation program.

Where each paradigm comes from

The names hint at how each method was born. It helps to know the lineage.

"Verifiable rewards" comes from model training. TypeSafe frames it this way. LLMs are often optimized with RLHF, or reinforcement learning from human feedback. That rewards outputs people prefer. They are also optimized with RLVR, or reinforcement learning with verifiable rewards. That rewards outputs a program can check. In evaluation, the same idea becomes an assertion. It works like a unit test for a call.

LLM judges inherit the RLHF mindset. They were trained to produce text that humans like. So they are good at fluent, reasoned critique. They are not trained to give the same number twice.

Jev takes a third route. TypeSafe trained it with RLCD, which stands for Reinforcement Learning for Calibrated Decisions. The target is not a preferred sentence or a checkable string. The target is a decision whose confidence matches reality. When a calibrated model says 90% confident, it should be right about 90% of the time. That is the textbook meaning of calibration).

One voice call, scored three ways

Abstract comparisons hide the real trade-offs. So let us score a single call. Here is the setup.

A caller phones a dental clinic to book a cleaning. The agent must read a recording disclosure. It must confirm the date before booking. It should stay patient if the caller hesitates. Here is a trimmed transcript and the tool log.

{
  "call_id": "c_48213",
  "turns": [
    {"role": "agent", "text": "Hi, this is Bright Smile Dental. This call may be recorded for quality."},
    {"role": "caller", "text": "Hi, I need a cleaning. Maybe next Tuesday?"},
    {"role": "agent", "text": "Sure. Next Tuesday is October 6. Does 10 a.m. work?"},
    {"role": "caller", "text": "Hmm, can we do the afternoon? Sorry, I'm juggling work."},
    {"role": "agent", "text": "No problem at all. I have 2:30 p.m. on Tuesday, October 6. Shall I book it?"},
    {"role": "caller", "text": "Yes, 2:30 on the 6th works."},
    {"role": "agent", "text": "You're booked for Tuesday, October 6 at 2:30 p.m."}
  ],
  "tool_calls": [
    {"name": "create_booking", "args": {"date": "2026-10-06", "time": "14:30", "service": "cleaning"}}
  ]
}

Three questions matter for this call. Did the agent do the task correctly? Did it say the required words? Was it good to talk to? Each paradigm handles these differently.

(a) Verifiable checks in Python

Start with what has a right answer. The caller confirmed October 6 at 2:30 p.m. The tool log shows what the agent actually booked. The disclosure is a fixed string. None of this needs a model.

import re
from datetime import date

def verify_call(call: dict, confirmed_date: date, confirmed_time: str) -> dict:
    agent_text = " ".join(t["text"] for t in call["turns"] if t["role"] == "agent")
    bookings = [c for c in call["tool_calls"] if c["name"] == "create_booking"]

    results = {}

    # 1. Exactly one booking tool call was made.
    results["single_booking"] = len(bookings) == 1

    # 2. The booking used the date and time the caller confirmed.
    if bookings:
        args = bookings[0]["args"]
        results["date_matches"] = args["date"] == confirmed_date.isoformat()
        results["time_matches"] = args["time"] == confirmed_time
    else:
        results["date_matches"] = results["time_matches"] = False

    # 3. The required disclosure appears in an agent turn.
    results["disclosure_spoken"] = bool(
        re.search(r"\bmay be recorded for quality\b", agent_text, re.IGNORECASE)
    )

    # 4. The disclosure came in the first agent turn.
    first_agent = next(t["text"] for t in call["turns"] if t["role"] == "agent")
    results["disclosure_first"] = "recorded" in first_agent.lower()

    # 5. No forbidden phrases (a policy list you maintain).
    forbidden = [r"\bguarantee\b", r"\bdiagnos\w*\b"]
    results["no_forbidden"] = not any(re.search(p, agent_text, re.I) for p in forbidden)

    return results

checks = verify_call(call, date(2026, 10, 6), "14:30")
assert all(checks.values()), f"Failed checks: {[k for k, v in checks.items() if not v]}"

Every line has a ground truth known before the call. Anyone can rerun it and get the same verdict. That is reproducibility, and it is why a check can gate a release.

The checks also resist gaming. An agent cannot talk its way past a tool log. If it books 10 a.m. instead of 2:30, check two fails. No phrasing changes that.

Now notice what the checks cannot see. Was the agent patient when the caller hesitated? Did it sound warm or robotic? No regex answers that. You could fake it with a rule like "contains 'no problem'." But that proxy is brittle and easy to game. So checks anchor the checkable, and stop there.

(b) An LLM judge prompt

Next, the nuanced dimension. Here is a typical judge prompt. It asks for a rubric score and a rationale.

JUDGE_PROMPT = """You are grading a voice agent call transcript.

Rubric: EMPATHY (1-5)
1 = dismissive or robotic; ignores caller cues
3 = polite but generic; acknowledges needs mechanically
5 = warm and adaptive; acknowledges hesitation and adjusts smoothly

Return JSON only:
{"empathy_score": <1-5>, "rationale": "<one or two sentences>"}

Transcript:
{transcript}
"""

A typical response looks like this.

{
  "empathy_score": 4,
  "rationale": "The agent reassured the caller with 'No problem at all' and offered an afternoon slot, though it did not acknowledge the work stress directly."
}

The rationale is the judge's real strength. A developer reading it knows what to fix. No other paradigm here writes that sentence.

But run the same prompt again. You might get a 4, then a 5, then a 4. The score is sampled, not fixed. Small prompt edits also move it. We cover these failure modes in the limits of LLM-as-judge for voice AI. There are also position and verbosity biases. And each call costs a long prompt plus paid output tokens.

The judge is also gameable. Goodhart's law warns that a measure that becomes a target stops measuring well. An agent tuned against a judge learns what the judge rewards. It may grow wordier or more effusive. The score rises. Real quality may not.

(c) A Jev request with Score and Noul questions

Now the same call through Jev. Jev takes state plus typed questions. It supports three question types.

Here is a raw request to the real endpoint. The field layout inside each question is simplified for readability. Check the TypeSafe documentation for the exact schema.

curl https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "AGENT: Hi, this is Bright Smile Dental. This call may be recorded for quality. CALLER: Hi, I need a cleaning. Maybe next Tuesday? ...",
    "model": "jev-latest",
    "questions": {
      "empathy": {
        "type": "score",
        "question": "How well did the agent acknowledge and adapt to the caller?",
        "levels": ["dismissive", "generic", "polite", "warm", "warm and adaptive"]
      },
      "confirmed_before_booking": {
        "type": "noul",
        "question": "Did the agent confirm the date and time with the caller before booking?"
      }
    }
  }'

The response comes back as typed answers. The values below are illustrative, not from a real run.

{
  "answers": {
    "empathy": {
      "score": 3.9,
      "confidence": 0.91,
      "probabilities": [0.00, 0.02, 0.17, 0.69, 0.12],
      "legend": ["dismissive", "generic", "polite", "warm", "warm and adaptive"]
    },
    "confirmed_before_booking": {
      "noul": true,
      "confidence": 0.97,
      "probabilities": {"yes": 0.97, "no": 0.03}
    }
  },
  "usage": {"input_tokens": 412, "output_tokens": 0}
}

The same request through the Python SDK is shorter. Treat this as a sketch. Constructor arguments may differ, so confirm them in the docs.

import os
from typesafe_sdk import Noul, Score, TypeSafeClient

client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"])

result = client.system_one(
    state=transcript_text,
    model="jev-latest",
    questions={
        "empathy": Score(
            "How well did the agent acknowledge and adapt to the caller?",
            levels=["dismissive", "generic", "polite", "warm", "warm and adaptive"],
        ),
        "confirmed_before_booking": Noul(
            "Did the agent confirm the date and time with the caller before booking?"
        ),
    },
)

empathy = result.answers["empathy"]
if empathy.confidence < 0.80:
    send_to_review(call_id, reason="low-confidence empathy score")

Three things stand out. First, the answer always fits the schema. A Score is a score and a Noul is yes or no. There are 0% type errors by construction. You never parse a malformed reply.

Second, you get a full distribution, not one sampled number. You can see the model split between "warm" and "polite." That is more honest than a bare 4.

Third, confidence is calibrated. That makes it a routing signal. The last lines of the SDK sketch send low-confidence cases to review. You cannot do that reliably with a judge's self-reported certainty.

What you lose is the rationale. Jev will not tell you why it scored 3.9. If you need the "why," you need a judge or a human.

What the evidence says: LangChain's study

Claims about consistency are easy to make. LangChain measured them. Their team ran Jev as an evaluator inside LangSmith and published the results.

The design was simple. They scored 100 cases, five runs each, for 500 decisions. Each decision was a pass or fail on whether an agent output met the bar. They compared every scorer against human labels.

Here are the full numbers.

The "derived" variance values are simple arithmetic. We multiplied Jev's reported variance by each reported ratio. They are illustrative.

Read the table carefully. Agreement and variance tell different stories. GPT-5.6 Terra nearly matched Jev on agreement. Yet its scores moved the most between runs. Claude Sonnet 4.6 varied less than Terra but agreed with humans far less often.

That split matters for voice agents. A judge can be right on average and still wobble on any single call. If you rerun a regression suite and scores shift, you cannot tell a real change from noise. Low variance is what makes a score comparable across releases.

LangChain also reported cost and speed. Jev averaged about $0.00035 per call and about 0.44 seconds. The whole run cost $0.34 with Jev, against $28.17 with Claude.

Keep the scope honest. This is one study, on one task, in one harness. It is strong evidence, not a universal law. Your calls, rubrics, and labels differ. Validate on your own data before you trust any scorer, including Jev.

Cost and latency per 1,000 calls

Now scale it up. Most production voice agents handle thousands of calls a day. Here is what each paradigm costs per 1,000 calls.

The pricing inputs are these. Jev costs $0.042 per million input tokens, and output is free. Frontier LLMs run about $0.20 to $10 per million input tokens. Output typically costs around five times the input rate.

For an illustrative judge call, assume 4,000 input tokens and 400 output tokens. That covers a five-minute transcript, the rubric, and a short rationale.

The judge math is simple. At $0.20 per million input tokens, 4,000 tokens cost $0.0008. At five times that rate, 400 output tokens cost $0.0004. That totals about $0.0012 a call. At $10 per million, the same call costs about $0.06. The Claude row divides $28.17 by 500 decisions.

Latency compounds too. At 0.44 seconds, 1,000 sequential Jev calls take about seven minutes. At ten seconds per judge call, the same batch takes nearly three hours. Parallelism helps both, but the gap stays.

The takeaway is not "judges are too expensive." A cheap judge on 1,000 calls costs about a dollar. The takeaway is about what you can afford to do every day. Scoring every call, rerunning suites, and testing rubric changes all multiply cost. That is where a cheap, fast scorer changes behavior. It is the logic behind scoring every call, not a sample.

The three paradigms compared

Here is the full comparison across the axes that decide which one to reach for.

The pattern is clear. Checks win on trust and lose on coverage. Judges win on explanation and lose on cost and consistency. Jev is broad, cheap, and consistent, but silent on the "why." For a deeper head-to-head on the model side, see Jev vs LLMs for voice agents.

When each is the right tool

Use a simple test for every metric on your scorecard. Ask what kind of question it is.

Is there a right answer a system of record can confirm? Use a verifiable check. Examples include booking created, correct date, refund amount, required disclosure, forbidden phrases, transfer to the right queue, and fields collected. If code and a fair human would always agree, write code.

Is it a judgment call you need on every call, as a number? Use Jev. Examples include empathy level, intent class, sentiment, whether the agent confirmed before acting, and whether the caller's issue sounded resolved. These are the rubric dimensions covered in Jev voice agent metrics.

Do you need the reasoning, not just the verdict? Use an LLM judge, sparingly. Examples include root-causing a failed escalation, explaining a disputed score, or drafting coaching notes. Validate the judge against human graders first. Use inter-rater reliability to measure it.

Is it high-stakes and ambiguous? Use a human. Examples include compliance disputes, safety edge cases, and calibration sets for every other scorer. Our human-in-the-loop guide covers how to spend that time well.

A layered scoring stack

The goal is not to pick a winner. It is to route each part of a call to the scorer that handles it best. The stack below does that in three layers.

Layer 1: Verifiable checks on every call. Run them first because they are cheap and exact. A failed check is a failed call. No model needs to weigh in. These are your release gates.

Layer 2: Jev on every call, for rubric dimensions. Score empathy, intent, confirmation, and resolution with typed questions. Store the score, distribution, and confidence. At cents per thousand calls, you can score all traffic, not a sample.

Layer 3: An LLM judge or human, only where needed. Route three kinds of cases here. First, Jev decisions below a confidence threshold. Second, calls where Jev and a check disagree. Third, calls a customer or reviewer disputes. This is the thin slice where a rationale earns its cost.

Here is the routing logic in code.

CONFIDENCE_FLOOR = 0.80  # tune on your own labeled calls

def score_call(call, confirmed_date, confirmed_time):
    report = {"call_id": call["call_id"]}

    # Layer 1: exact checks
    checks = verify_call(call, confirmed_date, confirmed_time)
    report["checks"] = checks
    if not all(checks.values()):
        report["verdict"] = "fail"
        return report  # no model needed

    # Layer 2: Jev on rubric dimensions
    jev = score_with_jev(call)  # Score + Noul + Choice questions
    report["jev"] = jev

    # Layer 3: escalate only uncertain or conflicting cases
    low_conf = [k for k, a in jev.items() if a["confidence"] < CONFIDENCE_FLOOR]
    conflict = jev["confirmed_before_booking"]["noul"] != checks["date_matches"]
    if low_conf or conflict or call.get("disputed"):
        report["escalated"] = judge_or_human(call, reasons=low_conf or ["conflict"])

    report["verdict"] = "pass"
    return report

A few rules keep the stack honest.

1. Write the ground-truth spec first. List every outcome with a knowable answer before you touch a model. This defines Layer 1.

2. Never let a model overrule a check. If the tool log says 10 a.m., no empathy score rescues the call.

3. Tune the confidence floor on labeled data. Pick the threshold where escalated cases actually contain most errors. Revisit it after model updates.

4. Keep the signals separate. Report check pass rates, Jev scores, and judge findings as distinct lines on your scorecard. A blended average hides which layer moved.

5. Feed escalations back. Human verdicts on Layer 3 cases become your next calibration set.

The order matters. Teams that start with a judge tend to grade everything with it. That includes facts a check would have nailed and volume Jev would have handled for cents. For a full walkthrough of Jev in this role, see Jev as a judge for voice agents.

Honest caveats

Every scorer here has limits. State them up front.

• Jev gives no explanations. It returns scores, choices, and probabilities. If a stakeholder asks "why," you need another layer.
• Jev reads text, not audio. A voice call must be transcribed first. Transcription errors flow straight into the score. Check your speech-to-text layer separately. Tone of voice and long pauses are not visible unless you encode them in the state.
• One study is not universal. LangChain's 500 decisions are strong evidence on one task. Validate on your own labeled calls.
• Highly consistent is not deterministic. Jev's variance was tiny, not zero. Treat it as a very stable estimate, not a proof.
• Calibration is statistical. A 0.9 confidence means roughly 90% right across many calls. It is not a guarantee on the next one.
• Jev is new. It entered early access on September 15, 2026. Expect the API and models to evolve.
• Checks can be wrong too. A check is only as good as its spec. If the spec misses a rule, the check silently passes.
• Judges still have a place. Nothing here says remove them. It says use them where their rationale is worth the cost.

Where independent evaluation fits

Every paradigm is easy to grade generously when you own the agent. You write the checks that pass. You tune the rubric that flatters. You pick the confidence floor that looks good. So the scoring method and who runs it are separate questions.

Evalgent is an independent, third-party evaluator. We design this stack against your real calls and systems of record. We write the Layer 1 checks from your spec. We define and validate Jev questions against human labels on your data. We set confidence floors from measured error rates. And we reserve judges and reviewers for the cases that need them.

Then we report each signal separately, so no soft number carries the verdict. The value of a reproducible score is that anyone can rerun it and get the same answer. That is the point of independent voice AI evaluation. The line between running scenarios and judging them is drawn in our testing vs evaluation guide.

Jev can't abstain: always include an escape option

Jev always returns one of the options you define. It has no built-in "I don't know." When a call transcript holds no evidence for any option, such as a two-word turn, a dropped call or a caller talking about something else, Jev still picks the least-wrong option, and that label looks just like a confident answer. TypeSafe's own Choice docs recommend adding an "other" or "none of the above" option when the list may not cover every input.

This also separates Jev from verifiable rewards. A verifier can return "no checkable claim here"; Jev can only do that if the option set includes it, so the escape option is part of the rubric design.

Four rules keep this from hurting you on live calls:

• Add an explicit escape option such as `not_enough_information`, `not_applicable` or `none_of_the_above` to every Choice question where evidence can be missing. Describe it as concretely as the real options, because Jev reads only the descriptions, not your option keys.
• Ask whether the evidence exists first. A Noul like "does the caller state a reason for calling?" before the Choice question separates "no evidence" from "evidence for option B".
• Check the distribution, not just the label. In one independent edge-case test, a forced choice with no valid option still returned a pick at 0.52 probability but only 0.04 confidence (jev-exploration). The numbers flagged the problem; the label did not. Set your human-review cutoff from your own labeled calls, because an independent study on data unlike Jev's training found its confidence can run high (jev-ood-calibration).
• Track the escape-option rate. If it climbs, your transcripts, STT or option set have drifted away from what callers actually say.

Frequently asked questions

What is the difference between verifiable rewards, LLM judges, and Jev?

Verifiable rewards check facts in code and return the same result every run. LLM judges grade subjective quality and explain themselves, but vary between runs. Jev, a System One model, returns typed, calibrated scores fast and cheaply, but gives no rationale. Checks anchor facts, Jev handles scalable rubric scoring, and judges cover cases that need reasoning.

Which is the best way to score a voice agent?

No single method wins. Layer them. Run verifiable checks first on anything with a knowable answer. Score rubric dimensions like empathy or intent with Jev on every call. Send only low-confidence, conflicting, or disputed cases to an LLM judge or a human. Each dimension goes to the scorer that handles it best.

How does Jev compare to an LLM judge for scoring?

In LangChain's study of 500 decisions, Jev matched human labels 100% of the time. GPT-5.6 Terra hit 99.8%, Luna 96.4%, and Claude Sonnet 4.6 80.0%. The judges' score variance ran 92 to 913 times higher. Jev cost $0.34 for the run versus $28.17 for Claude. The trade-off is that Jev writes no explanation.

Can Jev replace verifiable rewards?

No. A verifiable check confirms a fact against a system of record, such as the booking date in a tool log. Jev returns a calibrated probability, not a confirmed fact. Keep checks as release gates for anything checkable. Use Jev for judgment dimensions where a consistent score is what you need, and never let it overrule a failed check.

What are the ways to score a voice agent objectively?

The most objective signal is a verifiable check with a known correct answer, reproducible every run. Jev adds highly consistent, calibrated scores for judgment dimensions, with very low run-to-run variance. Its output is still a probability, not a fact. LLM judges are the least objective, since the same transcript can earn different scores.

Can these scoring methods be gamed?

LLM judges are the most gameable. Under Goodhart's law, an agent tuned against a judge learns its preferences, such as length or confident tone, without improving. Verifiable checks resist gaming because they read real outcomes like tool logs. Jev is harder to game than a judge, but any scored dimension can be optimized against, so audit with humans.

Does Jev work on voice agent audio?

No. Jev works on text or structured state, not raw audio. Transcribe the call first, then send the transcript and relevant state. Transcription errors flow into the score, so evaluate your speech-to-text layer separately. Audio cues like tone or long pauses only count if you encode them into the state you send.

How do I combine all three scoring paradigms?

Write a ground-truth spec and cover every checkable outcome with verifiable checks. Run Jev on every call for rubric dimensions and store its confidence. Escalate low-confidence, conflicting, or disputed cases to an LLM judge or human. Tune the confidence floor on labeled calls. Report the three signals as separate scorecard lines, never one blended average.

The bottom line

Verifiable checks own the facts, Jev scores every call on rubric dimensions with calibrated confidence, and LLM judges or humans handle only the uncertain slice. Layer them in that order, validate each on your own calls, and keep their signals separate.

Want scoring you can trust? Book a demo and we will design and validate this stack on your real calls.