Verifiable Rewards vs LLM Judges vs Jev: 3 Ways to Score

# Verifiable rewards vs LLM judges vs Jev: 3 ways to score

Quick answer: There are three ways to score a voice agent. Verifiable rewards check facts in code — objective, reproducible, but narrow. LLM judges grade nuance and explain themselves, but drift and can be gamed. Jev, a System One model, returns fast, cheap, calibrated decisions — a middle path, though it only classifies and scores.

Most teams score voice agents two ways and argue about which is better. One camp writes code to check facts. Did the booking post? Was the refund amount right? The other camp asks a language model to grade the fuzzy stuff. Was the agent warm? Was the answer helpful?

Both camps are half right. A new option changed the shape of the argument. In September 2026, TypeSafe AI shipped Jev, its first System One model, which returns typed, calibrated decisions without generating text. That gives you a third way to score, sitting between rigid rules and a fuzzy judge.

This post lays out all three paradigms. It shows what each measures well, where each breaks, and how to combine them so the number you ship actually means something. It builds on the two-way split covered in verifiable rewards vs LLM judges and adds the paradigm that post predates.

The three ways to score a voice agent

Every scoring method answers one question: how do you turn a call into a number you trust? The three paradigms answer it differently.

> Verifiable reward: a deterministic, programmatic check with a right answer known in advance. Code confirms an objective fact about the call, and the check returns the same result every run.

> LLM-as-judge: a language model prompted to grade a transcript on a quality dimension such as tone or helpfulness. The score is an estimate, and it can change between runs.

> System One model (Jev): a model that takes state plus a typed question and returns a probabilistic decision with calibrated confidence. It classifies and scores, but it does not generate text.

These are not competitors fighting for the same job. They cover different parts of the same call. The rest of this post is about matching each paradigm to the work it does best, then wiring them together. This is the scoring layer under any serious voice agent evaluation program.

Verifiable rewards: objective but narrow

A verifiable reward is a check with a known correct answer. You state what must be true, then compute it against a system of record.

# Verifiable checks after a booking call
assert crm.get_booking(call_id) is not None          # booking exists
assert crm.booking.date == "2026-04-12"              # correct date
assert refund.amount == 42.50                         # exact amount
assert "recorded for quality" in transcript.lower()  # disclosure spoken

Each line has a ground truth known before the call. The check passes or fails. There is no opinion in the loop, so anyone can rerun it and get the same verdict. That property, reproducibility, is what makes a verifiable reward safe to gate a release on.

Verifiable rewards are cheap, fast, and hard to game. An agent cannot talk its way past a check that reads the CRM. They suit anything with a factual answer: task completion, data accuracy, required disclosures, forbidden phrases, correct routing.

The limit is coverage. A verifiable reward cannot grade tone, empathy, or whether an explanation made sense. Those dimensions resist reduction to a rule. Force a check onto them and you get a brittle proxy that misses what actually matters. So verifiable rewards anchor the checkable, and nothing more.

LLM-as-judge: flexible but wobbly

An LLM-as-judge grades an agent against a rubric. You hand it a transcript, a scale, and a definition of quality. It returns a rating and usually a rationale.

This is the only paradigm here that explains itself in prose. Ask why a call scored low and the judge writes a sentence. That is genuinely useful for the subjective dimensions no assertion can touch — was the agent patient with a confused caller, did the summary stay faithful to the call.

The problems are well documented, and we cover them in depth in the limits of LLM-as-judge for voice AI. A judge is non-deterministic, so the same transcript can earn different scores across runs. It is sensitive to prompt wording and prone to position and verbosity bias. It is also slow and expensive at scale, because every call means another model inference with a long prompt.

Worst of all, a judge is gameable. Goodhart's law says a measure that becomes a target stops being a good measure. An agent tuned against a judge learns what the judge likes. It grows wordier or more confident because those traits raise the score, while real quality stays flat. Judges earn their place only on genuinely subjective questions, and only after you validate them against human graders using inter-rater reliability.

Jev and System One models: calibrated decisions

Jev is a different kind of model. It is not an LLM, and it does not generate text. You give it state plus a typed question, and it returns a probabilistic decision with calibrated confidence). TypeSafe AI calls this a System One model, and the mechanics are in the TypeSafe documentation.

It answers three question shapes. Choice picks one option from up to 255, with a probability per option and a confidence. Score returns a continuous value on ordered levels, plus a distribution and confidence. Noul returns a yes or no probability. Every answer fits a schema, so Jev cannot hallucinate and posts 0% type errors — it structurally cannot return something the schema does not allow.

The performance numbers reframe the cost of scoring. Jev runs in 70 to 500 milliseconds, roughly 40 to 200 times faster than a frontier LLM judge. Input costs $0.042 per million tokens, and classification output is free — about 400 times cheaper than an LLM on the same work. Its confidence is calibrated, meaning higher confidence tracks higher accuracy, and it was trained with a method TypeSafe calls RLCD.

The most striking evidence comes from outside TypeSafe. LangChain tested Jev as an evaluator and built a harness around it. In their testing, Jev's quality-score variance ran 92 to 913 times lower than GPT-5.6 Luna, Terra, and Claude Sonnet 4.6. In plain terms, it gave far more repeatable scores. LangChain measured 0.44 seconds and $0.00035 per call on average, for $0.34 total against $28.17 for Claude on the same run.

Repeatability is exactly the property an LLM judge lacks. That makes Jev attractive for scoring at scale. But be honest about the limits. Jev only classifies and scores through Choice, Score, and Noul. It cannot do open-ended reasoning or write an explanation for its verdict. It works on text or structured state, not raw audio. Calibrated confidence is a statistical property across many calls, not a guarantee on any single call. And it launched in early access on September 15, 2026, so it is new.

The three paradigms compared

Here is how the three score against the axes that decide which to reach for.

The pattern is clear. Verifiable rewards win on trust but lose on coverage. LLM judges win on flexibility and explanation but lose on cost and consistency. Jev sits between them — cheaper and more consistent than a judge, broader than a rigid check, but silent on the "why" and limited to typed answers.

How to choose and combine the three

The goal is not to pick a winner. It is to route each part of a call to the paradigm that scores it best. Here is the order that works.

1. Write the ground-truth spec first. Before any model, list every call outcome with a knowable correct answer — bookings, amounts, fields collected, disclosures, routing. This spec defines what verifiable rewards can cover.

2. Cover every checkable outcome with verifiable rewards. If code and a fair human would reach the same verdict, write a check. These are your release gates, because they are reproducible and hard to game.

3. Route scalable scoring to Jev. For high-volume dimensions that need a repeatable score — intent classification, sentiment level, policy adherence as a yes/no — use Jev's Choice, Score, or Noul. You get calibrated, consistent numbers at a fraction of judge cost.

4. Reserve an LLM judge for the genuinely nuanced. Keep the judge for the thin slice that needs open-ended reasoning or an explanation, like whether a complex de-escalation was handled well. Validate it against human graders first.

5. Use confidence to triage. Send Jev's low-confidence decisions to a judge or a human, and let its high-confidence ones stand. This spends the expensive paths only where the cheap path is unsure.

6. Keep the three signals separate. Report verifiable pass rates, Jev scores, and judge scores as distinct lines on your scorecard. A blended average hides which layer moved.

The order matters. Teams that start with a judge grade everything with it, including facts a check would have nailed and volume a calibrated model would have handled for cents.

Where independent evaluation fits

All three paradigms are easy to grade generously when you own the agent. You write the checks that pass, the rubrics that flatter, and pick the confidence thresholds that look good. That is why the scoring method and who runs it are separate questions.

Evalgent is an independent, third-party evaluator. We build verifiable rewards against your real systems of record, apply calibrated models like Jev for scalable scoring, and reserve validated judges for true nuance — then report each signal separately so no single soft number carries the verdict. The whole value of a reproducible score is that anyone can rerun it and get the same answer. That is the point of independent voice AI evaluation, and it is why a self-reported quality number deserves less trust than a check you can reproduce. The line between running these scenarios and judging their results is drawn in full in the testing vs evaluation guide.

Frequently asked questions

What is the difference between verifiable rewards, LLM judges, and Jev?

Verifiable rewards check facts in code and return the same result every run. LLM judges estimate subjective quality and can explain themselves, but drift between runs. Jev, a System One model, returns typed calibrated decisions fast and cheaply, but only classifies and scores. Verifiable rewards anchor facts, Jev handles scalable scoring, and judges cover true nuance.

Which is the best way to score a voice agent?

No single method wins. The best approach combines all three. Use verifiable rewards for anything with a knowable correct answer, Jev for high-volume calibrated scoring like intent or sentiment, and an LLM judge for the thin slice needing open-ended reasoning. Matching each dimension to the right paradigm beats forcing one method onto the whole call.

How does Jev compare to an LLM judge for scoring?

In LangChain's testing, Jev's quality-score variance ran 92 to 913 times lower than GPT-5.6 and Claude Sonnet 4.6, at 0.44 seconds and $0.00035 per call. That means far more repeatable scores, much faster and cheaper. The trade-off is that Jev only returns typed decisions and cannot write an explanation, which a judge can.

Can Jev replace verifiable rewards?

No. Verifiable rewards check an objective fact against a system of record, like whether a booking was created. Jev returns a probabilistic decision from state and a question, not a confirmed fact. Keep verifiable rewards as the release gate for checkable outcomes. Use Jev for scoring dimensions where a calibrated, consistent estimate is what you need.

What are the ways to score a voice agent objectively?

The most objective signal is a verifiable reward — a code check with a known correct answer, reproducible on every run. Jev adds calibrated, consistent decisions that reduce the randomness of an LLM judge, though its output is probabilistic, not a confirmed fact. LLM judges are the least objective, since scores can shift between runs on the same transcript.

Can these scoring methods be gamed?

LLM judges are the most gameable. Goodhart's law applies: an agent learns the judge's preferences and games the score without improving. Verifiable rewards resist gaming because they check real outcomes an agent cannot fake with phrasing. Jev is harder to game than a judge because its decisions are calibrated and consistent, but any scored dimension can be optimized against.

Does Jev work on voice agent audio?

No. Jev works on text or structured state, not raw audio. For a voice agent, you transcribe the call first, then feed the transcript and relevant state to Jev. This means transcription quality affects the input, so pair Jev scoring with checks on your speech-to-text layer rather than assuming the transcript is perfect.

How do I combine all three scoring paradigms?

Write a ground-truth spec, cover every checkable outcome with verifiable rewards, and gate releases on those. Route high-volume calibrated scoring to Jev, and reserve a validated LLM judge for genuinely nuanced dimensions. Use Jev's confidence to triage uncertain cases to a judge or human. Keep the three signals on separate scorecard lines instead of averaging them into one.

The bottom line

Verifiable rewards anchor facts, Jev delivers scalable calibrated scoring, and LLM judges cover the genuinely nuanced. Route each part of a call to the paradigm that scores it best, and keep the three signals separate.

Want scoring you can trust? Book a demo and we will build verifiable rewards against your real outcomes, apply calibrated models like Jev, and reserve judges for the nuance that needs them.