Jev as a Judge: A Calibrated Alternative to LLM-as-a-Judge

Most teams grade voice agents with an LLM-as-a-judge. That judge is slow and pricey. Worse, it can be inconsistent. A judge that scores the same call differently on reruns is not measuring your agent. It is measuring itself.

Jev launched on September 15, 2026, in early access. TypeSafe AI calls it the first "System One Model." It is not a language model. You send it a state and typed questions. It returns decisions with calibrated confidence. It returns no text.

This post goes deep on jev as a judge. First, the evidence: LangChain's study, number by number, including what it does not prove. Then a hands-on build. You will write a rubric, turn it into questions, and read a response. Next you will run a validation harness in Python and set routing thresholds. Last, you will set up drift monitoring.

What is the LLM-as-a-judge pattern?

LLM-as-a-judge is a well-known evaluation technique. You give a large language model a rubric and a transcript. The model reads both. It returns a score or a verdict, often with a written explanation.

LLM-as-a-judge: an evaluation method where a language model scores another model's output against a rubric. The judge acts as a stand-in for a human grader at machine speed.

Teams use it because it scales past manual review. A human reviewer can only listen to so many calls a day. An LLM judge can grade thousands. That is the appeal.

The pattern has three known weaknesses for voice agents. First, the judge drifts. The same transcript can earn a 7 on one run and a 5 on the next. Second, the judge is costly per call. Grading every call gets expensive fast. Third, the judge can write a reason that sounds right but is wrong.

Our deeper look at LLM-as-judge limits covers why these failures matter for release gates. The short version: a judge you cannot reproduce is hard to defend. If a vendor disputes a score, "the model felt it was a 5" is not evidence.

The LangChain study: can Jev be a better agent evaluator?

This is the best public evidence on Jev as a judge. LangChain published it as "Can Jev Be a Better Agent Evaluator?" They ran it in LangSmith, their own tracing platform. We cite the study for its numbers. We are not endorsing any tool.

The design: why replaying fixed runs isolates judge noise

The team captured five runs of a weather agent. Then they replayed each captured run one hundred times. That gives 500 judging decisions. Each judge graded every replay.

The key choice was holding the setup fixed. A live weather agent searches the web. Results change by the minute. The agent's own behavior also varies run to run. If you judged live runs, you could not tell two noise sources apart. Did the score change because the agent changed? Or because the judge changed?

Replaying fixed runs removes the first source. The input is identical every time. So any change in score comes from the judge alone. That is exactly what you want to measure in a judge.

Judge noise: the variation in a judge's verdict when the input does not change. A perfect judge has zero judge noise. It gives the same verdict on the same call every time.

Result 1: agreement with human evaluation

Each judge answered a binary question: does this run pass? The team compared each verdict to a human evaluation of the same run.

Jev agreed with the human label on all 500 decisions. Terra came close at 99.8%. On 500 decisions, that works out to about one miss (illustrative arithmetic). Luna at 96.4% implies about 18 misses. Claude at 80.0% implies about 100 misses. Those counts are our arithmetic, not published figures.

Notice what the misses mean here. The input never changed. So a miss is not a hard case. It is the same case graded differently on some replays. That is pure judge noise, showing up as disagreement.

Result 2: quality-score variance

The judges also gave a quality score, not just pass or fail. The team measured variance per case across the hundred replays. Then they averaged it.

Jev had the lowest mean per-case variance: 0.0000149. That is close to zero. The LLM judges were much higher. Claude's variance was 92x Jev's. Luna's was 433x. Terra's was 913x.

One detail stands out. Terra nearly matched Jev on pass/fail. Yet its quality scores wandered the most. Agreement and variance measure different things. A judge can land on the right side of a pass line while its score drifts. If you trend quality scores week over week, that drift becomes fake movement in your charts.

Variance: a measure of how spread out repeated measurements are. For a judge, low variance on a fixed input means the score is reproducible.

Result 3: cost and speed

Jev averaged 0.44 seconds per call. It cost $0.00035 per call on average. The full run cost $0.34 for Jev and $28.17 for Claude. That is roughly 83x cheaper on this workload (illustrative arithmetic).

This matches TypeSafe's list pricing. Jev input costs $0.042 per million tokens. Output is free. Frontier LLMs run about $0.20 to $10 per million input tokens. Output typically costs about five times input. Jev returns in 70 to 500 milliseconds. TypeSafe cites 3 to 329 seconds for LLMs.

What the study proves, and what it doesn't

It is strong evidence on one narrow question, not a universal verdict. On this task, Jev's verdicts were highly consistent across replays. It agreed with humans on the binary criterion every time. And it did this at a small fraction of LLM cost and latency.

It does not prove four things.

• It covers one agent type. The study used a weather agent. Your voice agent handles billing disputes, bookings, or claims. Those rubrics are harder and more subjective.
• It used a binary pass criterion. Pass or fail is an easier target than a five-level empathy score. Agreement on a binary label says little about fine-grained scales.
• It replayed captured runs. Replay is the right design for measuring noise. But it does not test live, messy traffic with new failure modes.
• Your rubric will differ. The study's criteria are not yours. Agreement depends heavily on how well you write the questions.

Also note what "consistent" means. TypeSafe says Jev "returns similar answers for similar inputs." That is highly consistent. It is not a promise of strict determinism. Plan for tiny variation and measure it yourself.

So the takeaway is modest and useful. Jev is a credible judge candidate. It earns a spot in your validation set. It has not earned your trust yet. Only your own labeled calls can do that.

Why a judge specifically benefits from Jev (jev as a judge)

Three properties make a judge trustworthy: consistency, calibration, and coverage.

Consistency comes first. In the replay study, Jev's quality scores barely moved on fixed input. Consistency is what makes a verdict auditable.

Calibration comes second. Jev returns calibrated confidence with every decision. Higher confidence should mean higher accuracy, on average. So you learn which verdicts to trust. See calibration in statistics) for the underlying idea.

Coverage comes third. At $0.042 per million input tokens, you can stop sampling. You judge every call, not a thin slice. Our case for why is in score every call, not a sample.

There are three question types. A judge maps cleanly onto all three.

• Choice: pick one of up to 255 options. Use it for outcome labels, like "resolved," "escalated," or "abandoned."
• Score: ordered levels that return a score, a distribution, a legend, and confidence. Use it for quality on a rubric, like empathy or clarity.
• Noul: a yes/no answer with a probability. Use it for pass/fail gates, like "followed the disclosure policy."

You can ask many questions about one state in a single call. So one judge pass can score outcome, quality, and every policy check at once. Our Jev voice agent metrics catalog maps common rubrics to these types.

Jev was trained with RLCD, which targets calibrated decisions rather than good text. It complements language models rather than replacing them. The TypeSafe docs cover the full API surface.

Jev as a judge vs LLM-as-a-judge

The table below compares judges on the properties that matter for judging. The numbers come from the LangChain study unless noted. Blank cells mean the study did not report that figure.

The pattern is clear on this workload. Jev wins on consistency, cost, and speed. The LLM judges win on explanation. That trade drives the design below.

The honest trade-off: no rationale

Here is the caveat most vendor decks skip. Jev does not explain its reasoning. It gives you a score, a label, or a yes/no with confidence. An LLM judge can say "the agent interrupted the caller twice, so empathy is low." Jev cannot.

That matters in specific cases. A disputed call needs a rationale. A new failure mode needs a human explanation. A regulator asking "why did this fail" needs words, not a float.

So the right design is a split. Use Jev for scale and consistency. Judge every call. Then send a small slice to an LLM judge or to humans for a written rationale. The two are not rivals. Our Jev versus LLM decision guide works through the combined pipeline.

Two more honest limits. Jev works on transcript or text state, not raw audio. You transcribe first, then judge. And calibrated is not the same as correct on any single call. Calibration is a property across many calls. A confident wrong answer is still possible.

How to build a Jev judge for voice agents

This is the build path from rubric to production. Follow it in order. Each step de-risks the next.

Step 1: Write the rubric in plain language first

Do not start in code. Start with a one-page rubric your human reviewers would sign. Each line should name one observable behavior. Here is an example for a billing support agent.

Keep lines atomic. "Was the agent polite and accurate?" is two questions. Split it. A judge cannot score a compound question well, and neither can a human.

Step 2: Express the rubric as Score and Noul questions

Now translate each line into a question. The heart of a Score question is its level descriptions. Anchor each level to something a reviewer can see in the transcript.

Anchored level: a scale level described by observable evidence, such as "restated the problem in the caller's words," rather than by a vague adjective like "good."

Here is the bundle with the Python SDK. Install it first.

pip install typesafe-sdk
export TYPESAFE_API_KEY="your-key-here"

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()  # reads TYPESAFE_API_KEY

EMPATHY_LEVELS = [
    "Ignored or talked over the caller's stated problem",
    "Acknowledged the problem with a generic phrase only",
    "Restated the specific problem in the caller's own terms",
]

JUDGE_QUESTIONS = {
    "resolution": Choice(
        instructions="How did the call end for the caller",
        criteria={
            "resolved": "The caller's request was completed on this call",
            "escalated": "The call was handed to a human agent",
            "abandoned": "The caller hung up before resolution",
        },
    ),
    "task_accuracy": Score(
        instructions="Did the agent complete the caller's actual request correctly",
        criteria=[
            "Wrong task, or a false statement about the account",
            "Right task, but a key detail was wrong or missing",
            "Right task, minor detail missing that the caller had to repeat",
            "Right task, every detail confirmed back to the caller",
        ],
    ),
    "empathy": Score(
        instructions="Did the agent acknowledge the caller's problem",
        criteria=EMPATHY_LEVELS,
    ),
    "disclosure": Noul(
        instructions="The agent stated that the call is recorded before collecting details",
    ),
    "identity_check": Noul(
        instructions="The agent verified identity before reading any account details",
    ),
    "overall_pass": Noul(
        instructions="A senior reviewer would accept this call as meeting the standard",
    ),
}

def judge(transcript: str, metadata: str):
    state = f"{metadata}\n\nTranscript:\n{transcript}"
    return client.system_one(state=state, questions=JUDGE_QUESTIONS)

Every level describes evidence, not a feeling. And each level excludes the others. That separation is what lets the judge, and your humans, land on the same level.

If your harness already runs on LangChain, the `langchain-typesafe` package wraps the same call.

from langchain_typesafe import Noul, Score, TypeSafeClassifier

classifier = TypeSafeClassifier()

response = classifier.invoke({
    "state": state,
    "questions": {
        "empathy": Score(
            instructions="Did the agent acknowledge the caller's problem",
            criteria=EMPATHY_LEVELS,  # same anchored list as above
        ),
        "overall_pass": Noul(
            instructions="A senior reviewer would accept this call as meeting the standard",
        ),
    },
})

if not response.nouls["overall_pass"].noul:
    flag_for_review(call_id)

Noul answers come back under `response.nouls`. Check the TypeSafe docs for the exact accessor on Score answers in your version. Early-access APIs can shift.

Step 3: Read the response

The raw API is a single POST to `https://api.typesafe.ai/v1/systemone`. The body carries the state, `"model": "jev-latest"`, and the questions. The response returns one typed answer per question. Here is an illustrative response for one call. The values are made up for teaching.

{
  "answers": {
    "resolution": {
      "choice": "resolved",
      "confidence": 0.93,
      "probabilities": {"resolved": 0.93, "escalated": 0.05, "abandoned": 0.02}
    },
    "task_accuracy": {
      "score": 2.71,
      "confidence": 0.81,
      "legend": {
        "0": "Wrong task, or a false statement about the account",
        "1": "Right task, but a key detail was wrong or missing",
        "2": "Right task, minor detail missing that the caller had to repeat",
        "3": "Right task, every detail confirmed back to the caller"
      },
      "probabilities": {"0": 0.01, "1": 0.06, "2": 0.14, "3": 0.79}
    },
    "empathy": {
      "score": 1.12,
      "confidence": 0.58,
      "legend": {
        "0": "Ignored or talked over the caller's stated problem",
        "1": "Acknowledged the problem with a generic phrase only",
        "2": "Restated the specific problem in the caller's own terms"
      },
      "probabilities": {"0": 0.12, "1": 0.64, "2": 0.24}
    },
    "disclosure": {"noul": true, "confidence": 0.98, "probabilities": {"true": 0.98, "false": 0.02}},
    "identity_check": {"noul": true, "confidence": 0.95, "probabilities": {"true": 0.95, "false": 0.05}},
    "overall_pass": {"noul": true, "confidence": 0.86, "probabilities": {"true": 0.86, "false": 0.14}}
  },
  "usage": {"input_tokens": 1840, "output_tokens": 48}
}

Read it like a reviewer would. Disclosure and identity are near-certain. Task accuracy leans strongly to the top level. Empathy is the weak spot. Its confidence is 0.58, and its probability mass is split. That call is a candidate for human review on empathy alone.

The legend ties each numeric level back to your wording.

Step 4: Validate against human labels

This is the step teams skip. It is also the step that makes the judge defensible. You need a labeled set: real calls, graded by trained reviewers against the same rubric.

Aim for a few hundred calls. Include the hard ones: angry callers, transfers, partial resolutions. A set of easy calls will flatter any judge.

The harness below does three checks. It measures agreement and Cohen's kappa. It checks calibration by confidence bucket. And it reruns each call several times to measure variance.

import statistics
from collections import defaultdict

def cohens_kappa(human, judge):
    """Agreement corrected for chance, for two raters on the same items."""
    n = len(human)
    labels = set(human) | set(judge)
    observed = sum(h == j for h, j in zip(human, judge)) / n
    expected = sum(
        (human.count(lbl) / n) * (judge.count(lbl) / n) for lbl in labels
    )
    return (observed - expected) / (1 - expected) if expected < 1 else 1.0

def validate(labeled_calls, question="overall_pass", reruns=5):
    human, judged = [], []
    buckets = defaultdict(lambda: [0, 0])  # bucket -> [correct, total]
    variances = []

    for call in labeled_calls:
        runs = [judge(call["transcript"], call["metadata"]) for _ in range(reruns)]
        first = runs[0].answers[question]

        human.append(call["labels"][question])
        judged.append(first.noul)

        bucket = min(int(first.confidence * 10), 9) / 10  # 0.5, 0.6, ... 0.9
        buckets[bucket][0] += int(first.noul == call["labels"][question])
        buckets[bucket][1] += 1

        probs = [r.answers[question].probabilities["true"] for r in runs]
        variances.append(statistics.pvariance(probs))

    agreement = sum(h == j for h, j in zip(human, judged)) / len(human)
    print(f"agreement: {agreement:.1%}")
    print(f"cohen's kappa: {cohens_kappa(human, judged):.3f}")
    print(f"mean rerun variance: {statistics.mean(variances):.7f}")
    for b in sorted(buckets):
        correct, total = buckets[b]
        print(f"confidence {b:.1f}+: accuracy {correct / total:.1%} on {total} calls")

Why kappa and not just agreement? Raw agreement can mislead. Suppose 95% of your calls pass. A judge that always says "pass" scores 95% agreement. Kappa corrects for that. It asks how much better the judge is than chance. The idea comes from inter-rater reliability research on human graders.

Now read the calibration table. In a calibrated judge, accuracy rises with confidence. Calls at 0.9 or higher should be right more often than calls near 0.6. If the 0.9 bucket is no better than the 0.6 bucket, the confidence signal is not working on your data. Do not gate on it yet.

The rerun variance should sit near zero. The study found 0.0000149 on replayed weather runs. Your number will differ. What matters is that it stays small next to the gap between levels.

For Score questions, compare levels instead of booleans. Round the score to the nearest level, then compute kappa on levels. Weighted kappa is better still, since missing by one level is less bad than missing by three.

Also score your human reviewers against each other. If two humans agree only 75% of the time, a judge cannot fairly be held to 98%. Human-to-human agreement is your ceiling.

Step 5: Set thresholds and route low confidence

Now use the calibration table to pick a gate. Find the lowest confidence where accuracy meets your bar. Suppose your bar is 97% agreement with humans. If the 0.8 bucket hits 97% and the 0.7 bucket does not, your threshold is 0.8.

Then route every verdict by confidence.

AUTO_ACCEPT = 0.80   # fit from your calibration table, not guessed
RATIONALE_Q = {"empathy", "task_accuracy"}

def route(call_id, result):
    for name, answer in result.answers.items():
        if answer.confidence >= AUTO_ACCEPT:
            record_verdict(call_id, name, answer)
        elif name in RATIONALE_Q:
            queue_llm_rationale(call_id, name)   # needs an explained reason
        else:
            queue_human_review(call_id, name)    # policy checks go to people

Two queues serve two needs. Quality questions that need a written reason can go to an LLM judge. It writes the explanation a reviewer can read. Policy and compliance questions go straight to a person. A disputed disclosure call should never rest on another model's opinion.

Watch the queue size. If 40% of calls fall below the gate, the rubric is the problem, not the threshold. Go back to Step 2. Our guide to human-in-the-loop review covers how to staff and sample these queues.

Step 6: Monitor judge drift over time

A judge validated in September can drift by December. Your agent changes. Your callers change. The model behind `jev-latest` may update. Any of these can move agreement.

Build three habits.

• Keep a frozen golden set. Hold back 100 labeled calls that never change. Rescore them weekly. If agreement or kappa drops, investigate before trusting new scores.
• Label a fresh slice each month. Have reviewers grade a small random sample of recent calls. This catches new failure modes the golden set cannot.
• Track the low-confidence rate. A rising share of calls below the gate is an early warning. It often means callers are doing something your rubric never described.

Rerun the full Step 4 harness after any change to the agent, prompt, or rubric. For live alerting on production traffic, see our note on Jev voice agent call monitoring.

Rubric-writing pitfalls that break a judge

Most bad judges are bad rubrics. The model gets blamed for a question a human could not answer either. Watch for three pitfalls.

Overlapping levels. "Good" and "Very good" overlap. So do "Mostly resolved" and "Partly resolved." When two levels can both describe the same call, humans split between them. The judge splits too, and confidence drops. Fix it by giving each level a boundary. Ask what evidence moves a call from level 1 to level 2.

Vague anchors. "The agent was empathetic" is a feeling. "The agent restated the problem in the caller's words" is evidence. Vague anchors make graders use their own private definitions. Your kappa will sag, and not because of the judge.

Missing context in the state. The judge only sees the state you send. If the rubric asks "did the agent follow the refund policy," the policy must be in the state. So must key metadata: the caller's intent, account status, and any tool results. A judge cannot grade a policy it never read.

Here is a quick before-and-after on one question.

One more trap: audio. Jev reads text. If your rubric depends on tone, pauses, or talk-over, the transcript must capture them. Add markers like `[interrupts]` or `[4s silence]` during transcription. Otherwise score those dimensions separately.

A Jev judge still isn't ground truth

This is the core thesis. A Jev judge is a fast, consistent estimator. It is not ground truth. A judge you did not validate is just an opinion at scale.

Ground truth comes from humans, from real outcomes, or from a deterministic check on a known fact. The judge approximates that. Calibration tells you how good the approximation is. It does not make the judge correct by itself.

The LangChain study is a good reason to test Jev. It is not a reason to skip testing. Their 100% came from a weather agent with a binary criterion. Your billing agent with a four-level accuracy scale is a different exam.

So two rules hold. Validate the judge against human labels before you trust it. Route every low-confidence verdict to a person. Skip either step and you have automated a guess. Our broader voice agent evaluation guide and the testing versus evaluation primer put the judge inside the full measurement stack.

Where Evalgent fits

Evalgent is the independent party that builds and validates the judge on your real calls. We do not sell voice agents. We are the neutral evaluator. That separation is the point. A vendor grading its own agent has an incentive problem. An outside auditor does not.

We turn your rubric into anchored Jev questions. We build the human-labeled set from your calls, not a demo. We run the harness above and report agreement, kappa, calibration, and variance plainly, weak questions included. Then we set the gate, staff the low-confidence review, and rescore a golden set as your agent changes. Our note on independent voice AI evaluation explains why third-party status changes what a score is worth. For a wider view of the model, see Jev for voice agent evaluation. If you want a Jev-based judge you can defend to a buyer or a regulator, book a demo.

Jev can't abstain: always include an escape option

Jev always returns one of the options you define. It has no built-in "I don't know." When a call transcript holds no evidence for any option, such as a two-word turn, a dropped call or a caller talking about something else, Jev still picks the least-wrong option, and that label looks just like a confident answer. TypeSafe's own Choice docs recommend adding an "other" or "none of the above" option when the list may not cover every input.

As a judge, a forced pick quietly inflates or deflates scores. A rubric that asks whether the agent confirmed the appointment, on a call where no appointment came up, needs a `not_applicable` level, or those calls get scored as passes or failures they never earned.

Four rules keep this from hurting you on live calls:

• Add an explicit escape option such as `not_enough_information`, `not_applicable` or `none_of_the_above` to every Choice question where evidence can be missing. Describe it as concretely as the real options, because Jev reads only the descriptions, not your option keys.
• Ask whether the evidence exists first. A Noul like "does the caller state a reason for calling?" before the Choice question separates "no evidence" from "evidence for option B".
• Check the distribution, not just the label. In one independent edge-case test, a forced choice with no valid option still returned a pick at 0.52 probability but only 0.04 confidence (jev-exploration). The numbers flagged the problem; the label did not. Set your human-review cutoff from your own labeled calls, because an independent study on data unlike Jev's training found its confidence can run high (jev-ood-calibration).
• Track the escape-option rate. If it climbs, your transcripts, STT or option set have drifted away from what callers actually say.

Frequently asked questions

What is jev as a judge for voice agents?

Jev as a judge means using TypeSafe's System One model to grade voice-agent calls. You send a transcript state and rubric questions. It returns typed verdicts with calibrated confidence in 70 to 500 milliseconds. In LangChain's study it matched human labels on all 500 decisions. It returns a number or label, not a written explanation.

Is Jev a good LLM-as-a-judge alternative?

For scale and consistency, the evidence is promising. In LangChain's replay study, Jev's score variance was 92 to 913 times lower than three LLM judges. It cost $0.34 total versus $28.17 for Claude. It trades away written rationale. That was one study on one agent type, so validate it on your own labeled calls first.

Does a Jev judge explain its reasoning?

No. A Jev judge does not write prose or explanations. It returns a score, a label, or a yes/no answer, each with calibrated confidence. An LLM judge can describe why it scored a call low. Jev cannot. Route disputed calls and new failure modes to an LLM judge or to humans for a written rationale.

Can Jev replace an LLM judge entirely?

Not for every case. Jev can replace the LLM judge for consistent, cheap, full-population scoring. It cannot replace it where you need a written rationale. The practical setup is a split. Jev judges every call. An LLM judge or humans handle the low-confidence cases and the ones that need an explanation.

How do I validate a voice agent judge?

Build a set of a few hundred real calls labeled by trained reviewers. Score them with the judge. Compute agreement and Cohen's kappa. Check that accuracy rises with confidence, bucket by bucket. Rerun each call several times to measure variance. Compare against human-to-human agreement. Repeat after any change to the agent, prompt, or rubric.

Why does consistency matter for a judge?

A judge must give the same verdict on the same call, or scores cannot be trended or defended. Strictly, Jev is highly consistent rather than deterministic. TypeSafe says it returns similar answers for similar inputs. In LangChain's replay study, its mean per-case variance was 0.0000149. Measure rerun variance on your own calls anyway.

How does calibrated confidence help a Jev judge?

Calibrated confidence attaches a trust level to each verdict. Across many calls, higher confidence should track higher accuracy. Check that on your labeled set first. Then gate on it. Verdicts above your threshold pass through automatically. Verdicts below it route to a person or an LLM judge. Only the uncertain calls cost reviewer time.

Does a Jev judge work on raw call audio?

No. Jev judges transcript or structured text state, not raw audio. You transcribe the call first, then send the transcript and questions to Jev. Tone, pauses, and talk-over must be marked in the transcript or scored separately. Plan for this limit when you design the rubric and the state.

The bottom line

Jev as a judge delivered perfect agreement, near-zero score variance, and a tiny fraction of LLM cost in LangChain's study, but that study covered one agent and one binary criterion. A Jev judge earns trust only after it is validated on your own labeled calls, gated on calibrated confidence, and monitored for drift.

Book a demo to have Evalgent validate your judge on your calls.