Score Every Call, Not a Sample: Voice-Eval Economics

Voice teams have always sampled. You pull a few hundred calls, grade them, and hope they represent the rest. The reason was never a lack of interest. It was cost.

Grading is expensive. A human reviewer takes minutes per call. An LLM judge takes seconds and cents per call. Multiply either by a million monthly calls, and full review is off the table. So teams sample and live with the blind spots.

That constraint is moving. A cheap, fast, calibrated decision model changes voice agent evaluation economics. This post does three things. It works the cost math line by line. It shows exactly which insights full coverage unlocks, with numbers. And it names the honest limits.

Why voice agent evaluation has always meant sampling

Sampling) is a workaround, not a goal. You sample because you cannot afford to measure everyone. A census gives better data. It is just usually too expensive.

Sampling: measuring a subset of a population to estimate a property of the whole. A census measures every unit instead, trading cost for completeness.

Voice evaluation inherited that constraint. Grading a call means judging intent, resolution, policy, and tone. For years the only graders were humans and, later, LLM judges. Both are slow. Both cost real money at scale.

Human review is the gold standard for nuance. It is also the slowest option. A QA analyst might grade 30 to 50 calls a day. At a million calls a month, full human review is impossible.

LLM judges made scoring faster. They did not make it free. An LLM judge reads the transcript, reasons, and writes a verdict. That costs seconds and a fraction of a cent to a few cents each time. Its output also varies run to run and needs parsing. We cover those tradeoffs in testing vs evaluation for voice agents and in the limits of LLM-as-judge.

So teams sample. Our guide to sampling live calls for voice agents covers how to sample well when you must. This post argues you now have to far less often.

Meet the scorer that changes the math

Jev is TypeSafe AI's first "System One Model," launched September 15, 2026, in early access. It is not an LLM and generates no text. You send it a state, such as a transcript, plus questions. It returns typed, probabilistic decisions with calibrated confidence. TypeSafe describes it in its System One launch post.

System One model: a model that takes unstructured state plus questions and returns typed, probabilistic decisions with calibrated confidence. It classifies, scores, and verifies rather than writing free-form text.

Jev answers three shapes of question:

• Choice. Pick one option from up to 255, with a probability per option and a confidence. Good for intent and outcome labels.
• Score. Rate a state against ordered levels. Returns a score, a distribution, and confidence. Good for sentiment, risk, and quality grades.
• Noul. A yes or no question, returned as the probability the statement is true. Good for policy and compliance checks.

Many questions run against one state in a single call, in parallel. Extra questions cost only their own input tokens. That detail is the hinge of this whole post. It makes a 20-question rubric barely more expensive than a 3-question one.

Jev was trained with RLCD and uses a parallel sampler. TypeSafe reports 70 to 500ms per call, versus 3 to 329 seconds for frontier LLMs. That is roughly 40 to 200 times faster. More on the model in Jev for voice agents.

The worked cost math

Every evaluation choice is a cost-benefit analysis. The benefit is coverage and confidence. The cost is time and money per graded call. Sampling only won because the per-call cost was high. So let us price a call precisely.

All figures in this section are illustrative. Your transcript lengths, rubric, and vendor prices will differ.

Jev: one call, ten rubric questions

Start with a typical support call. Assume a 2,000-token transcript. Add 10 rubric questions at about 30 tokens each. That is roughly 300 tokens of questions, or about 2,300 input tokens total.

• Input cost: 2,300 tokens x $0.042 per million = $0.0000966 per call.
• Output cost: $0. Jev output is free, which TypeSafe calls "too cheap to meter."
• Total: under one hundredth of a cent per call.

Scale it up. 1,000 calls cost about $0.10. 100,000 calls cost about $9.66. One million calls cost about $96.60.

An LLM judge doing the same work

Now price a mid-range frontier LLM judge. Frontier input prices run from about $0.20 to $10 per million tokens. Output typically costs around 5x input. Assume $3 per million input and $15 per million output.

• Input: 2,300 tokens x $3 per million = $0.0069.
• Output: assume 300 tokens of reasoning and verdicts x $15 per million = $0.0045.
• Total: about $0.0114 per call, or roughly 118x the Jev figure.

One million calls cost about $11,400. That is the number sampling exists to avoid. Reasoning-heavy judges that write longer rationales cost more.

What a real measured run showed

Illustrative math is useful. Measured numbers are better. LangChain tested Jev as an agent evaluator in its LangSmith study. It ran 500 decisions: 5 runs across 100 replays.

Jev averaged about 0.44 seconds and $0.00035 per call. The full run cost $0.34 for Jev versus $28.17 for Claude. That is about 83x cheaper in that run. The per-call figure is higher than our $0.0000966 because those states were larger. Even at $0.00035, one million calls cost about $350.

TypeSafe's own workflow evals report 193.6x faster and 444.6x cheaper. TypeSafe says those gains are likely the high end of real-world results. Treat the LangChain run as the more conservative anchor.

Cost at scale

The takeaway is not a precise ratio. It is a change in kind. Scoring a million calls moves from a budget line to a rounding error. At that price the question flips. You stop asking "which calls can we afford to grade?" You start asking "what else can we learn from every call?"

There is a fitting irony in the name. Jev honors economist William Stanley Jevons. The Jevons paradox says cheaper resources get used more, not less. Make scoring cheap, and teams score everything, in more ways.

What the cost difference unlocks

This is the heart of the argument. A census is not just "more data." It enables specific analyses that a sample cannot support at any sample quality. Each one below comes with the math or an example.

1. True population rates instead of estimates

Take 100,000 calls a month and a 2% sample. That is 2,000 graded calls. Say a failure mode, such as a wrong refund amount, truly occurs in 0.3% of calls.

In your sample you expect about 6 hits. Six is too few to trust. A 95% confidence interval around 6 of 2,000 runs from about 0.14% to 0.65%. That is a nearly fivefold range. You cannot tell a minor issue from a serious one.

Worse, the count itself is noisy. With 6 expected hits, about one sample in seven shows 3 or fewer. Your dashboard could report half the real rate by pure chance.

Now score all 100,000 calls. You see about 300 real failures. The rate is measured, not estimated. Even treated as a draw from the underlying process, the interval tightens to about 0.27% to 0.34%.

2. Rare and long-tail failure discovery

Voice failures follow a long tail. A few issues are common. Hundreds are rare, and many rare ones are severe. Think of a disclosure skipped only on one flow, or a loop only on one accent.

A sample finds the head of the distribution. It rarely finds the tail. A failure at 0.05% of 100,000 calls yields about 1 expected hit in a 2% sample. Often you see zero. A census finds all 50 incidents and lets you cluster them.

Rare-failure discovery also needs open questions. Add a Noul like "Did anything unusual or unexpected happen on this call?" Its high-probability hits become a daily triage queue.

3. Segment-level slicing

Averages hide problems. Real issues live in segments: one intent, one locale, one carrier, one hour. A sample cannot slice deeply because each slice gets tiny.

Say Spanish-language calls are 5% of traffic. A 2% sample of 100,000 calls holds only 100 of them. A 0.3% failure yields 0.3 expected hits. You learn nothing. A census gives you 5,000 Spanish calls and about 15 real failures.

Slice by any axis you log: intent, locale or accent, time of day, campaign, agent version, and carrier or channel. Cross two axes, such as "billing intent on the new agent version after 6pm." Only a census has enough calls in each cell.

4. Per-turn scoring instead of per-call

Per-call scores tell you a call went wrong. Per-turn scores tell you where. A 12-turn call scored per turn means 12 Jev calls, each with a small state.

At census prices, that is affordable. You can see that containment breaks at turn 4 on address changes. You can see interruptions cluster right after the agent reads a policy. That is the level of detail teams need to fix prompts and flows. See Jev voice agent metrics for turn-level metric ideas.

5. More rubric dimensions per call

With LLM judges, teams trim rubrics to three or four questions to save cost. With Jev, each extra question adds only its own input tokens, roughly 30 each.

Going from 3 to 20 questions adds about 510 tokens. At $0.042 per million, that is about $0.00002 more per call, illustratively. You can check greeting, identity verification, disclosure, empathy, accuracy, escalation, and resolution in one pass.

6. Full-power A/B tests and release comparisons

Comparing agent versions needs statistical power. Say you want to detect a failure rate moving from 2.0% to 2.5%. At 95% confidence and 80% power, you need about 13,800 calls per arm.

A 2% sample of 100,000 weekly calls, split two ways, gives about 1,000 per arm. That test is badly underpowered. It will miss real regressions. A census gives about 50,000 per arm, so the test resolves within days. Pair this with model routing experiments to compare configurations fairly.

7. Drift detected in hours, not weeks

At 100,000 calls a day, you score about 4,000 calls an hour. That is more than a month of a typical sampled review, every hour.

A prompt change, a new ASR vendor, or a carrier outage shows up in the next hour's rates. With sampling, the same shift hides in noise for weeks. Our post on voice agent metric drift covers thresholds.

8. A 100% compliance audit trail

Regulated flows need proof, not estimates. "We checked 2% of calls" is a weak answer to an auditor. "Every call was scored for the required disclosure" is a strong one.

A census gives each call a timestamped, typed record. Each record holds the question, the probability, and the confidence. Low-confidence calls route to humans. The rest form a complete trail.

9. Customer and cohort quality histories

When every call is scored, each customer gets a quality history. You can see that a high-value account hit three failed transfers this month. You can see new customers get worse resolution than tenured ones. Samples almost never capture the same customer twice.

10. Labeled datasets for fine-tuning and regression suites

Every scored call becomes a labeled example. Filter for high-confidence failures and you have a regression suite built from real traffic. Filter for high-confidence successes and you have training data. The low-confidence middle becomes your human-labeling queue.

11. Backfilling history when the rubric changes

Rubrics evolve. You add a question about a new policy. With LLM judges, re-scoring last quarter is a costly project. At about $0.0001 per call, re-scoring 3 million past calls costs roughly $290, illustratively. You get a consistent baseline for the new metric overnight.

12. Real-time dashboards and alerting

Jev answers in 70 to 500ms. Scores can land seconds after a call ends. Dashboards update continuously, and alerts fire on live rates instead of weekly reports. For in-call patterns, see Jev voice agent call monitoring.

13. Linking quality to business outcomes

The biggest unlock is joining quality to outcomes. Join every call's scores to conversion, churn, refunds, and CSAT. Now you can ask which rubric dimensions actually predict churn.

A sample rarely overlaps enough with outcome data to answer that. A census joins every call. You may find empathy scores barely matter while first-call resolution drives renewal. That changes what you optimize.

Insight map

What the API call looks like

Here is the shape of one scoring request, following TypeSafe's quickstart. It uses the documented schema: `instructions` for every question, plus `criteria` as an object of options for a Choice and an ordered list of levels for a Score.

{
  "state": "AGENT: Thanks for calling Acme. How can I help?\nCALLER: I was charged twice for my order...\nAGENT: I see a duplicate charge of $42. I have refunded it.",
  "model": "jev-latest",
  "questions": {
    "intent": {"type": "choice", "instructions": "What is the caller's primary intent?", "criteria": {"billing": "Charges, refunds, invoices", "order_status": "Where an order is", "cancellation": "Wants to cancel", "technical": "Something is broken", "other": "Anything else"}},
    "resolved": {"type": "noul", "instructions": "Was the caller's issue fully resolved on this call?"},
    "disclosure_given": {"type": "noul", "instructions": "Did the agent state the required refund timeline?"},
    "empathy": {"type": "score", "instructions": "How empathetic was the agent?", "criteria": ["Poor", "Fair", "Good", "Excellent"]}
  }
}

The response returns one typed answer per question, each with confidence and probabilities. Values below are illustrative.

{
  "answers": {
    "intent": {"type": "choice", "choice": "billing", "confidence": 0.97, "probabilities": {"billing": 0.97, "order_status": 0.02, "cancellation": 0.0, "technical": 0.0, "other": 0.01}},
    "resolved": {"type": "noul", "noul": 0.93},
    "disclosure_given": {"type": "noul", "noul": 0.08},
    "empathy": {"type": "score", "score": 1.94, "confidence": 0.84, "legend": {"0": "Poor", "1": "Fair", "2": "Good", "3": "Excellent"}, "probabilities": {"0": 0.03, "1": 0.18, "2": 0.61, "3": 0.18}}
  },
  "usage": {"input_tokens": 412, "output_tokens": 58}
}

Notice `disclosure_given` at 0.08. That call resolved the issue but likely skipped a required disclosure. A sample would catch that on 2% of calls. A census catches it on all of them.

Batch-scoring a day of calls by segment

Below is a sketch of a daily census job. It scores every call with a 10-question rubric. Then it aggregates failure rates per segment. Constructor arguments are simplified. Match them to the SDK docs.

pip install typesafe-sdk

from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()  # reads your API key from the environment

RUBRIC = {
    "intent": Choice(instructions="Primary caller intent?",
                     criteria={"billing": "Charges, refunds, invoices", "order_status": "Where an order is", "cancellation": "Wants to cancel", "technical": "Something is broken", "other": "Anything else"}),
    "resolved": Noul(instructions="Was the issue fully resolved on this call?"),
    "verified_identity": Noul(instructions="Did the agent verify identity before sharing account data?"),
    "disclosure_given": Noul(instructions="Did the agent state the required refund timeline?"),
    "wrong_info": Noul(instructions="Did the agent give any factually wrong information?"),
    "escalation_needed": Noul(instructions="Should this call have been escalated to a human?"),
    "caller_repeated": Noul(instructions="Did the caller have to repeat themselves more than once?"),
    "unusual_event": Noul(instructions="Did anything unusual or unexpected happen on this call?"),
    "empathy": Score(instructions="How empathetic was the agent?",
                     criteria=["Poor", "Fair", "Good", "Excellent"]),
    "sentiment_end": Score(instructions="Caller sentiment at the end of the call?",
                           criteria=["Negative", "Neutral", "Positive"]),
}

FAIL_IF_TRUE = {"wrong_info", "escalation_needed", "caller_repeated", "unusual_event"}
FAIL_IF_FALSE = {"resolved", "verified_identity", "disclosure_given"}
REVIEW_BELOW = 0.70  # route low-confidence answers to humans


def score_call(call):
    result = client.system_one(state=call["transcript"], questions=RUBRIC)
    return call, result.answers


def failures(answers):
    out = []
    for name in FAIL_IF_TRUE | FAIL_IF_FALSE:
        p = answers[name].noul
        failed = p >= 0.5 if name in FAIL_IF_TRUE else p < 0.5
        if failed:
            out.append(name)
    return out


def daily_census(calls, segment_keys=("locale", "agent_version", "channel")):
    totals = defaultdict(int)
    fails = defaultdict(int)
    review_queue = []

    with ThreadPoolExecutor(max_workers=32) as pool:
        for call, answers in pool.map(score_call, calls):
            intent = answers["intent"].choice
            segments = [("intent", intent)] + [(k, call[k]) for k in segment_keys]
            failed = failures(answers)

            for seg in segments:
                totals[seg] += 1
                for f in failed:
                    fails[(seg, f)] += 1

            if any(a.confidence < REVIEW_BELOW for a in answers.values()):
                review_queue.append(call["id"])

    report = {}
    for (seg, f), n in fails.items():
        report[(seg, f)] = round(100 * n / totals[seg], 2)  # percent of calls
    return report, review_queue

Three design choices matter here. All 10 questions ride one call, so the rubric costs little more than a single question. Every segment gets real counts, not extrapolations. And low-confidence answers go to a human queue instead of silently counting.

Sampling vs scoring every call

The table compares a sampled subset against a census on the axes that matter. The census column assumes a cheap, calibrated scorer like Jev.

Selection bias deserves a note. Samples often skew by accident. Reviewers pull calls from business hours, or from escalated tickets. A census has no such skew. For why real traffic beats synthetic tests, see benchmark voice agents on your own data.

The honest limits of scoring every call

A cheaper scorer is not a free lunch. Full-population scoring has real boundaries.

Scoring everything does not make the scorer right. A cheap scorer that is wrong on every call is worse than a good sample. It produces confident, precise, wrong numbers. Calibrated does not mean correct on your data until you validate it.

Validate agreement and calibration first. Label a few hundred representative calls with humans. Measure agreement per question. Check calibration: calls scored at 0.9 confidence should be right about 90% of the time. LangChain's study found Jev matched human labels 500 of 500 times, with 92x to 913x lower variance than the LLMs tested. That is strong evidence on their task. It is not proof on yours.

Transcript-level only. Jev reads text and structured state, not raw audio. It sees what was said, not how it sounded. Prosody, crosstalk, and tone from the waveform still need audio-aware methods. Transcription errors also flow straight into scores.

Decides and scores, nothing more. Jev answers Choice, Score, and Noul questions well. It does not write narrative critiques or suggest fixes. Match the question to the tool.

Highly consistent, not deterministic. Jev's outputs are far more stable than LLM judges. Still, treat them as probabilistic and set thresholds with care.

Early access. Jev launched in September 2026. Pricing and behavior may change. Re-validate after model updates.

This is where independence matters. If the same team builds the agent, picks the scorer, and grades the results, that is a conflict of interest. That is the case for independent voice AI evaluation. Evalgent pairs full coverage with independent validation of the scorer on your own calls. Compare scorer types in verifiable rewards vs LLM judges vs Jev and Jev as a judge.

How to move from sampled to full-population scoring

Use this sequence to shift from sampled review to a census without losing rigor.

1. Define your scoring questions. Write the exact Choice, Score, and Noul questions for your quality bar. Cover intent, resolution, policy, and escalation. Aim for 10 to 20.

2. Build a labeled validation set. Have humans grade a few hundred representative calls. Stratify across intents, locales, and versions.

3. Validate the scorer against human labels. Measure agreement and calibration per question. Rewrite or drop questions that disagree with humans.

4. Score the full population. Wire the validated scorer to every finished call. Score transcripts as they land.

5. Set confidence thresholds for review. Route low-confidence answers to humans. See human-in-the-loop evaluation.

6. Slice, alert, and join outcomes. Track rates per segment hourly. Join scores to CSAT, conversion, and churn.

7. Keep independent evaluation on your own calls. Have a third party re-score a sample and audit the scorer. The census is your coverage. Independent evaluation is your evidence.

For the broader framework, start with voice agent evaluation. Then read containment vs deflection once you have full coverage.

Jev can't abstain: always include an escape option

Jev always returns one of the options you define. It has no built-in "I don't know." When a call transcript holds no evidence for any option, such as a two-word turn, a dropped call or a caller talking about something else, Jev still picks the least-wrong option, and that label looks just like a confident answer. TypeSafe's own Choice docs recommend adding an "other" or "none of the above" option when the list may not cover every input.

Scoring every call makes this matter more, not less. At 100% coverage, short, dropped and off-topic calls are a meaningful share of your volume, and without an escape option they all receive confident scores that skew your dashboards.

Four rules keep this from hurting you on live calls:

• Add an explicit escape option such as `not_enough_information`, `not_applicable` or `none_of_the_above` to every Choice question where evidence can be missing. Describe it as concretely as the real options, because Jev reads only the descriptions, not your option keys.
• Ask whether the evidence exists first. A Noul like "does the caller state a reason for calling?" before the Choice question separates "no evidence" from "evidence for option B".
• Check the distribution, not just the label. In one independent edge-case test, a forced choice with no valid option still returned a pick at 0.52 probability but only 0.04 confidence (jev-exploration). The numbers flagged the problem; the label did not. Set your human-review cutoff from your own labeled calls, because an independent study on data unlike Jev's training found its confidence can run high (jev-ood-calibration).
• Track the escape-option rate. If it climbs, your transcripts, STT or option set have drifted away from what callers actually say.

Frequently asked questions

What does it mean to score every voice agent call?

Scoring every voice agent call means grading 100% of production calls instead of a sample. Each finished transcript goes to a scorer that labels intent, resolution, policy, and escalation. It is a census, not a subset. It becomes practical when scoring costs a fraction of a cent per call and returns in under a second.

How much does voice agent evaluation cost per call?

It depends on the grader. Illustratively, Jev scoring a 2,300-token call costs about $0.0000966, since output is free. A mid-range LLM judge at $3 input and $15 output per million tokens costs about $0.0114. In LangChain's measured study, Jev cost $0.00035 per call, and $0.34 total versus $28.17 for Claude.

Can you really score 100% of calls instead of sampling?

Yes, when per-call cost is low enough. Illustratively, Jev scores one million calls with 10 questions each for about $97. LangChain's measured per-call cost implies about $350. That is cheaper than most sampling programs. You still validate the scorer against human labels and keep independent evaluation on your own calls.

Is scoring every call better than sampling live calls?

For coverage, yes. A 2% sample of 100,000 calls shows about 6 hits of a 0.3% failure. A census shows about 300. Census scoring also supports segment slicing, powered A/B tests, and hourly drift alerts. Sampling still matters for nuanced human review and for validating the scorer. The best setup is a hybrid.

What is voice agent evaluation economics?

Voice agent evaluation economics is the cost-benefit trade behind how many calls you grade. The benefit is coverage and confidence. The cost is time and money per graded call. Sampling won because grading was expensive. When per-call cost falls near zero, the trade flips toward scoring the full population.

Does scoring every call remove selection bias?

It removes sampling-based selection bias, because there is no sample to skew. You measure every call that happened. It does not remove every bias. The scorer can carry its own biases, and transcription errors flow into scores. That is why you validate the scorer against human labels and audit it independently on your own calls.

What are the limits of using Jev to score every call?

Jev decides and scores. It does not generate language. It reads transcripts and structured state, not raw audio, so it misses tone and prosody. Its confidence is calibrated, but that is not correctness on your data until validated. It is early access, so re-validate after updates. Scoring everything wrongly is worse than a good sample.

Do you still need independent evaluation if you score every call?

Yes. A census gives coverage, not proof that your scorer is right. An independent third party re-scores a sample, checks calibration, and audits the scorer on your own calls. That is your evidence. Grading your own agent with your own scorer is a conflict of interest, so independent evaluation stays essential at full coverage.

The bottom line

Cheap, calibrated scoring turns a 2% guess into a measured census of every call. The real value is the insight that coverage unlocks, as long as an independent party validates the scorer.

Ready to see what your voice agent actually does on every real call? Book a demo and we will score your calls and audit the scorer as an independent third party.