Metrics for a financial services voice agent

Most teams measuring a banking voice agent borrow a support scorecard and stop there. They track containment, handle time, and a satisfaction score. Those numbers matter, but they miss what makes financial services different. Here, a wrong answer is not just annoying. It can move money, leak an account balance, or breach a regulation.

This post covers the KPIs that actually predict a safe banking voice agent. It gives reasonable starting targets and shows how to measure each one honestly. It is a metrics post, not a testing walkthrough. For the how-to on building test suites, see the financial services testing guide. Here we focus on the scoreboard.

Why financial services metrics are different

A support agent that mishears a word usually recovers. It asks again, and the caller repeats. The cost of a small error is a slightly longer call. Nobody gets hurt.

A banking agent does not have that luxury. Get the account number wrong, and you read a stranger's balance aloud. Skip a required disclosure, and you have a compliance gap. Authenticate the wrong person, and you have a fraud incident. The blast radius of a single mistake is far larger.

That changes how you read every metric. Some numbers stop being percentages you optimize and become gates you must pass. Authentication is not a score to nudge upward over quarters. It is a wall. Either the caller proved who they are, or they did not. There is no partial credit for almost proving it.

The other shift is entity accuracy. In support, the transcript can be fuzzy and the call still succeeds. In banking, the entities are the task. An account number, a routing number, a dollar amount, a confirmation code — one wrong digit changes everything. We treat these as critical entities, and we score them pass or fail per call.

The metrics that actually matter

A financial services voice agent needs a scorecard built around safety, not just efficiency. The following KPIs carry most of the signal. Read them as a set, because in banking the safety metrics constrain the efficiency ones far more tightly than elsewhere.

Caller authentication success rate

Authentication is the gate before any account action. The metric has two halves, and you must track both. The first is the legitimate-caller pass rate — how often a real account holder clears verification. The second is the imposter rejection rate — how often the agent refuses someone who cannot prove identity.

These pull against each other. Loosen verification to help real callers, and you let imposters through. Tighten it to block fraud, and you lock out legitimate customers. The balance depends on your risk appetite and the sensitivity of the action requested.

Treat authentication as a hard pass or fail per call. A caller is either verified to your standard or not. There is no 90% verified. Many teams layer multi-factor authentication so a single misheard answer does not open an account. The key performance indicator you watch is the false-accept rate, and it should be as close to zero as your controls allow.

Compliance and disclosure adherence

Banking calls carry required scripts. Certain disclosures must be read. Certain consent language must be captured. Certain phrases are prohibited because they imply advice you are not licensed to give. Adherence measures whether the agent said the required things and avoided the forbidden ones.

Score this per call as a checklist, not a vibe. Each required disclosure is a box. Each prohibited phrase is a violation. A call passes only if every required box is checked and no violation appears. One missed disclosure fails the call, even if everything else was perfect.

The trap is measuring adherence on averages. A 98% disclosure rate sounds strong until you realize the missing 2% are real regulatory gaps. Frameworks like the NIST AI Risk Management Framework push teams to treat these as controls, not soft targets. In a regulated domain, the tolerated failure rate is often zero.

Fraud and suspicious-activity handling

Some calls are attacks. A caller probes for account details, pressures the agent to bypass a step, or gives answers that do not add up. Fraud handling measures whether the agent recognizes these patterns and responds correctly — refusing the action, escalating, or flagging the interaction.

Measure it two ways. Detection rate is the share of known suspicious scenarios the agent catches. False-flag rate is the share of legitimate callers wrongly treated as suspicious. A good agent catches most attacks without accusing real customers. Part of this connects to know your customer obligations, where identity and intent must be established before you act.

You cannot measure this from production alone, because real attacks are rare and unlabeled. You inject known fraud scenarios into your test suite and score the response. Detection you cannot reproduce on demand is not a metric. It is a hope.

Critical-entity accuracy

This is the metric most support scorecards miss entirely. Critical entities are the values that carry the transaction — account numbers, routing numbers, dollar amounts, dates, confirmation codes. Accuracy here is not word error rate averaged over a call. It is exact-match, per entity, per call.

Score it strictly. If the caller says "four thousand two hundred" and the agent hears "four thousand two hundred and twenty," that call failed. The rest of the conversation does not rescue it. A single misheard digit in an account number is a full task failure, full stop.

Underlying transcription quality still matters, and teams track word error rate on realistic audio. But the headline number is entity-level exact match. Report it separately for the highest-stakes entities, because an error in a dollar amount is worse than one in a spoken pleasantry.

Latency plays a role too. Long response gaps make callers repeat digits and talk over the agent, which corrupts the entities you most need exact. The classic telephony reference for acceptable one-way delay is the ITU-T recommendation G.114, below roughly 400 milliseconds. Conversational agents feel best well under that.

Escalation accuracy

Escalation measures whether the agent hands off to a human at the right moments. In banking, the stakes tilt the balance toward escalating more readily. When an action is high-value or a caller is distressed about fraud, a wrong self-service answer is expensive.

Track two rates. False containment is the share of calls that should have escalated but did not — the caller needed a human and never got one. Premature escalation is the share of transfers the agent could have handled. For sensitive actions, err toward escalation. The escalation design guide covers how to define the handoff triggers behind this metric.

Containment versus resolution, and PII

Two more ideas round out the scorecard. Both behave differently in banking than in general support.

Containment is the share of calls handled without a human transfer. Resolution is whether the caller's actual need was met, safely and correctly. In banking, never report containment alone. A contained call that leaked data or skipped a disclosure is a failure that looks like a success. Resolution here means the task completed, the caller was verified, the disclosures were read, and nothing sensitive escaped.

PII and security are the quiet floor under everything. The agent must never read an account balance to an unverified caller. It must never repeat a full account number when a masked one suffices. It must never store or expose sensitive data outside policy. The PII handling guide details what to test, but the metric is blunt — any confirmed leak of protected data is a critical failure, counted separately and gated at zero.

Neither of these belongs buried in an average. They belong as pass-or-fail gates on top of the efficiency numbers, because in a regulated domain the floor matters more than the ceiling.

Target ranges to start from

The targets below are reasonable starting points for a general banking voice agent, not universal truths. Adapt them to your risk appetite, your regulator, and the sensitivity of each action. A balance-inquiry agent and a wire-transfer agent should not share the same authentication bar.

Read the table as a set, never row by row. The safety gates constrain the efficiency numbers. You cannot buy containment by loosening authentication. For a fuller template you can adapt, see the metrics scorecard template.

How to build a financial services metrics scorecard

Use these steps to turn the KPIs above into a scorecard you can run and trust.

1. Separate gates from scores first. List which metrics are hard pass or fail — authentication, disclosure adherence, PII leaks, critical entities — and which are ranges. Gates are not averaged. One failure fails the call.

2. Define your critical entities explicitly. Write down every value that carries a transaction — account, routing, amount, date, code. Score each as exact match per call, and report the highest-stakes entities on their own.

3. Inject fraud scenarios you can label. Real attacks are rare and unlabeled in production. Build a suite of known suspicious calls so detection and false-flag rates come from reproducible cases.

4. Score compliance as a checklist per call. Turn every required disclosure and prohibited phrase into a box. A call passes only when every box is right. Never blend this into a percentage that hides gaps.

5. Test authentication at both edges. Measure the legitimate-caller pass rate and the imposter rejection rate together. Tuning one moves the other, so you must watch the trade in one view.

6. Sample every call for PII leaks. Have reviewers check whether sensitive data reached an unverified caller or appeared where policy forbids. Treat any confirmed leak as a critical, separately counted failure.

7. Re-baseline after every change. A prompt tweak, a model swap, or a new script can move authentication, entities, and disclosures at once. Re-measure against the same gates after each meaningful change.

A short worked example

Suppose a banking voice agent handles 10,000 calls in a month. The dashboard shows 78% containment and a 4.2 satisfaction score. The team is pleased.

Now apply the gates. Critical-entity accuracy on account numbers and dollar amounts is 97.5%. That sounds high, but it means roughly 250 calls carried a wrong digit somewhere. Each is a full task failure. Disclosure adherence is 98%, so about 200 calls missed a required disclosure — 200 regulatory gaps hiding inside a strong-looking average.

Authentication looks fine at a glance. But a review finds the false-accept rate is 0.4%, meaning around 40 unverified callers cleared the gate. On a wire-transfer flow, that is not a rounding error. It is 40 potential fraud incidents. A PII sample turns up three calls where a balance was read to a caller who had not fully verified.

The read is clear. This is not a 78% success story. It is an agent with an authentication leak, a disclosure gap, and an entity-accuracy problem that a support scorecard would never surface. The fix is not more automation. It is tightening the gates first, then re-earning containment safely. For where these numbers should sit before launch, see the production readiness bar.

Measuring financial services voice agent metrics with Evalgent

Evalgent is an independent platform for testing and evaluating voice agents, so the authentication, compliance, and entity numbers you report are measured against outcomes rather than self-reported by the system under test. Its five primitives map directly onto the scorecard above.

• Scenarios reproduce real banking situations — balance inquiries, disputed charges, wire requests, and injected fraud attempts — so metrics come from representative and adversarial traffic.
• Profiles vary caller accent, background noise, and stress, exposing the entity and authentication edges that averages hide.
• Metrics track authentication, disclosure adherence, fraud handling, critical-entity accuracy, and escalation as a linked set, with hard gates where they belong.
• Evaluations score each call against your rubric, treating a misheard digit or a missed disclosure as a task failure rather than a minor slip.
• Reviews give humans a labeled sample to verify PII handling and keep the definition of a safe call honest over time.

Teams use this to catch authentication leaks and disclosure gaps before they reach production. To see it on your own call types and regulations, book a demo. For vendor selection, the vendor evaluation guide pairs well with this scorecard.

Frequently asked questions

What is the most important metric for a financial services voice agent?

Caller authentication success, treated as a hard pass or fail. Before any account action, the caller must prove identity to your standard. Watch the false-accept rate especially, because a single unverified caller who clears the gate can become a fraud incident. Efficiency metrics only matter once this gate holds.

Why is a single misheard digit a task failure?

Because in banking the entities carry the transaction. An account number, routing number, or dollar amount with one wrong digit is not a fuzzy transcript. It is the wrong account or the wrong amount. Score critical entities as exact match per call. The rest of the conversation cannot rescue a wrong digit.

How do you measure caller authentication success?

Track two numbers together. The legitimate-caller pass rate is how often real account holders clear verification. The false-accept rate is how often imposters get through. Tightening one moves the other, so watch them in one view. Treat each call as verified or not, with no partial credit.

How should compliance and disclosure adherence be scored?

As a checklist per call, never as an average. Each required disclosure is a box that must be checked. Each prohibited phrase is a violation. A call passes only when every box is right and no violation appears. In a regulated domain, the tolerated failure rate is often zero.

How do you test fraud handling if real attacks are rare?

You inject known suspicious scenarios into your test suite. Real fraud is rare and unlabeled in production, so it cannot anchor a reliable metric. Build reproducible attack cases — probing, pressure, inconsistent answers — and score detection rate and false-flag rate against them. Detection you cannot reproduce on demand is not a metric.

What is critical-entity accuracy?

It is exact-match accuracy on the values that carry a transaction — account numbers, routing numbers, dollar amounts, dates, and confirmation codes. Unlike averaged word error rate, it is scored per entity, per call. One wrong digit fails the call. Report the highest-stakes entities separately, since an error there is worse than a misheard pleasantry.

Should a banking voice agent escalate more than a support agent?

Usually yes. In banking, a wrong self-service answer on a high-value or fraud-related call is expensive and sometimes irreversible. Track false containment and premature escalation, but tilt the balance toward escalating for sensitive actions. When the caller is distressed or the action is high-value, reaching a human quickly is the safer default.

How do you measure PII and security for a voice agent?

Sample calls and check whether sensitive data reached an unverified caller or appeared where policy forbids. The metric is blunt — any confirmed leak of protected data is a critical failure, counted separately and gated at zero. Never let this hide inside an average, and never report containment without checking for leaks.

The bottom line

In financial services, the safety gates come first and the efficiency numbers come second. Measure authentication, disclosures, and critical entities as pass-or-fail, and a single misheard digit stops looking like a rounding error.