Migrating Between Voice Agent Vendors: A Playbook

# Migrating between voice agent vendors: a playbook

Quick answer

To migrate a voice agent vendor without regressions, freeze one shared test set. Run the old and new vendor on identical cases. Compare call by call, not in aggregate. Gate the cutover on a fixed evidence bar. Run both in parallel or shadow, then keep a rollback ready.

Most vendor migrations start with a demo and a discount. Someone promises lower latency or a cheaper per-minute rate. The contract gets signed. Then the new vendor goes live, and real callers meet it for the first time. That is where the regressions show up.

There is a calmer way to do this. Treat the migration as a change that must pass the same bar as any release. A new vendor is not special. It is a large change to a system callers depend on. So it deserves the same evidence you would demand of a new prompt or a new model.

This playbook is the operational procedure for moving between vendors. It assumes the decision to switch is already made. The question here is narrow. How do you land the change without breaking anything your callers rely on?

Why a vendor migration is just a regression test

A voice agent is a pipeline. Speech-to-text turns audio into words. A language model decides what to say. Text-to-speech turns the reply into a voice. Orchestration and telephony hold it together. A vendor migration replaces some or all of that.

Your callers do not care why behavior changed. They only notice that it did. A new vendor and a new prompt are the same class of event. Both alter what the agent hears, decides, and says. Both can introduce a regression — a case that used to pass and now fails.

So the migration is a regression test at heart. You are asking one question. Does the new vendor match or beat the old one on the cases you care about? Everything in this playbook exists to answer that question with evidence, not a sales deck.

The common mistake is trusting the vendor's own benchmark. That benchmark used their audio, their metrics, and their chosen happy paths. It says nothing about your accents, your policies, or your hardest calls. Your own evaluation on your data is the only number that predicts real behavior.

What breaks when you migrate

Regressions in voice are rarely loud. The agent still answers. It just gets worse in ways a dashboard hides. That is why you compare on identical calls instead of summary numbers.

A new speech-to-text engine changes what the agent hears. It may mishear an accent it never saw. It may mangle account numbers or clip callers mid-sentence. Word error rate can rise on your domain terms even when it looks fine overall.

A new language model changes what the agent decides. It may skip a required disclosure. It may call the wrong tool. It may follow a generic policy instead of yours. These misses pass a shallow review and still create compliance risk.

A new voice changes what the caller hears. It may mispronounce a product name. It may feel slower, which raises perceived latency even when the logic is identical. A new platform changes turn-taking and interruption handling all at once.

Aggregate numbers make all of this worse. A new vendor can raise the average and still break your hardest cases. The wins are loud and the losses are quiet. You want the specific list of scenarios that got worse. That list is your regression report.

Freeze a shared test set before you touch anything

You cannot compare two vendors fairly unless they face the same inputs. So the first move is to freeze a shared test set. This is the single most important step in the whole migration.

Build the set from your real calls. Capture happy paths, edge cases, interruptions, accents, and adversarial callers. Include the flows that carry legal or financial weight. A few dozen well-chosen scenarios beat hundreds of shallow ones. The point is coverage of what actually happens on your line.

Version the set and lock it. Both vendors must meet identical inputs on every run. If the test set drifts between runs, the comparison means nothing. This is the discipline behind comparing vendors on the same test cases.

Define expected outcomes before you run anything. Decide what a pass looks like for each scenario. Write down the metrics and the thresholds. A migration is only credible when the bar is set in advance. Setting it after you see the results is how bias creeps in.

Run the old and new vendor in parallel

Once the set is frozen, run it against both vendors. This is a parallel run — the old and new systems process the same work so you can compare them directly. It is a classic, low-drama way to de-risk a migration.

In production you have two safe patterns. A shadow run sends live calls to the new vendor silently, beside the live one, with no caller impact. A canary routes a small slice of real traffic to the new vendor and watches the result. Both let you observe live behavior before you commit.

Keep an independent evaluation between you and both vendors. The party that scores the migration should not be the party selling you the new contract. Neutral scoring on your fixed set is what makes the result trustworthy to your own leadership.

How to migrate between voice agent vendors

Here is the step-by-step plan. Work through it in order. Do not skip the parallel run to hit a launch date.

1. Freeze a shared test set. Capture your real calls as a versioned suite. Cover happy paths, edge cases, interruptions, accents, and regulated flows. Lock it so both vendors face identical inputs. This suite is the spine of the whole migration.

2. Define the evidence bar first. Write down the metrics and the thresholds that decide the switch. Set them before you see any result. Include your must-not-regress cases, like disclosures and escalation to a human.

3. Baseline the incumbent. Run the frozen suite against your current vendor. This is your reference. You are not asking whether the new vendor is good in the abstract. You are asking whether it matches or beats this baseline.

4. Run the candidate on the same suite. Replay the identical set against the new vendor at realistic concurrency. Use the same profiles for accent, pace, and line quality. Both vendors must meet the same range of callers, including under load.

5. Compare call by call. Line up each scenario, old versus new. Look for cases that used to pass and now fail. The value is not the mean score. It is the named list of calls that regressed. Averages hide the losses you most need to see.

6. Gate the cutover on the bar. Hold the candidate to the thresholds you set in step two. If it misses on a critical metric, the migration fails. There are no exceptions for a cheaper price or a slicker demo.

7. Cut over in stages with rollback ready. Ship behind a canary or a blue-green deployment so you can switch back fast. Widen traffic only as live metrics hold. If they slip, roll back and add the failure to your suite.

Migration phases at a glance

Each phase has one goal and one exit criterion. You do not enter the next phase until the current one passes. This table is the whole playbook in one view.

Gate the cutover on an evidence bar

The gate is where discipline pays off. A migration should never ship on "it seems fine." It ships when the candidate clears a bar you defined before you looked at any result.

Anchor the bar to your own thresholds, not the vendor's. This is your production readiness bar applied to a migration. Task completion, latency, containment, and required disclosures all have numbers. If the new vendor misses one critical number, the migration does not proceed.

Frameworks for managing this kind of risk already exist. The NIST AI Risk Management Framework treats measurement and governance as core to trustworthy AI. A fixed evidence bar is how you put that principle into practice for a voice migration.

Beware the loud win. A new vendor that improves the average can still break your regulated flow. One failed disclosure is a compliance issue, not a rounding error. The gate exists to catch the quiet losses that a headline metric buries.

Plan the rollback before you cut over

A safe migration needs an easy exit. If switching back is hard, you will stay on a broken vendor under pressure. So build the escape hatch before you need it.

Put one abstraction layer between your app and every vendor. Your code calls your layer. Your layer calls whichever vendor is live. Then the live vendor is a config value. A rollback becomes one change back to the known-good option, not a rewrite.

Test the rollback path before the cutover. Prove you can flip back in seconds. Rehearse it the way you rehearse any incident. The moment to discover your rollback is broken is not during a live regression on real calls.

This layer also makes future moves cheap. It is the difference between a routine deploy and a scary migration. Managing more than one vendor gets easier too, which supports a multi-vendor strategy if you want one.

Where Evalgent fits

A migration is only safe when you can prove the new vendor is at least as good on your calls. Evalgent is the independent party that provides that proof, separate from any vendor. Scenarios capture your real calls as a fixed, versioned suite you own. Profiles vary caller accent, pace, and line quality, so both vendors face the same range. Metrics score each run against expected outcomes on one fixed definition, which makes the two runs directly comparable. Evaluations replay the identical suite against the incumbent and the candidate at concurrency, so they meet on equal terms. Reviews let your team hear the call behind any score, so a regression is never just a number. Because the suite is yours and runs against anyone, the migration becomes a gated change instead of a leap of faith. To put a neutral evaluation between you and every vendor, book a demo. Our overview of voice agent evaluation and the guide to testing versus evaluation show how the pieces fit together.

Frequently asked questions

What does it mean to migrate a voice agent vendor?

It means moving your voice agent from one vendor to another. You might migrate the speech-to-text engine, the language model, the voice, or the whole platform. The goal is to change the vendor while keeping caller behavior at least as good as before, with no regressions on the flows you care about.

How do I migrate without introducing regressions?

Freeze one shared test set of your real calls. Run the old and new vendor on identical cases. Compare call by call, not in aggregate. Gate the cutover on a bar you set in advance. Then run both in parallel or shadow and keep a rollback ready before you widen traffic.

Why not just trust the new vendor's benchmark?

Because it used their audio, their metrics, and their chosen happy paths. It says nothing about your accents, your policies, or your edge cases. A vendor benchmark is marketing. Your own evaluation on your data is evidence. Only the second one predicts how the migration behaves on real calls.

What is a parallel or shadow run in a migration?

A parallel run processes the same work through both vendors so you can compare them directly. A shadow run sends live calls to the new vendor silently, beside the live one, with no caller impact. Both let you watch real metrics against your baseline before you commit any traffic to the switch.

How big should my test set be before I migrate?

Big enough to cover your real call types, not just happy paths. Include edge cases, interruptions, accents, regulated flows, and adversarial callers. A few dozen well-chosen scenarios beat hundreds of shallow ones. The suite must be versioned and frozen, so both vendors face identical inputs on every run.

When should I gate the cutover?

Set the evidence bar before you run anything. Decide the metrics and thresholds that must hold. Gate the switch on those numbers, not on a demo. If the candidate misses a critical threshold, the migration fails. This keeps a cheaper price or a slick pitch from overriding your callers' experience.

How do I roll back a vendor migration safely?

Put an abstraction layer between your app and every vendor. Then the live vendor is a config value, and a rollback is one change back to the known-good option. Test that path before the cutover. Without the layer, a rollback is a code rewrite under pressure during a live regression.

Should I migrate one component or the whole platform?

Migrate the smallest piece that solves your problem. Isolating one layer, like speech-to-text, keeps the change easy to measure and reverse. Moving the whole platform shifts everything at once, which raises risk. If you must move the platform, still compare against the same frozen suite and gate on the same bar.

The bottom line

A vendor migration is a change like any other, so hold it to the same evidence bar. Freeze one shared test set, prove the new vendor matches or beats the old on identical calls, then cut over in stages with a rollback ready.