Voice and Screen State Drift in Voice Agents

Your voice agent lives inside a product. The caller talks while something else is on a screen. It might be their own app, a page shared through a co-browsing voice agent flow or a staff-side agent-assist panel. That means voice and screen state both describe one order, appointment or application. When they disagree, nobody sees an error. The caller hears "two large pepperoni" and the screen says one.

This post is about voice agent state drift. It covers where drift comes from, eight drift types, the controls that close each one, and a test that catches it before callers do. It sits next to our posts on workflow testing (partial failure across steps) and integration testing (tool calls and contracts). Those ask whether each step finished. This one asks whether every surface agrees.

What voice agent screen sync means

State drift: in a voice agent with a screen, two surfaces that should describe the same record hold different values, and neither surface knows it.

Good voice agent screen sync means these three copies agree. There are at least three:

• What the caller heard. Audio that actually played, which can be shorter than what the model generated.
• What the screen shows. Whatever the client last rendered, which can be stale. This is the voice agent UI state.
• What the server stored. The record other systems read, such as the POS ticket or CRM row.

The server copy counts downstream. The other two are views, and a caller acts on the view they have. They say "yes, confirm" to the audio. They click "Pay" on the screen. Both land on the server, which must know which version each was based on.

This is different from a single wrong value. A mis-heard code is covered in alphanumeric capture. Drift is a correct value present on one surface and missing on another.

Why it happens: three clocks that never agree

Voice and UI run on separate clocks. Speech is slow and sequential. UI events are fast and parallel. The server sits between them and commits on its own schedule.

In this illustrative timeline the server commits "qty 2" as version 9. The screen push takes about 80 ms and the render about 40 ms. The agent needs the tool result, then the model, then the first TTS audio. In this example the audio starts roughly 350 ms after commit. So the screen changes about 120 ms after commit and the voice confirms about 350 ms after commit. A caller who clicks in the first 120 ms clicks against version 8.

The defect is that the click carries no version. The server cannot tell it was stale, so it applies it.

Three other facts widen the gap. Voice may ride SIP or WebRTC while the UI rides a WebSocket or HTTP, so no shared clock orders an utterance against a click. Next, WebSocket gives you no flow control. MDN says the `WebSocket` interface doesn't support backpressure, so a burst of messages can pile up in the client before it renders them (MDN WebSockets API, checked November 2026).

Finally, models read the UI too. If your agent reads the page, it picks a channel to believe. A July 2026 study of GUI agents (arXiv 2607.04334) built 735 probes across web, mobile and desktop interfaces to test what happens when pixels and structural data conflict, including 225 zero-edit divergences mined from live production websites. On unedited stale snapshots from live pages, models followed the outdated structure on up to 0.88 of probes. Among its mitigations, only a training-free consistency gate reduced both hijacking and task error. It covers GUI agents, not phone agents, but the lesson transfers: never let one channel silently win. Our posts on browser control and web form filling cover agents that drive the page.

A taxonomy of eight drift types

Name the failure and you can write a test for it. This working taxonomy is our framework, not a standard.

Types 2, 5 and 8 corrupt the record. Types 1, 3, 4 and 6 corrupt what the caller believes. Both end with a human acting on something false.

Make the server the single source of truth

This is optimistic concurrency applied to a call. The rule is short. The client and the voice model never write state. They send intents, and the server decides what the state is.

Server-authoritative state: the server holds the only writable copy of the session record, and every surface renders a projection of it.

That gives you three concrete requirements.

1. A version on every record. Increment it on each accepted change.

2. An intent, not a value. Send `add_item(pepperoni, qty=1, base_version=8)`, not "the order is now X".

3. One projection for all surfaces. The agent's tools, the UI and the receipt read the same view.

For HTTP, the standard tool is the entity tag. MDN describes the `If-Match` request header as making a request conditional. If the conditional does not match, the response is `412 Precondition Failed`. MDN's ETag page says ETags help prevent simultaneous updates from overwriting each other, which it calls "mid-air collisions". Your version number can be the ETag.

Here is a simplified server handler. It is not production code. It shows where the check sits.

# Simplified sketch: version check on every intent, voice or click.
class StaleState(Exception):
    pass

def apply_intent(order, intent):
    # intent = {"id": "...", "type": "set_qty", "item": "pep", "qty": 2,
    #           "base_version": 8, "source": "voice" | "ui"}
    if intent["id"] in order.seen_intents:
        return order.snapshot()              # idempotent replay
    if intent["base_version"] != order.version:
        raise StaleState(order.version)      # map to HTTP 412 or a tool error
    order.apply(intent)
    order.version += 1
    order.seen_intents.add(intent["id"])
    return order.snapshot()                  # includes the new version

Two details matter. The voice tool path must pass `base_version` too, or the agent becomes the one writer that can clobber the screen. And a stale rejection is not an error to hide. Return the fresh snapshot so the agent can say "that changed, it is now two".

For the agent, a rejection is a tool result like any other. Cover it in your integration tests.

Event ordering between speech turns and UI events

Inside one ordered channel, order is safe. MDN says `RTCDataChannel.ordered` defaults to `true`. Across channels, nothing orders them for you. A click on the HTTP path and a transcript event on the voice path have no shared sequence.

So stamp your own. Every event that can change or describe state carries two fields:

• `session_seq`: a counter the server increments per session.
• `state_version`: the record version the event was based on or produced.

Apply a plain rule. If an incoming event has a `state_version` below the server's, treat it as stale. If a render receives a lower sequence number than the last one drawn, discard it. This kills reorder drift (type 5) and most stale state reads (type 1).

One more trap is spoken. The agent says "I have added it" while a slower tool call still runs. Wire confirmation to the commit ack, not the intent to call the tool, or you ship premature confirms (type 3). The same discipline shows up in workflow testing, where a step can look done before its write-back lands.

Barge-in: what the caller heard versus what was generated

Interruptions make drift worse because speech is cut mid-sentence while the UI already holds the full result.

The agent generates "Added a large pepperoni, extra cheese, and garlic knots." The caller interrupts after "extra cheese". The screen lists all three. The caller never heard the third. A "yes, that's right" now agrees to something unheard.

The mechanics are in OpenAI's Realtime conversations guide, which names `gpt-realtime-2.1` as of November 2026. For WebSocket connections it says the client manages audio playback "and thus must stop playback and handle truncation." You record how many milliseconds the user actually heard and send `conversation.item.truncate` with that played duration as `audio_end_ms`. For WebRTC and SIP, the guide says "The server will automatically truncate unplayed audio when there's a user interruption." The same page warns the model cannot precisely align transcript and audio, so truncation removes the transcript text for the unplayed portion.

Your conversation state is only as good as your played-audio tracking. The UI also needs a rule for unheard content:

• Mark it. Show unheard items as "pending confirmation" until the caller acknowledges them.
• Re-offer it. The agent's next turn restates the unheard portion.
• Do not commit on an ambiguous yes. If the last confirmation covered cut-off speech, ask again.

For false and missed interruptions, see interruption detection evaluation. The drift question is narrower: what did the interruption leave on screen?

Reconnects and stale WebSocket state

A dropped socket does not tell the UI that anything is wrong. MDN's client guide says the `close` event fires when a connection is closed by either side or by an error. It does not describe automatic reconnection after an unexpected close, so your code owns that. RFC 6455 says a Ping frame may serve as a keepalive or as a way to verify the remote endpoint is still responsive. Use that to notice a silent drop sooner.

The failure is a reconnect gap (type 6). The call carries on and the screen freezes at version 6. When the socket returns, a naive client draws version 11 on top of a version 6 it never reconciled.

Close it with a handshake:

1. On WebSocket reconnect, the client sends its last `session_seq` and `state_version`.

2. The server replies with a full snapshot plus the current version, or a replay of missed events if the gap is small.

3. The client replaces its state. It never merges over it.

4. The agent is told the screen was offline. If it spoke during the gap, it can recap: "Your screen just refreshed. You have two pepperoni and a total of $31."

Test the tab wake-up path as well as the network drop. MDN also advises closing sockets when a user leaves a page, since an open connection may keep it out of the bfcache.

Races: the caller clicks while the agent speaks

A voice agent race condition starts when a caller does not wait politely. They tap "Remove" while the agent says "Anything else?" Now two intents race, and one came from a human who is looking at a screen.

Decide the precedence before the race happens. A workable default:

• A click beats a stale voice intent. The click reflects what the caller can see and chose to touch.
• A confirmed voice intent beats a stale click. If the click was based on an older version, reject it with 412 Precondition Failed and refresh.
• Neither silently wins. Whoever loses gets told.

Both paths also need idempotency. If a UI retry and a voice retry carry different identifiers for the same intent, you charge twice (type 8). The IETF Idempotency-Key header draft (draft-08, last updated 22 October 2026, still an Internet-Draft and not an RFC) is a useful reference. It says the key "MUST be unique and MUST NOT be reused with another request with a different request payload." It describes a 422 for reuse with a different payload and a 409 when a retry arrives while the first request is still processing. Treat it as a pattern, not a standard.

Give each logical intent one id, shared by every surface that might send it. A "pay" tool call and a "Pay" button should collide on the same key. For money, pair this with rollback actions so a double action can be undone.

A race that fires one time in ten will not show in one manual call. At an illustrative 10% per run, 20 runs catch it at least once with probability 1 − 0.9^20, about 88%. Thirty runs give about 96%. That takes automated repetition.

Accessibility: announcing screen changes without talking over the agent

If your product serves screen reader users, a voice agent and a screen reader may speak at the same time. WCAG 2.2 success criterion 4.1.3 (Level AA) says status messages must be programmatically determinable through role or properties so assistive technologies can present them without receiving focus. The Understanding page gives the intent as making users aware of "important changes in content that are not given focus."

In practice, an order line that updates because of a voice command is a status change. Put it in a live region. MDN's `aria-live` page says `polite` presents updates "at the next graceful opportunity," while `assertive` can interrupt, "potentially clearing the speech queue of previous updates." Use `polite` for item changes. Reserve `assertive` for task-stopping errors.

MDN also notes that content added after the accessibility tree is built may not be noticed. Create the live region empty at page load, then update its text. Test with a screen reader running while the agent talks. This is a design note, not legal advice; check your own accessibility obligations separately.

Echo and replay confusion

Two quieter drifts come from loops. Echo: the agent reads back "two pepperoni", the caller's mic picks it up, and STT treats it as the caller. Do not apply any state change from a transcript segment that matches the agent's last output, and check your echo cancellation. Replay: after a reconnect or queue redelivery, an old event arrives again and "remove item" runs twice. Deduplicate by event id and `session_seq`, as in the race section.

How to test voice agent screen sync and drift

Voice agent screen sync needs a test that observes three states at once. Here is the procedure.

1. Define the state projection. Pick the fields that matter, such as item, quantity, total, slot, status. All three sources must be reduced to the same shape.

2. Stamp versions and sequence numbers. Log `state_version` and `session_seq` on every tool call, UI event and render. Without them drift detection finds mismatches but cannot explain them.

3. Derive state from the call. Parse what the caller said and what the agent said it did into the projection. Use the played audio where possible, not the generated text.

4. Read the UI. Drive a real browser or client in the test. Read the rendered values, not the props, so rendering bugs count.

5. Read the server. Fetch the stored record after each turn.

6. Compare at every turn boundary. Flag any field where the three differ. Tag it with a taxonomy type.

7. Inject faults. Delay the UI push, drop the socket, interrupt at a chosen word, fire a click mid-utterance, replay an event.

8. Repeat for races. Run each timing-sensitive scenario 20 to 30 times.

9. Gate releases. Block a prompt or UI change if any scenario shows drift that persists past the next turn. The release flow is in prompt versioning and safe shipping.

Here is a scenario set in YAML. It is a sketch of a format, not a specific tool's schema.

# Illustrative scenario format for a drift test harness
- id: DR-01-stale-read
  fault: delay_ui_push_ms=800
  script: caller says "make it two"; agent confirms
  assert: ui.qty == server.qty within 1000ms of confirmation
- id: DR-02-click-during-speech
  fault: ui_click remove_item at agent_speech_t=+300ms
  assert: one winner; loser gets 412 and is told; no ghost item
- id: DR-03-barge-in-third-item
  fault: caller_interrupt after word 6 of 9
  assert: unheard item flagged pending; no commit on bare "yes"
- id: DR-04-reconnect-gap
  fault: drop_ws for 5s during 2 server commits
  assert: snapshot on reconnect; ui.version == server.version
- id: DR-05-duplicate-intent
  fault: send voice and click for same intent_id
  assert: exactly one charge; second returns prior result
- id: DR-06-echo
  fault: loop agent audio into caller mic at -12 dB
  assert: no state change from echoed segment

And the comparison itself. Again a simplified sketch:

FIELDS = ["item", "qty", "total"]

def drift(call_state, server_state, ui_state):
    out = []
    for f in FIELDS:
        vals = {"call": call_state.get(f), "server": server_state.get(f),
                "ui": ui_state.get(f)}
        if len(set(map(str, vals.values()))) > 1:
            out.append({"field": f, **vals})
    return out   # empty list means all three surfaces agree

Set pass criteria per scenario: zero persistent drift, and transient drift under a bound you choose from your own latency data (for example, 1 second after a confirmation).

The call side is the hard part to do by hand. Evalgent generates scenarios from your agent's prompt and flows, and its caller profiles vary accent, speech pace, interruption level, response delay and background noise. The simulator reaches the agent over PSTN, SIP or WebRTC. Evalgent judges from the audio, not only the transcript, which matters for the heard-versus-generated gap above. The browser reads and the server reads in this procedure are your own harness. In production, you can push call recordings to Evalgent for monitoring. It maps calls to scenarios and flags tool-call failures, which is where a rejected stale write would surface. Neither replaces the version stamps you add.

What to do first

Add `base_version` to both paths and return 412 on mismatch. Log played-audio duration. Run DR-01 to DR-03 on every release. Check each action's blast radius first, so you know which drifts cost money. Writes that outlive the call also need a post-call write-back check. Later in this series: specific system integrations.

Frequently asked questions

Why does my voice agent say something different from what the screen shows?

The two surfaces hold separate copies of state and update at different times. The agent speaks after a tool result and TTS delay. The screen updates after a push and render. A stale version, a dropped socket or an interrupted sentence leaves them disagreeing. Stamp versions on both and compare them after each turn.

Should the voice agent or the UI be the source of truth?

Neither. The server should be. Both the agent and the UI send intents with a base version, and the server decides the state. Each surface renders a projection of that record. This stops the voice path or the click path from overwriting the other without knowing the state had changed.

How to keep voice agent and UI state in sync?

Use server-authoritative state with a version per record. Push each change with a session sequence number. Reject stale writes with a 412. On reconnect, send a full snapshot, not a partial merge. Confirm aloud only after the server acknowledges the commit, so speech never runs ahead of the stored record.

How to test voice agent and screen state drift?

Reduce the call, the UI and the server record to one shape. Compare them after every turn while injecting faults: delayed pushes, dropped sockets, mid-sentence interruptions and mid-speech clicks. Repeat timing-sensitive cases 20 to 30 times. Fail the release when drift persists past the next turn.

What happens to a voice agent when the WebSocket reconnects?

MDN's client guide describes a `close` event but not automatic reconnection after an unexpected close, so your code must reconnect. The screen may have missed commits during the gap. Send the last known version, receive a snapshot, replace local state, and let the agent recap what changed if it spoke meanwhile.

How do you handle barge-in when the screen has already updated?

Track how much audio the caller actually heard. Mark unheard items as pending on screen and have the agent restate them. Do not accept a bare "yes" that followed cut-off speech. With OpenAI's Realtime WebSocket path, the client reports played duration through `conversation.item.truncate`.

How do you prevent race conditions between voice and click actions?

Give every logical action one intent id and a base version. Apply the first valid intent and reject the second with a 409 or 412. Define whether a click or a confirmed voice command wins before launch. Tell the caller when their action lost, and refresh the screen.

How do you make voice agent screen updates accessible to screen reader users?

Put changed content in an `aria-live="polite"` region that exists empty at page load, then update its text. WCAG 4.1.3 asks that status messages be exposed without moving focus. Reserve `assertive` for blocking errors, and test with a screen reader running while the agent speaks.

The bottom line

Voice and screen drift is a versioning problem, not a speech problem: one server record, one version per change, and every surface renders from it. Test it by comparing what the caller heard, what the server stored and what the screen shows after every turn, under injected delays, drops, interruptions and clicks.