AI Turn-Taking: Design a Safe Interruption Policy

Safe AI turn-taking requires four separate decisions: whether speech is present, whether the user has finished, whether the avatar should yield, and whether any downstream action may continue. Voice activity detection can support the first decision. It cannot make the other three on its own.

For a bank, government service or regulated enterprise, this distinction matters. A system that treats every pause as the end of a turn will interrupt account numbers, names and considered answers. One that treats every sound as a barge-in will stop whenever somebody says “yes”, coughs or when its own speaker audio leaks back into the microphone. Worse, cancelling the avatar’s voice may leave a payment, booking or case update running in the background.

The right design is a documented turn contract: a state machine, evidence set and action boundary that can be tested across users, languages, devices and deployment models.

Separate the four decisions

1. Is somebody speaking?

Voice activity detection, or VAD, classifies regions of an audio stream as speech or non-speech. It can trigger a listening indicator, preserve audio from just before speech began and help suppress unnecessary processing during silence.

It does not know whether the speaker has completed a thought. A pause after “my reference is…” is acoustically quiet but semantically unfinished. Background speech may be genuine voice activity but belong to somebody else.

2. Has the user finished the turn?

Endpointing decides when to commit the user’s input for response. A fixed silence window is simple and predictable, but it trades speed against false endings. Semantic turn detection can consider the partial transcript and conversational context, but it adds another model, configuration and failure surface.

Current OpenAI Realtime API documentation, for example, distinguishes server VAD from semantic VAD and makes the trade-off explicit: semantic detection can wait longer when the user appears unfinished. That is a useful architectural pattern, not a requirement to use a particular provider.

3. Should the avatar yield?

Barge-in is a separate policy. A user may say “stop”, begin a correction or offer a short backchannel such as “right” while the avatar is speaking. Those events should not necessarily have the same result.

Useful policies distinguish an explicit stop, a probable new turn and a probable acknowledgment. They also define what the avatar does after a false interruption: resume the remaining sentence, restate it, ask for confirmation or remain silent.

4. What must be cancelled?

Stopping audio output is not the same as cancelling generation, retrieval, tool execution or a business transaction. Every interruptible response needs a cancellation scope. The safest default is to let the avatar’s presentation layer yield quickly while protected actions remain behind an explicit confirmation and authorisation boundary.

This complements the safe action-gateway design for avatar tool calls: conversational timing must never become an accidental transaction-control mechanism.

Write a six-state turn contract

A practical implementation can model each session with six observable states.

1. Listening: audio capture is active and the interface makes that state clear.
2. Speech candidate: activity has crossed the configured threshold, but no completed user turn exists yet.
3. Endpoint candidate: acoustic or semantic evidence suggests the thought may be complete; a short decision window remains open.
4. Input committed: the transcript and relevant context are frozen for this response, with a correlation identifier and configuration version.
5. Avatar speaking: generation, voice and rendering output are active; interruption rules and cancellation handles are available.
6. Interrupted or recovering: output has yielded, pending work is cancelled or contained according to policy, and the system decides whether to listen, resume, clarify or hand off.

Do not collapse these into “user talking” and “agent talking”. The intermediate states are where teams diagnose why an avatar spoke too early, ignored a correction or carried on with an action after the visible conversation stopped.

Frameworks expose these stages differently. LiveKit’s turn-taking guidance, for example, separates user-activity detection, interruption handling, pre-emptive generation, audio preprocessing and speech scheduling. The valuable lesson is the separation of controls—not any universal default value.

Make microphone behaviour part of the design

Turn logic inherits every weakness in the capture path. Browser and device processing can alter the signal before it reaches the detector. The W3C Media Capture and Streams specification defines constraints for echo cancellation, automatic gain control and noise suppression. Teams should record the requested and resolved settings rather than assume every browser, kiosk or headset behaves identically.

Test the avatar’s own audio leaking into the microphone. Echo cancellation can reduce this problem, but room acoustics, speaker volume, device placement and processing support vary. A five-metre display in a public venue has a different acoustic boundary from a headset in a contact centre.

Give users visible confirmation that the microphone is available, muted, listening or recovering. Provide captions, text input and a deliberate stop control. The broader guide to accessible AI-avatar interaction explains why voice cannot be the only route through a service.

Tune by journey, language and risk

One global threshold is rarely defensible. A fast information kiosk may favour short endpoint delays. A benefits interview, healthcare intake or wealth discussion may need longer pauses and more explicit confirmation. Numeric identifiers and addresses should tolerate hesitation without committing fragments.

Language matters too. Pause length, sentence-final cues, code-switching and recognition stability vary. The right Arabic configuration may differ by dialect, acoustic environment and task; it should not be inferred from English results. Yepic’s Arabic avatar evaluation guide provides a broader framework for testing language, voice and cultural behaviour in the target environment.

Accessibility testing should include people who speak slowly, stammer, use atypical prosody, have respiratory pauses or rely on an alternative input mode. Optimising only for the median speaker can make a system feel faster while excluding users who most need time.

Measure errors, not only speed

End-to-end response time remains important, but a turn-taking scorecard should include:

• speech-start detection delay;
• end-of-turn decision delay;
• false endpoints, where the avatar cuts into an unfinished turn;
• missed endpoints, where it waits after the user is clearly finished;
• true and false interruption rates;
• time from interruption to audible and visible silence;
• recovery accuracy: resume, restart, clarify or remain stopped;
• actions attempted or completed after an interruption;
• task completion and abandonment by language, device and relevant user group.

Measure these during representative concurrent load, not only in a quiet single-user demo. Queueing and network variation can extend the apparent endpoint or cancellation path. The existing guide to load-testing the whole avatar conversation shows how to keep latency and quality evidence from the same run.

Test the failure cases that demos avoid

Build an evaluation set containing complete questions, mid-sentence pauses, lists, dates, account references, self-corrections, soft speech, loud speech, overlapping speakers, backchannels, coughs, keyboard noise, television audio and the avatar’s own voice. Repeat it across supported languages, approved devices and realistic rooms.

Then add operational failures: packet loss, jitter, delayed transcripts, duplicated events, a detector restart, TTS cancellation delay and a tool call that crosses the interruption boundary. Verify that event order is idempotent. A late “speech ended” event must not revive a cancelled response or commit the same input twice.

Retain a privacy-minimised evidence record containing session and turn identifiers, state transitions, timestamps, detector and policy versions, selected confidence bands, cancellation result, action status and test-case label. Raw audio and full transcripts should be retained only when justified by the use case, policy and legal basis. Yepic’s guide to privacy-minimised AI audit trails provides a useful starting point.

Compare deployment models honestly

Public cloud can provide managed speech and semantic-turn models, rapid upgrades and pooled capacity. It may add network dependence and send live audio beyond the customer environment, depending on the design and contract.

Private cloud can combine managed infrastructure with defined regional, tenant and network boundaries. Responsibility for audio processing, telemetry and model updates still needs to be explicit.

Customer-hosted inference can keep capture, transcription, endpointing and avatar generation within the organisation’s environment, subject to the scoped architecture. It also makes the customer and implementation team responsible for model lifecycle, device compatibility, capacity and observability. On-premise is not automatically lower latency or more accurate.

A hybrid design can run immediate speech detection at the edge while performing transcription or semantic endpointing elsewhere. That may improve responsiveness, but teams must document which audio or derived signals cross each boundary and what happens during disconnection.

Twelve questions for architecture and procurement

1. Which component detects speech, and where does it run?
2. How is end-of-turn detection different from VAD?
3. Which parameters vary by journey, language and device?
4. How are backchannels distinguished from real interruptions?
5. How quickly do voice and video stop after a valid interruption?
6. What happens after a false interruption?
7. Can generation, retrieval and TTS be cancelled independently?
8. Which actions may continue after presentation stops?
9. How are duplicate, delayed and out-of-order events handled?
10. What alternative input and stop controls are provided?
11. Which event evidence is retained without storing unnecessary audio?
12. What change triggers a multilingual and device-specific retest?

Where Yepic’s experience applies

Yepic has spent years developing proprietary talking-photo and real-time avatar technology, including low-latency multimodal interaction, multilingual delivery, APIs and optimisation for commercial GPUs. Private, sovereign and customer-hosted implementations can be scoped for a customer’s environment and GPUs, alongside cloud and private-cloud designs where those are more appropriate.

For Abu Dhabi Aviation and Oracle, Yepic supported development and production environments, API and iframe integration, real-time streaming, captions, microphone behaviour, WebRTC and network testing, browser remediation, cybersecurity work and ongoing maintenance. The enterprise avatar integration demonstrates why microphone and media behaviour must be treated as production architecture, not a demo setting. It is not presented as a completed customer-hosted turn-detection deployment.

The practical next step is to choose one high-value conversation, write its six-state turn contract, mark the irreversible action boundary and test the failure matrix on the actual devices, languages and network path. That gives architecture, risk and procurement a common answer to a deceptively simple question: when is it really the avatar’s turn?