Acoustic Echo Cancellation: Stop the Avatar Hearing Itself

Acoustic echo cancellation (AEC) for voice AI should stop an avatar’s own speech returning through the microphone while leaving the microphone open for the person to interrupt. The reliable design is not “turn on noise cancellation”. It is a complete endpoint contract: preserve a clean reference to the audio being played, cancel that signal close to capture, qualify one primary processing boundary, and test every supported device and room during double-talk.

This matters because a real-time avatar can hear itself in several damaging ways. Its speech may be transcribed as a new user request, trigger a false interruption, create a feedback loop or make a genuine speaker difficult to recognise. In a bank branch, citizen-service kiosk or transport control room, the result is not merely awkward. It can corrupt the conversational state and undermine confidence in the service.

AEC is therefore part of the application’s control surface. It connects endpoint hardware, operating-system or browser processing, voice activity detection, speech recognition and the avatar’s interruption policy. Hosting models on the customer’s GPUs does not remove the acoustic problem at the edge.

Separate four different audio problems

Teams often use “echo”, “feedback” and “noise” interchangeably. That leads to the wrong control being approved.

• Acoustic echo is the avatar’s far-end playback travelling from a loudspeaker, through a room, into the microphone. AEC estimates that path using the playback reference and removes the matching component from capture.
• Feedback is an unstable loop that produces ringing or howling. Gain structure, microphone placement and routing matter alongside cancellation.
• Background noise includes ventilation, traffic, keyboard sounds and other speakers. Noise suppression or voice isolation can help, but these controls do not replace a matched echo reference.
• Reverberation is the room’s decaying reflections of every sound, including the user’s voice. Long or changing room tails make echo estimation harder and may require acoustic treatment, dereverberation or different hardware placement.

These controls can interact. Aggressive noise suppression may remove quiet or atypical speech. Automatic gain control can amplify residual echo. Two independent echo cancellers can distort audio or fight each other. Procurement should therefore evaluate the combined capture chain, not collect isolated feature checkboxes.

Map the five-stage echo path

The simplest useful architecture model contains five stages.

1. Render reference: the exact avatar audio stream that the endpoint intends to play.
2. Playback chain: mixing, volume, codecs, operating-system processing, amplifier and loudspeaker.
3. Acoustic path: distance, surfaces, people, furniture and changing room conditions.
4. Capture chain: microphone array, analogue and digital processing, operating-system or browser capture and channel conversion.
5. Conversation chain: AEC output feeding voice activity detection, speech recognition, turn decisions and the avatar application.

An AEC normally needs two inputs: the captured near-end signal and a time-aligned copy of the far-end render signal. The WebRTC Audio Processing Module, for example, describes primary capture and reverse render streams processed frame by frame, with echo cancellation, noise suppression and gain control as separate functions. Its integration guidance places processing close to the hardware abstraction layer because delay and routing knowledge are endpoint concerns.

If an application sends only microphone audio to a remote server and never exposes what the loudspeaker actually rendered, a server-side model has a weaker cancellation problem. It may attempt source separation, but it lacks the clean matched reference used by conventional AEC. This is why the cancellation boundary should usually be at the browser, native application, operating-system voice-processing path or installed room DSP.

Choose one primary cancellation boundary

Four boundaries are common, and each can be valid.

Browser WebRTC processing

A web application can request the echoCancellation media constraint. The current W3C Media Capture specification says that a value of true asks the user agent to apply echo cancellation, while leaving the extent of removal to the user agent. That request is useful, but it is not an acceptance result. Record the browser, operating system, device route and the track’s effective settings, then test the audio produced.

Native or operating-system voice processing

Mobile and desktop platforms may expose voice-processing modes with integrated AEC, noise suppression and gain control. These modes can have better access to device timing and the playback route. They can also change sample rates, channel layouts or audio character. Switching from speaker to headset, Bluetooth or an external interface may select another path entirely.

Room hardware or DSP

Fixed kiosks, meeting rooms and public installations may use a microphone array or dedicated DSP that owns the acoustic reference. This can deliver consistent behaviour for a known space, but firmware, wiring, output routing and physical changes become part of the controlled configuration.

Application-managed processing

A custom native application can integrate an AEC library and control the reference, buffers and metrics directly. That offers visibility but transfers substantial timing, platform and maintenance work to the implementation team.

Do not enable every layer by default. Cascaded AEC, gain and suppression stages can introduce pumping, clipping, delay and near-end speech loss. The architecture record should name the primary owner and list any unavoidable upstream or downstream processing.

Preserve double-talk and genuine barge-in

Muting the microphone whenever the avatar speaks prevents self-transcription, but it also prevents the user speaking over the avatar. That is half-duplex gating, not echo cancellation.

AEC must work during double-talk: the avatar and user speaking at the same time. The far-end component should be suppressed while near-end speech remains intelligible enough for detection and recognition. This is essential for natural interruption, urgent correction and users who do not follow an expected conversational rhythm.

Audio interruption and business cancellation must also stay separate. Detecting “stop” may silence synthesis immediately, but it should not silently reverse a payment, submit a different form or cancel an already committed action. The turn-taking policy should define presentation, generation and action cancellation independently.

Create an endpoint audio contract

For each supported endpoint family, record at least:

• device, microphone, speaker and their physical geometry;
• operating system, browser or native application version;
• capture API, constraints, sample rate, channels and frame size;
• primary AEC owner, mode and effective settings;
• noise suppression, voice isolation and automatic gain controls;
• playback route, mixer, maximum supported volume and clipping behaviour;
• render-reference source, transformations and expected alignment;
• warm-up or adaptation behaviour after start, reconnect and route change;
• room class, distance, orientation and acoustic assumptions;
• approved fallback when the expected processing path is unavailable.

This contract belongs beside the enterprise WebRTC network design. Network statistics can explain packet loss, jitter and round-trip time, but they do not prove that the captured signal is free of playback echo.

Design for six predictable failure modes

1. Missing or wrong reference: system audio is mixed after the reference tap, or another application plays sound that the canceller never sees.
2. Delay drift: buffering, resampling or device changes move the reference relative to capture. The acoustic estimate becomes stale.
3. Non-linear playback: a loudspeaker, amplifier or limiter clips. The room receives a distorted signal that no longer matches the reference.
4. Route change: the user connects headphones, Bluetooth or an external display. The acoustic path changes or disappears, but processing remains in its old state.
5. Long or changing room tail: reflective spaces, moving people or a repositioned kiosk exceed the qualified acoustic assumptions.
6. Processing conflict: stacked AEC, noise suppression or voice isolation removes near-end speech or produces audible artefacts.

For browser deployments, hardware and browser variation deserves the same seriousness as firewall traversal. Yepic’s Abu Dhabi Aviation and Oracle integration included real-time streaming, captions, microphone behaviour, WebRTC and network testing, browser remediation and cybersecurity support across development and production environments. That experience illustrates why endpoint behaviour must be tested as part of the whole service. It should not be read as a claim of a completed customer-hosted AEC programme.

Test the room, route and conversation together

A useful acceptance plan combines signal measurements with conversational outcomes. Run at least these scenarios for every supported endpoint class:

• far-end avatar speech only, at several playback volumes;
• near-end user speech only, including soft and off-axis speech;
• double-talk with different relative levels and interruption timings;
• background speech, steady noise and impulsive noise;
• headset, speaker, Bluetooth and display-audio route changes;
• startup, reconnect and immediate speech before the canceller adapts;
• quiet and reflective rooms, with the expected user distances;
• multilingual, accented and atypical speech relevant to the service;
• deliberate clipping, volume changes and CPU pressure;
• long sessions and repeated avatar-user interruptions.

Echo return loss enhancement and residual echo level can help diagnose the signal path, but a high engineering metric is not enough. Also measure whether avatar words appear in the user transcript, false speech starts, interruption success, near-end recognition quality, audible artefacts, adaptation time and processor load. Add these cases to the service’s end-to-end load tests and synthetic conversation probes where the endpoint can be represented faithfully.

Accessibility review is part of audio qualification. Quiet voices, assistive devices, atypical prosody and alternative interaction modes may behave differently under aggressive suppression. The accessible-avatar design should include captions and non-voice paths rather than assume every user can recover from a missed interruption.

Deployment location does not solve endpoint acoustics

Cloud, private-cloud and customer-hosted avatar architectures can all use effective AEC. They can all fail at it too.

A customer-hosted design can, when properly scoped, keep selected audio, speech-processing, model and telemetry components inside the customer environment. That can support residency and control requirements. It does not make a browser’s echo-cancellation implementation consistent, provide a clean reference automatically or correct poor microphone placement. It also makes the customer and implementation team responsible for qualifying supported endpoints, packaging native components where required, monitoring versions and maintaining test fixtures.

A managed cloud service may simplify upgrades and central observability, but the physical echo path remains in the branch, kiosk, room or user device. A hybrid design may therefore keep latency-sensitive capture processing at the endpoint while placing other components according to data classification, latency, integration and operational requirements.

Twelve questions for architecture review

1. Which component owns AEC for every supported endpoint and route?
2. Where is the render reference tapped, and what transformations follow it?
3. How are delay, drift and route changes detected or reinitialised?
4. Can the user interrupt during avatar playback without the avatar transcribing itself?
5. Which noise suppression, voice isolation and gain stages are active?
6. How do we detect duplicated or cascaded audio processing?
7. Which device, operating-system, browser and room combinations are supported?
8. What is the behaviour during warm-up, reconnect and processing failure?
9. Which signal and conversational measures define acceptance?
10. How are quiet, accented and accessibility-relevant speech patterns tested?
11. What audio or telemetry is retained, where does it flow and who may access it?
12. Which configuration or software changes trigger requalification?

Approve an audio system, not an AEC checkbox

The production gate should be an endpoint matrix with named processing ownership, a verified render-reference path, double-talk evidence and conversational acceptance results. AEC succeeds when the avatar does not hear itself, the user can still interrupt, and the rest of the stack receives trustworthy near-end speech.

Yepic can scope real-time avatar deployments across cloud, private-cloud and customer-hosted environments, including integration and endpoint testing. The appropriate audio architecture depends on the hardware, room, browser or native stack, privacy boundary and operational model; it should be proven on the deployment’s real endpoints before production.